Due to competition and low chances of obtaining funding (Garner et al 2013), researchers are now more than ever struggling to obtain the resources to fund their research project. As applying for funding can be time-intensive, researchers might feel like spending more time on writing applications than on research. So, when having a great idea, why not try and sell it to multiple funders and see which one ‘bites’?

“With grant success at all-time low, scientists are working harder than ever to fund their research. They respond to the competitive economic times by submitting more applications. They may also simultaneously or serially submit applications to multiple funding agencies to increase their odds of getting funding. Some grant agencies allow the submission of applications with identical or highly similar specific aims, goals, objectives and hypotheses. But we believe that researchers should not accept duplicate funding for the same work – either the whole study or any part of it.”

Quote from: Same work, twice the money? – Harold R. Garner, Lauren J. McIver & Michael B. Waitzkin – Nature 493 (2013).

It is therefore not unsurprising for a research proposal to be submitted to multiple funding bodies, either in identical or slightly modified form. This system of parallel applications increases the chance of obtaining funding. In addition, when funding is only partially granted, additional funding obtained from (an)other source(s) can help to acquire the budget needed for the complete project.

While parallel applications aren’t necessarily problematic, some good practices should be kept in mind. First of all, researchers should be transparent towards the funding body and acknowledge if a (partially overlapping) proposal is under evaluation elsewhere. The same principle should be followed in case of overlap with an already granted project. Although there might be good reasons for the overlap, this should be communicated in a clear way. Finally, if during the application phase, the proposal is granted elsewhere, proper action should be taken in order not to obtain double funding. Researchers should not accept funding twice.

Depending on your discipline, statistical analysis is crucial to substantiate your results. The problem that many researchers are faced with is that they are trained to be researchers and not necessarily to be a statistician, which is a whole field of its own. With this in mind, researchers have to be critical with regards to the statistical analysis they perform and consult experts wherever necessary. Just like when working with advanced technical equipment, you consult someone with expertise in working with this equipment, you have to be aware of how the equipment works and what information can and cannot be drawn from it.

Good academic practices on statistical analysis

The way the data are analysed can severely impact the conclusion and validity of your research. As such, researchers should aim to have a sound statistical analysis strategy even before collecting the first data. Some good practices:

• Everything starts with a clearly defined research question. Set up your study and plan the statistical analysis according to the research question. Consider the number of observations you will go for? Which are the variables you will consider? Which are the relevant statistical tests?
• Take courses in the field of data analysis. Although this does not make you an expert statistician, you get a much better idea what the different techniques are capable of and which pitfalls you should be aware of.
• Accross all stages of a research project, statisticians are involved. In the budget calculation of your project, foresee some money for statistical assistance.
• Be aware of the limitations of different statistical tests. Be critical and do not use a test just because this is how another researcher did it or because it is the standard method in your research group.
• Be critical of your data. A statistically significant result does not automatically mean that the result is robust or relevant. Are there reasons to explain the result?
• Exploring the use of different statistical tests is not in itself a problem as long as the tests have been designed for the intended purposes and not only to be able to ‘choose’ the best results. Analyses should account for the number of tests carried out.
• Similarly, omission of data might be acceptable, but the reasons for this should be acknowledged and valid.

Source: Best practices for statistical data analysis

Preregistration

Several (questionable) research practices can lead to publication bias and false positive findings. Important examples of this kind of practices are p-hacking, HARKing (see below) and not publishing negative research results.

An effective solution is to define the research questions and analysis plan before observing the research outcomes – a process called preregistration. As such, preregistration helps the researcher to interpret the data and focus on the original research question (what can and cannot be concluded), to increase transparency when reporting research, and prevent from falling into questionable research practices. As a consequence, the focus of the work shifts from having ‘nice’ results to having ‘accurate’ results.

What is preregistration about?

When preregistering a study, the researcher (or research team) determines the plan of the research to be performed before starting the data collection and/or data analysis. Basically, this plan contains:

• the research question
• the hypothesis
• the dependent and independent variables that will be collected
• the study design
• the planned methodology
• the sample size and the method to determine the sample size
• the way the data will be analyzed
• the outlier and data exclusion criteria

The plan is then timestamped and deposited into a registry. This does not necessarily mean that the plan has to be publicly available or that there is no flexibility in the analysis of the data after these have been obtained, nor that one should stop performing exploratory research. It should, however, be clear to the reader of the article which parts of the research are hypothesis testing and which are unexpected findings that drive a new hypothesis. Furthermore, any deviations from the preregistration should be made explicit and justified.

Preregistration is well taken up in the field of clinical trials. Here, more and more medical journals will only consider publication if the trial has been registered beforehand. Nowadays, preregistration is finding its way in psychology and into other fields as well.

Registered reports

Registered reports are a more formalised way of looking at preregistration and are an even more powerful tool to prevent publication bias. Within this approach, the proposed plan is sent to a journal before the start of the study (figure 1). The journal subsequently sends the plan (not the results) for peer-review, provides feedback and in principles ommits itself to publish the study if conducted according to the submitted plan. One of the main advantages of this approach is that the study design is reviewed early on, allowing to intervene before starting the study. Moreover, in the form of a registered report, preregistration reduces publication bias as journals will publish the work, independent of the outcome (figure 2 and 3) and acknowledges the value of negative results.

Based on the definition proposed by the US National Academies of Sciences, Engineering, and Medicine (Reproducibility and Replicability in Science (2019), p. 6), “reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis. This is different from replicability which is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.” Please note that within the literature reproducibility and replicability are often used interchangeably. So while they are strictly speaking not synonyms, a failure to either reproduce or replicate findings both may point to the same, namely inconsistent results.

During the last decade, concerns have been raised regarding the lack of reproducibility and replicability of research findings in many disciplines, including social sciences (Caramer, 2018), psychology (Open Science Collaboration, 2015) and biomedical sciences (Pusztai, 2013), raising concerns on the reliability of science (Nature, 2016), as denoted as the ‘reproducibility crisis’.

Why is a study not reproducible or replicable?

There are many different reasons why a study might not be reproducible, including a faulty experimental design, variables that are not taken into account, wrong statistical analysis, biased reporting focusing on the desired effects, no access to all information necessary to reproduce or conditions that can’t be recreated. Within biomedical sciences, failures to replicate research sometimes have been explained by the use of wrong reagentia, including incorrect labelled mice or cell lines (Kafkafi et al, 2018; Eckers et al, 2018).

Failing to reproduce and replicate data is an inherent feature of science. Therefore, failure to reproduce/replicate a study does not necessarily mean that a study is faulty and cannot be trusted. In some research fields the interpretation of the data, f.e. using historical sources, doesn’t necessarily requires reproducibility yet might still be desirable. However in other research fields, to be able to draw conclusions, researchers have to take sufficient measures in order to make reproducibility/replicability of the research possible. Potential measures can include:

• clear standard operation protocols
• detailed logging of all aspects of the research throughout the project
• detailed logging of all metadata
• …

In addition, sufficient details regarding research methodology and analysis have to be available to everyone not directly involved in the research.

Available tools for reproducible and replicable research

As mentioned before, p-hacking and HARKing significantly reduce the potential to replicate research findings. As such, the use of preregistration and registered reports will have a very positive effect on the replicability of research.

Moreover, to increase the reproducibility and replicability of research, reporting guidelines and checklists have been drawn up for several disciplines. These usually “specify a minimum set of items required for a clear and transparent account of what was done and what was found in a research study, reflecting, in particular, issues that might introduce bias into the research” (Adapted from the blog “It’s a kind of magic: how to improve adherence to reporting guidelines” Marshall, D. & Shananhan, D. (2016, February 12)).

Some examples include:

• The ARRIVE (Animal Research: Reporting of in Vivo Experiments) guidelines intended to improve the reporting of research using animals.
• The Materials Design Analysis Reporting (MDAR) Checklist for Authors that is applicable to studies in the life sciences.
• The ICMJE Recommendation for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals.
• Within psychology, reporting standards for quantitative research in psychology have guidelines published by the American Psychological Association
• Inspiration for qualitative research can be drawn from the SRQR and COREQ standards..
• Researchers in humanities or empirical social sciences can look into the standard published by the American Educational Research Association (AERA).
• Resources for Reproducible Research by Reproducibility for Everyone

Finally, within certain fields, specific guidance and tools may exists to increase the reproducibility of the analyses (focusing on the code and software used).

Research integrity is all about doing research in a responsible way. Research wants to answer or reflect upon questions about the real world. Therefore, researchers set up studies or experiments, designed specifically to be able to answer their research questions. The choices in the study design are crucial, largely determining the quality of the research. Because of their importance, the methods, rules, models and principles used in research are a topic of (theoretically) study on their own, together called methodology.

It is essential for all researchers to be familiar with the good practices of methodology and design in their field of study.

Good academic practices on (different) methodologies

Methodologies differ largely among disciplines – literature researchers of course use other methods than molecular biologists. Courses in discipline specific methodology are an obligatory part of most university education programmes, as well as doctoral training programmes.

It’s impossible to summarise all disciplines’ methodologies, but in most research fields, for instance, the following considerations are key:

• The variables measured (both influencing factors and outcome) should correspond to the question you want to answer, as directly as possible.
• The design of the experiment or study should enable you to answer the research question.
• Some form of randomisation might be advisable to eliminate the unwanted influence of confounding factors (like unknown or unmeasurable ones).
• The sample size should be appropriate: large enough to be able to statistically detect existing phenomena (according to a power calculation) – but, if applicable, not larger than ethically responsible.

Good academic practices on design and methodology

When you have carefully designed and planned your research actions and analysis, many research disciplines consider it a good practice to:

• Specify all choices for the study design and research activities in a protocol (that may even be published).
• Preregister the study and plan of analysis.
• Have your preregistered study and analysis plan peer reviewed, and in principle accepted for publication, before gathering data – so-called registered reports.
• These exemplary practices reduce the risks of unacceptable research practices like bad study design (because your study design will be reviewed before actually doing the research) or HARKing.

As a researcher you have the responsibility to conduct research in line with (inter)national and institutional norms and values, and where applicable, legal obligations. This does, however, not mean that this responsibility should lie entirely in the hands of the researcher. Although research usually starts with an idea, this idea can only be further developed when the necessary infrastructure and research tools are available. Given the complexity of modern research, support by the host research institution and/or your department/faculty is indispensable when aiming for high-quality research and research integrity. The following module will look at how good academic research practices respect the basic principles of integer research:

Support from your host institution can come in many forms. Institutions can promote awareness and encourage a culture of research integrity, for example by having clear guidelines and policies in place to promote good research practices. However, research institutions are also expected to provide the necessary proper infrastructure. This includes the necessary space and equipment required to carry out a research project, including an infrastructure for the management and protection of data and research materials in all their forms.

In addition, due to the complexity of research, it is necessary for institutions to have support in place to assist researchers in need of advice. Researchers should be able to find help with good academic practices, ethics approvals (in the form of ethics committees), legal issues and their own safety and that of others when planning and conducting research.

Researchers themselves have responsibilities. In the first place, it is of utmost importance that researchers familiarise themselves with the good academic research practices described in the ALLEA Code, and the host institution’s expectations and procedures with regards to research. Researchers should ensure that they use scientific methods that are justifiable within their field, and must be precise and accurate when performing research. They should check whether the tools they intend to use (for instance, laboratory equipment, standard questionnaires, archives) are adapted to the work to be performed and are ready to be used in optimal conditions. (Code of ethics for scientific research in Belgium, 2009)