Possible flaws in a study design

Unreliable research findings due to flawed study design
When a study is designed in a way not aligned with methodological good practice, you may get wrong answers to your research question. This is not an improbable scenario, but a major problem in science today, as shown by Ioannidis in his 2005 article “Why most published research findings are false”.
Unreliable results, attributable to flawed study design, constitute a terrible waste. That your research is useless, is not only a waste of your time and energy, but even worse, it may distort the literature with wrong insights, guiding future research in irrelevant directions.
As such, designing bad research if it could have been prevented, may be seen as an unacceptable research practice – precisely because of its detrimental impact on the literature. Unreliability of findings can be reduced by carefully designing your research, learning good design practices from the literature and double-checking methodology matters with your supervisor and colleagues.
Bias
Inappropriate study design is one of the most common causes of bias in research which results in false positive findings.
According to Ioannidis, the most important drivers of the high rate of false positive findings in clinical medicine and biomedical research, are:
- Solo, siloed investigator limited to small sample sizes
- No pre-registration of hypotheses being tested
- Post-hoc cherry picking of hypotheses with best p-values
- Only requiring p <0.5
- No replication
- No data sharing
Integration of sex and gender analysis as a mark of research excellence
Wherever human beings or animals are involved in research, sex and gender will be an issue and should be considered and addressed in the research design. Addressing the sex and gender dimension of research implies that sex and gender are considered as key analytical and explanatory variables in research. If relevant sex or gender issues are missed or poorly addressed, research results will be partial and potentially biased. Responsible and excellent research practices therefore consider sex and gender aspects in all stages of the research cycle.
Gender biased research
Gender biases originate in the often unintentional and implicit differentiation between men and women by placing one gender in a hierarchical position relative to the other in a certain context, as a result of stereotypical images of masculinity and femininity. An example of gender bias in research focuses on the experience and point of view of either men or women, while presenting the results as universally valid for both sexes. Gender biases influence the participation of men and women in research and the validity of research.
These tools can support and inspire you in making your research sex and gender-sensitive:
- Gendered Innovations Practical methods for sex and gender analysis in science, health & medicine, engineering and environment. Case studies provide concrete illustrations of how sex and gender analysis leads to innovation. (Stanford University)
- Toolkit Gender in EU-funded research Toolkit with accessible introduction and practical tools on the integration of the gender dimension in research. Analysis of case studies drawn from health; food, agriculture and biotechnology; nanosciences, materials and new production technologies; energy; environment; transport; socio-economic sciences and humanities; science in society and international cooperation.
- Webinars
- Integration of sex and gender analysis into research: how to integrate sex, gender, and intersectional analysis into research design, and how this can lead to discovery and improved research methodology.
- Gender in research (proposals and projects)
Gender-blind research
Gender-blind research does not take gender into account when relevant, being based on the often incorrect assumption that possible differences between men and women are not relevant for the research at hand. This is the case when general categories such as ‘people’, ‘patients’ or ‘users’ do not distinguish between men and women. Research based on such categories may well draw partial conclusions based on partial data. This limits the generalizability and applicability of research findings.
Gender-sensitive research is qualitatively better and more valid. When research takes into account the differences between men and women in the research population, the results will be more representative.