1 Automated Syllabus of Metascience Papers
Built by Rex W. Douglass @RexDouglass ; Github ; LinkedIn
Papers curated by hand, summaries and taxonomy written by LLMs.
2 Metascience
2.1 Publication Bias
Consider adopting Registered Reports (RRs) as a publication format, where peer review and the decision to publish occur before results are known, in order to reduce publication bias and increase the credibility of findings. (Scheel, Schijen, and Lakens 2021)
Carefully consider and address the risk of selective reporting of nonsignificant results and reverse P-hacking, where researchers manipulate data or analyses to achieve nonsignificant results, as these practices can distort the scientific literature and lead to biased estimates of effect sizes. (Chuard et al. 2019)
Carefully consider the limitations of various publication bias detection methods, such as low statistical power and assumptions of homogeneous true effect size, and choose appropriate methods based on the specific context and characteristics of your meta-analysis. (Aert, Wicherts, and Assen 2019)
Carefully consider the potential impact of including unpublished and grey literature study data in meta-analyses, as these sources may introduce biases and affect the pooled effect estimates and overall interpretation of the results. (C. M. Schmucker et al. 2017)
Interpret mixed results (i.e., a combination of both significant and non-significant findings) as potentially providing strong evidence for the alternative hypothesis, especially when statistical power is high and Type I error rates are controlled. (Lakens and Etz 2017)
Exercise caution when interpreting the results of meta-analyses using p-uniform and p-curve methods, particularly when dealing with heterogeneous data sets, as these methods may produce erratic behavior, implausible estimates, or overestimate effect sizes. (Aert, Wicherts, and Assen 2016)
Consider using pre-analysis plans (PAPs) to specify hypotheses and analyses prior to data collection in order to minimize issues of data and specification mining and provide a record of the full set of planned analyses. (“Editorial Statement on Negative Findings” 2015)
Prioritize sharing negative results, even if they are not as highly valued in the current scientific culture, because they contribute to filling gaps in knowledge and moving towards unabridged science. (Matosin et al. 2014)
Base your power calculations on realistic external estimates of effect sizes, rather than on inflated estimates derived from preliminary data or arbitrary thresholds of minimal substantive importance. (Gelman and Carlin 2014)
Publish your protocols and statistical analysis plans before conducting clinical trials, and strictly adhere to them during analysis to avoid selective reporting biases caused by discrepancies in analyses between publications and other study documentation. (Dwan et al. 2014)
Consider the impact of dissemination bias when conducting systematic reviews, as a substantial portion of studies approved by RECs or included in trial registries remain unpublished, leading to selective reporting of results and potentially skewed conclusions. (C. Schmucker et al. 2014)
Prioritize the timely submission of your studies to peer-reviewed journals to prevent non-publication, which is often attributed to lack of time or low priority. (Fujian Song, Loke, and Hooper 2014)
Compare study protocols to final publications to detect and mitigate the impact of outcome reporting bias, which has been found to increase the likelihood of statistically significant results being fully reported by a factor of 2.2 to 4.7. (Dwan et al. 2013)
Carefully consider and address potential publication bias when interpreting the results of meta-analyses, particularly if statistically significant outcomes are overrepresented in the sample of studies included. (Kicinski 2013)
Proactively register your studies prospectively, search thoroughly for unpublished studies, and consider using statistical methods to assess and mitigate publication bias in your analyses. (F. Song et al. 2010)
Compare your final trial publications to your initial protocols to ensure accurate and comprehensive reporting of outcomes, as evidence indicates that statistically significant outcomes have higher odds of being fully reported compared to non-significant ones, leading to biased estimates of treatment effects. (Dwan et al. 2008)
Employ rigorous and transparent methods to identify, select, extract, and analyze data in order to minimize various forms of bias in systematic reviews, such as publication bias, language bias, and time-lag bias. (Tricco et al. 2008)
Be aware of and attempt to correct for potential biases leading to an excess of statistically significant findings in a body of evidence, such as publication bias, selective analyses, and selective outcome reporting. (J. P. Ioannidis and Trikalinos 2007)
Critically appraise the control of bias in individual trials, as the influence of different components like adequate randomization, blinding, and follow-up cannot be predicted. (Gluud 2006)
Report a comprehensive set of results rather than cherry-picking statistically significant or otherwise favorable findings, in order to minimize the risk of introducing publication bias in situ (PBIS) and ensure the validity of the overall body of literature. (Phillips 2004)
Be aware of the potential for the “winners curse” in scientific publication, whereby the most extreme and spectacular results may be preferentially published, potentially leading to overestimation and distortion of the true relationship being studied. (n.d.)
Carefully compare the characteristics of clinical trials reported in regulatory submissions to the FDA with those reported in published journal articles, as discrepancies in primary outcomes, statistical significance, and conclusions may indicate selective reporting or publication bias. (n.d.)
2.2 Replication Crisis
Carefully consider the trade-offs involved in choosing between pursuing novel but risky hypotheses (low prestudy probabilities) versus more reliable but less exciting ones (high prestudy probabilities), as well as the impact of statistical power and sample size on the reliability of published research. (Campbell 2022)
Be aware of the potential impact of researcher degrees of freedom or forking paths, which refers to the numerous ways that data can be analyzed and presented, leading to potentially spurious results. Therefore, it is recommended to pre-register analysis plans and avoid cherry-picking results based on post-hoc data exploration. (Gelman 2022)
Ensure strict separation between training and test data in predictive modeling, avoiding practices like imputing missing values using information from both datasets, reusing imputed datasets for training and testing, and using proxy variables for the target variable, as these can introduce data leakage and lead to overly optimistic performance claims. (Pineau et al. 2020)
Embrace variation and uncertainty, avoid the temptation to seek statistical significance as a definitive proof of an effect, and recognize the limitations of peer review and statistical significance testing in ensuring the accuracy of scientific findings. (Gelman 2018)
Ensure adequate statistical power to reduce the risk of false positives and effect size exaggeration, particularly in cognitive neuroscience where power tends to be lower than in psychology. (Szucs and Ioannidis 2017)
Prioritize collecting high-quality data through larger sample sizes, reducing measurement error, and employing within-person designs, while avoiding the pitfalls of null hypothesis significance testing and selectively reporting statistically significant results. (Gelman 2017)
Prioritize minimizing the rate of false positives and increasing the base rate of true hypotheses to enhance the reliability of scientific discovery, particularly through replication efforts. (McElreath and Smaldino 2015)
Consider implementing practices such as large-scale collaborative investigation, replication culture, registration, exchange, reproducibility practices, use of appropriate statistical methods, standardization of definitions and analysis techniques, stricter levels for claiming discoveries or successes, improved study design standards, better communication and dissemination systems, and increased training of scientific workforces in methodology and statistics to improve the reliability and effectiveness of your research. (J. P. A. Ioannidis and Khoury 2014)
Move beyond relying solely on p-values and instead incorporate multiple factors such as effect sizes, plausible mechanisms, and replication efforts to ensure robust and reliable conclusions. (Gaudart et al. 2014)
Avoid relying solely on statistical significance as a measure of the validity of your findings, as it provides limited information about the likelihood of an effect being real or not. Instead, researchers should consider other factors such as prior evidence and biological plausibility to support your conclusions. (Sullivan 2007)
Acknowledge and discuss the limitations of your work in a dedicated section, as this helps readers understand the validity and applicability of the findings, and ultimately contributes to the integrity and transparency of the scientific literature. (J. P. A. Ioannidis 2007)
Aim for multiple replications of statistically significant findings to improve the positive predictive value (PPV) of true relationships, particularly when the pre-study odds of a true relationship are low. (Moonesinghe, Khoury, and Janssens 2007)
Strive to make your work reproducible by creating and sharing a replication dataset containing all necessary information to reproduce your results, including raw data, codebooks, software, and analytic scripts. (King 1995)
Prioritize high-quality, well-powered studies with appropriate controls and careful consideration of potential sources of bias, as initial findings may be subject to the Proteus Phenomenon, where subsequent studies reveal smaller or even contradictory effects. (n.d.)
Prioritize conducting rigorous, unbiased studies that address important questions and utilize appropriate methods, including representative samples, valid measures, and robust analytic strategies, to minimize the likelihood of producing misleading or incorrect findings. (NA?)
Prioritize collaboration and transparent communication during the replication process, recognizing that transparency alone may not be sufficient to ensure reproducible results due to potential issues related to study design, data quality, and other contextual factors. (NA?)
2.3 Open Science
Prioritize open science practices such as open access, open data, preregistration, reproducible analyses, replications, and teaching open science to enhance the transparency, reproducibility, and credibility of your work. (Crüwell et al. 2019)
Consider using natural experiments, like the unexpected NIH-DuPont agreements that suddenly provided low-cost access to hundreds of genetically engineered mice, to identify causal relationships in observational data. (Murray et al. 2016)
2.4 Research Misconduct
Ensure transparency and accountability in your work by promptly addressing and disclosing any instances of noncompliance or misconduct discovered by regulatory bodies like the FDA, rather than allowing them to go unreported in the peer-reviewed literature. (Seife 2015)
Prioritize integrity in all stages of the scientific process, as even seemingly minor instances of misconduct such as redundant publication, failure to disclose conflicts of interest, or fabrication of data can compromise the validity and reliability of findings. (Williams 1997)
2.5 Peer Review
- Consider the tradeoff between the benefits of expertise and the risks of bias when selecting evaluators, as the authors demonstrate that evaluators with expertise in a specific field may have an informational advantage in separating good projects from bad, but they may also exhibit personal preferences that impact your objectivity, leading to a complex relationship where the benefits of expertise weakly dominate the costs of bias. (Li 2017)
2.6 Questionable Research Practices
- Consider the impact of publication bias, average power, and the ratio of true to false positives in the literature when interpreting the distribution of p-values, rather than jumping to conclusions about inflated Type 1 error rates due to questionable research practices. (Lakens 2015)
2.7 Reporting Guidelines
- Receive formal training in writing, use of reporting guidelines, and authorship issues to produce high-quality, transparent, and complete accounts of your research, enabling replication and utilization of results. (Moher and Altman 2015)
2.8 Scientific Skepticism
- Avoid misinterpreting p-values as the probability of the null hypothesis being true and recognize that the choice of statistical significance threshold is arbitrary and subject to debate. (Woolston 2015)
2.9 Systematic Review
- Conduct a thorough systematic review before embarking on new studies, ensuring that your work builds upon existing knowledge rather than duplicating efforts or missing crucial context provided by previous research. (Clarke 2004)
2.10 Transparency And Openness Promotion
- Utilize a consensus-based, comprehensive transparency checklist to ensure openness and accountability throughout the entire research process, including preregistration, methods, results and discussion, and data, code and materials availability. (Aczel et al. 2019)