How do you spot a low-quality meta-analysis in the peptide literature?

3 replies 890 viewsmeta-analysisreviewsevidence-appraisal

sift_and_weighTrusted MemberOP

8 Jul 2026

Meta-analyses get treated as the top of the evidence pyramid, but in niche peptide areas I keep running into ones that pool tiny heterogeneous studies and produce a confident-looking forest plot out of basically nothing. Garbage in, tidy-looking garbage out.

What are your red flags? Mine so far: (1) pooling wildly different models/species, (2) huge heterogeneity (I2) that gets hand-waved, (3) obvious publication bias with no funnel plot, (4) including preprints and conference abstracts uncritically, (5) authors overlapping with the primary studies being pooled.

Would like to sharpen this checklist with people who read more of these than I do.

mara_labcoatVerified Expert

9 Jul 2026

Your list is already better than most published review sections. I'd add: check whether they pre-registered a protocol (PROSPERO or similar) or whether the inclusion criteria look retrofitted to the result. Post-hoc inclusion criteria are how you smuggle in the bias.

Also scrutinize how they handled risk-of-bias, a real meta-analysis grades each included study and does sensitivity analyses excluding the weak ones. If dropping the low-quality studies collapses the effect, that's your answer. And a forest plot pooling five species is a category error, not evidence.

napkin_statsRegistered Member

10 Jul 2026

The I2 hand-wave is my biggest pet peeve too. Once heterogeneity is high, the pooled point estimate is arguably meaningless and they should be explaining the variance, not averaging over it. I've started reading the funnel plot and risk-of-bias table before I even look at the headline effect size.

sift_and_weighTrusted Member

11 Jul 2026

Pre-registration and the 'drop the weak studies and see if the effect survives' test are great additions, adding both. Between this and the appraisal checklist in the pinned thread I think we've basically got a working rubric for the whole category now.

Not medical advice. Content on peptides.cx is an educational and community resource. It is not medical advice, diagnosis, treatment, prescribing guidance, dosing instruction, or emergency support. Always consult a qualified medical professional before making health-related decisions.