How to Lie with Statistics: A Field Guide for GIE Reviewers
Written by Lyndon Hernandez, MD, MPH, FASGE, Chair of the GIE Editorial Review Board
Understanding the pitfalls of peer review is critical to the continued success of GIE as a high-quality journal. Yet, young, aspiring reviewers get little guidance on how to parse the methodology section of a paper. The peer review process is more of an art than science, so it is not surprising that inter-rater reliability between individual reviewers is low . Statistics is one component of a manuscript where many reviewers falter. Contrary to the traditional approach of letting uninitiated reviewers wither on the vine, we summon all GIE reviewers to contribute to our collective understanding of faulty statistics, either by submitting tips or posting a comment to share untold stories. Herein is the first of a series of tips on how to bend the truth- for what better way to spot a lie than to learn the tricks of the trade?
Tip 1: Exclude Cases without Telling Your Readers
Commonly seen in: Phase I and II trials, particularly papers with an algorithm on patient enrollment that is suspiciously absent.
Notes: Excluding cases is a great way to inflate your treatment effect. You can hide the truth at the front end (example A) or at the back end (example B) of a study as we shall see in our examples.
Example A: In treating pancreatic cancer with Drug X, we see the effect of excluding 50 patients from the analysis (see table  below) can double the median survival. Voila! A study arm with excluded early deaths can appear superior when in fact there is no incremental benefit.
|Elimination of early deaths|
|All cases analyzed||< 1 month||< 3 months||< 6 months|
|# of patients||100||100||100||100|
|Median survival||6 mos.||7 mos.||9 mos.||12 mos.|
Example B: In a study on endoscopic Treatment Y for high-grade dysplasia of the esophagus, 20 patients entered the trial and all 20 completed their treatments.
- 5 patients are lost to follow-up at 6 months after completion of treatments.
- 15 followed beyond month 12
- 12 of 15 respond
- Is the success rate…
12/15 = 80% or 12/20 = 60% ?
The Solution: Require clear accounting of all patients. This problem is easily overlooked, so make this part of your mental checklist when you review (or design your own) clinical trials.
The information presented in Endoscopedia reflects the opinions of the authors and does not represent the position of the American Society for Gastrointestinal Endoscopy (ASGE). ASGE expressly disclaims any warranties or guarantees, expressed or implied, and is not liable for damages of any kind in connection with the material, information, or procedures set forth.