Lies, Half-Truths, and Statistics

How to Lie with Statistics: A Field Guide for GIE Reviewers

Written by Lyndon Hernandez, MD, MPH, FASGE, Chair of the GIE Editorial Review BoardDr. Lyndon Hernandez

Understanding the pitfalls of peer review is critical to the continued success of GIE as a high-quality journal. Yet, young, aspiring reviewers get little guidance on how to parse the methodology section of a paper. The peer review process is more of an art than science, so it is not surprising that inter-rater reliability between individual reviewers is low [1]. Statistics is one component of a manuscript where many reviewers falter. Contrary to the traditional approach of letting uninitiated reviewers wither on the vine, we summon all GIE reviewers to contribute to our collective understanding of faulty statistics, either by submitting tips or posting a comment to share untold stories. Herein is the first of a series of tips on how to bend the truth- for what better way to spot a lie than to learn the tricks of the trade?

Tip 1: Exclude Cases without Telling Your Readers

Commonly seen in: Phase I and II trials, particularly papers with an algorithm on patient enrollment that is suspiciously absent.

Notes: Excluding cases is a great way to inflate your treatment effect. You can hide the truth at the front end (example A) or at the back end (example B) of a study as we shall see in our examples.

Example A: In treating pancreatic cancer with Drug X, we see the effect of excluding 50 patients from the analysis (see table [2] below) can double the median survival. Voila! A study arm with excluded early deaths can appear superior when in fact there is no incremental benefit.

Elimination of early deaths
All cases analyzed < 1    month < 3 months < 6 months
# of patients 100 100 100 100
# analyzed 100 89 71 50
Median survival 6 mos. 7 mos. 9 mos. 12 mos.
Response rate 30% 34% 42% 60%

Example B: In a study on endoscopic Treatment Y for high-grade dysplasia of the esophagus, 20 patients entered the trial and all 20 completed their treatments.

  • 5 patients are lost to follow-up at 6 months after completion of treatments.
  • 15 followed beyond month 12
  • 12 of 15 respond
  • Is the success rate…

12/15 = 80%         or        12/20 = 60%  ?

The Solution: Require clear accounting of all patients. This problem is easily overlooked, so make this part of your mental checklist when you review (or design your own) clinical trials.


References:
[1] Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The validity of peerreview in a general medicine journal. PLoS One. 2011;6(7):e22475.
[2] Adapted from: Endpoints and how to lie with statistics, Ken Stanley, Ph.D. (with permission).
 

The information presented in Endoscopedia reflects the opinions of the authors and does not represent the position of the American Society for Gastrointestinal Endoscopy (ASGE). ASGE expressly disclaims any warranties or guarantees, expressed or implied, and is not liable for damages of any kind in connection with the material, information, or procedures set forth.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s