By Elizabeth Akincilar-Rummer
How many times have you heard a journalist say something like, “Researchers report a significant breakthrough…”, “Groundbreaking research shows… “, or “According to a new scientific study…”? The news constantly bombards us with “research” or “studies” to provoke an emotional response, whether it’s fear, excitement, or surprise. The point is, to catch your attention. Here are some of my favs, Interacting with Women Makes Men Stupid, Male Science Nerds are More Likely to be Virgins, and Smelling Farts Can Prevent Cancer. However, occasionally, there is decent research that provokes a huge emotional response. For example, a recent article published in Cosmo detailed some very disturbing stats about postpartum women that really caused some waves in social media. Check out Stephanie’s blog post for more details.
The real question is, how much should we believe in the research that we hear on the news or read on the internet? Unfortunately, much of what we read and hear is not exactly truthful. Often, results are manipulated by researchers who are trying to prove a point or promote a product, or even more often, the results are misinterpreted by those who are reporting on it to get the attention of their listeners or readers.
Press releases are often to blame for misinterpreting the results of recent research. To clarify, press releases are some sort of communication, usually written, that are directed to news media folks for the purpose of announcing something that seems very newsworthy. So, what ends up often happening is a journalist reads research released by an academic institution or a company, and picks up on one component of the study that sounds particularly compelling. They may focus on a subgroup within the study without providing proper explanation, or they may highlight tiny short term studies and miss the big picture. Many times these press releases are accepted without much scrutiny and these exaggerated notices are catapulted into full fledged news stories. As you can imagine, this can be dangerous. Much of the public assumes what is reported in the news is, in fact, true, and may change their behavior or beliefs as a result. This is particularly dangerous when medical research is misinterpreted. For example, in late 2015 the nonprofit Translational Genomics Research Institute (TGen) issued a press release claiming a breakthrough in treatment for glioblastoma multiforme, a rare but often deadly tumor that occurs in the brain and central nervous system. They even stated that this amazing drug had already been FDA approved. As you can imagine, this gave incredible hope to those suffering with this terrible disease. Unfortunately, after further investigation, the drug was approved on dogs, not humans, and while it was experimentally used in patients with Alzheimer’s disease and other illnesses, it had not been tested on people with glioblastoma. Unfortunately, this is one of many examples of medical research that was incorrectly reported in the media. Additionally, many times press releases will prematurely report on medical research before it has been peer-reviewed, published, or even finished! These details are usually in the small print at the end of the press release, therefore often ignored.
Knowing that research is exaggerated or misinterpreted by the media is not all that surprising for many of us. Better judgement tells us to question many things said in the media. However, what about “real” research? I’m talking about the research that one would find on PubMed, a search engine accessing a database of abstracts and references for most medical research. This is research from academic institutions and research organizations, primarily. Can we trust the majority of that research? Sorry to disappoint you, but the consensus is, a big resounding NO. Dr. John Ioannidis has spent his career proving that most published research findings are false. Yes, you heard that right, most research is garbage. In fact, a whopping 80% of nonrandomized studies, which are most studies, turn out to have false results! Even randomized studies, the gold standard, are wrong 25% of the time! He explains that these findings are due to 3 primary causes: researchers were frequently manipulating data analyses, they were utilizing their findings to advance their careers rather than report good science, and lastly, they were using the peer review process, a process in which journals ask researchers to review studies to decide what to publish, to suppress opposing views. This news may be very disturbing for many of us, especially those of us in the medical field. As medical providers, we put a lot of faith in the words, “research shows.” If we can’t trust research, what do we look to for guidance as medical providers? How do we justify what we’re preaching and practicing? Well, prepare to be disappointed.
For nearly 20 years it has been widely known in the field of research that most studies are enormously flawed. In Dr. Ioannidis’s paper, Why Most Published Research Findings Are False, the most downloaded paper in PLOS Medicine, he explains exactly why most studies’ results are not trustworthy. Expecting some backlash from the research community, Dr. Ioannidis took it one step further. In a paper published in the Journal of the American Medical Association, he examined 49 of the most highly regarded research findings in medicine over the previous 13 years. These articles included hormone replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease and a daily dose of aspirin to control blood pressure and prevent heart attacks and strokes. Upon review of these 49 studies, 45 of them claimed to have found effective medical interventions. Of these 45, 34 of them have been retested and 14 of these, or 41%, have been shown to be wrong or significantly exaggerated. These are the cream of the crop studies we’re talking about!
The funny thing is, when Dr. Ioannidis presents his findings at medical meetings all over the world, he is not met with resistance or anger. In fact, most researchers and medical professionals are not surprised at all.
So, do we abandon research? Should we stop believing what we’re told by leading research institutions? Do we stop listening to the recommendations of our medical providers that are basing their recommendations on the latest research? The answer is, no. Medical anarchy is not the solution.
How do we recognize quality research? When should we trust the results of studies? How do we filter all the garbage that is reported in the media? How do we analyze and synthesize research findings we read or are told about?
The short answer is, not easily. As Benjamin Franklin said, “Believe none of what you hear, and only half of what you see.” However, there are some basic components about research that will help you to determine if it’s worth paying attention to. Here’s a cheat sheet.
- Be sure that the study was indeed published. Then, scrutinize where it was published. To even consider the results of a study, it must be published in a peer-reviewed publication.
- Consider the impact ranking of the journal. Basically, the more prestigious the journal, the more you can trust what’s in it.
- Be suspicious of any study that is sponsored by a group that would benefit from the results turning out as they did. The greater the financial and other interests and prejudices, the less likely the findings are true.
- The bigger the claim, the more scrutiny it likely deserves. Groundbreaking results are sexy. They attract more attention, therefore the media is more likely to pick it up and misrepresent it.
- A study with a small sample size (less than 500 subjects) will not be powerful enough to support a claim. In order to get significant results that could actually affect medical diagnostics and/or treatment, you need a large number of subjects.
- If it sounds too good to be true, it likely is. Research that is built on an already existing body of evidence is more trustworthy than results that completely refute it.
- If the effect size is small, the findings are likely not true. Even if a study shows a statistically significant result, if the effect size is very weak, the results are essentially meaningless.
- Look for studies that are replicating the results of previously performed studies, or a systematic review of a number of studies. Basically, studies of studies are more trustworthy.
What does this mean for our community of pelvic health practitioners and people suffering with pelvic floor disorders? Truth be told, we have a long way to go to have a solid base of good research from which to make well informed diagnostic, assessment and intervention decisions. In general, pelvic pain is considered a young field in medicine. Therefore, we are still lacking a lot of research. However, we are gaining speed. If you do a quick literature search on PubMed, searching “pelvic pain” from 1996-2006, it yields approximately 5,200 studies. However if you complete the same search from 2006-2016, over 8,400 studies pop up. There has been a much bigger push for high quality research concerning pelvic pain in the last 10 years. Groups such as the MAPP (Multidisciplinary Approach to the Study of Pelvic Pain) Research Network, formed in 2008, are one of the larger organizations working hard to provide us with this much needed research. Their goal is to better understand the underlying causes of the two most prominent chronic urological pain disorders, interstitial cystitis/painful bladder syndrome and chronic prostatitis/chronic pelvic pain syndrome. The UNC Pelvic Pain Research Center is another organization leading research in pelvic pain conditions in women, such as vulvodynia and endometriosis.
Over the brief 15 years I have been in this field, I have seen enormous positive changes in the diagnosis and treatment of pelvic floor disorders. We’ve made many mistakes, but fortunately, have learned from those mistakes. The current research that exists has helped us recognize our errors and guide us to more effective treatment strategies for this patient population. I eagerly await the research that is yet to come that will again, show me a better way to be a more effective practitioner so I can help my patients live a pain-free, functional life.