Connections between salt and heart disease or passive smoking and lung cancer are discovered in one study only to be disproved by another; and claims made for wonder drugs are routinely disappointed in the real world. Too much research seems to be based on flawed statistics. What's going on?by Robert Matthews / November 20, 1998 / Leave a comment
It’s a scary world out there: breathing in other people’s cigarette smoke, eating too much salt, living near power lines-all have a stack of scientific studies linking them to life-threatening diseases, from cancer to heart disease. Just as well, then, that there is also an impressive list of studies of these same risks finding no links at all. In any case, doctors now have a host of clinically proven wonder drugs to save our lives if we ever do fall prey to these diseases. But here is another funny thing: there is an equally impressive stack of evidence showing that these wonder drugs are anything but wonderful.
Surely these are statistical blips, the inevitable outcome of research into complex issues. Or are they? Last year, the British Medical Journal published a study by a team of cardiac specialists at Queen’s Medical Centre in Nottingham. Dr Nigel Brown and his colleagues had asked a simple question: how big an impact on heart attack survival have all those “clinically proven” wonder drugs had since the early 1980s? Trials of these drugs suggested that they would double survival rates. But what Brown found on the wards was disquieting. Back in 1982, the death rate on the wards was 20 per cent. Ten years later, it was the same. Somewhere between the clinical trials and the wards, the wonder drugs had lost their life-saving abilities.
A similar picture emerges from cancer research; so many therapies have failed to perform on the wards that talk of wonder drugs is now taboo. According to the US National Cancer Institute, despite all those “clinically proven” therapies that have emerged since the launch of President Nixon’s war on cancer in 1971, the overall survival prospects of patients have risen by only 4 per cent.
Researchers have tried to explain these cases of “the vanishing breakthrough.” Some blame the tabloid-like preference of academic journals for positive findings rather than boring refutations. Others cite dodgy experimental methods, a failure to rule out other causes and over-reliance on data from tiny samples. But there is another culprit whose identity has been known within academic circles for decades. Warnings about its effects on the reliability of research findings have been made repeatedly since the 1960s. Yet scientists are doing nothing about them.
The warnings focus on techniques that lie at the very heart of modern science: so-called “significance tests,” the statistical…