Lies, damn lies, statistics & meta-analysis – their contribution to the weak case for minimum pricing
Few days go by without the public being subjected to some health scare or miracle cure delivered to them by an ill-informed but very enthusiastic media. Generally, these stories refer to “a new study” or “latest research” implying that “scientists”, “doctors” or “experts” have actually performed a novel experiment that has scientifically demonstrated something new and potentially useful.
In the vast majority of cases, this isn’t true. What has often happened is that a special interest group has reviewed some historical data, re-analysed it, applied a bit of spin in support of their case, published it somewhere not over insistent on scientific rigour such as a medical journal and issued a press release usually full of impressive sounding numbers.
These reviews appear in many formats but all suffer from the fundamental problem that they tend to conclude whatever the authors want them to. The most advanced form of this data manipulation epidemic is the meta-analysis, which can be viewed as a sort of amplifier. The idea is to take a number of studies that may be individually insignificant or even contradictory and combine them in a way that enhances consistencies.
Two major limitations of the approach as identified by numerous academic sources are publication bias and agenda bias. These factors are especially problematic in public health which is observably doctrinaire.
Publication bias normally refers to the tendency for positive results to be more likely to be published than those that support the null hypothesis thus distorting collective analysis of outcomes. Public health publications are often policy driven rather than objective or evidence based and the dogmatic nature of this approach fuels a more extreme form of publication bias caused by suppression of non-conformist ideas.
The depth of this problem was exposed in 2003 when the BMJ published Enstrom and Kabat whose work suggested that passive smoking appeared to be less lethal than previously claimed. The authors were set upon by the public health industry and The BMJ itself was subject to attack for its heretical challenge to public health orthodoxy. The vast majority of critics didn’t even address the content of the paper. The unsavoury incident led Ungar and Bray to write Silencing Science in which they conclude that an intelligent debate on the effects of passive smoke has become impossible. Irrespective of the debate over its content, the reception of the BMJ paper serves to illustrate the extreme extent of publication bias in public health.
Public health also suffers from agenda bias. The bedrock of science is sceptical objectivity and this is particularly important with meta-analysis because freedom to choose which studies to include, how to weight them and how to interpret the results introduces a degree of subjectivity. In science the tendency to use this freedom to engineer favourable outcomes is usually offset by the value placed on scientific /academic integrity. Such ethical considerations are less restraining in public health where the discourse is dominated by policy driven orthodoxy rather than a desire for genuine discovery and the adversarial exploration of competing hypotheses.
In public health the authors of meta-analyses are all too often wishful thinking medics or public health activists who exhibit a depressing tendency to make the data fit the theory. Attempts to justify smoking bans by claiming dramatic post-ban falls in heart attacks have unsurprisingly produced some of the most unconvincing meta-analyses including this from Stanton Glantz a man obsessed by his personal war against tobacco and this produced by a cardiologist from Kansas
The outcomes of these meta-analyses are pre-determined by biased study selection. The authors chose to ignore the poor quality, methodological weaknesses and arguably fraudulent nature of the selected studies highlighting another weakness of meta-analysis. The authors should have been exposed by the “success” of their efforts which imply that 15-20% of heart attacks are caused by passive smoking. This is implausible to the point of being ridiculous but amazingly, their output is still referenced.
The dubious use of meta-analysis is not confined to tobacco control. Many of us have been left scratching our heads by claims made for minimum alcohol pricing. The notion that a modest financial measure that would not inconvenience the majority could have a significant impact on problem drinking and youth drinking appears counterintuitive and depends on some odd assumptions about price elasticity and behaviour.
The politician’s claims are based on the work of Petra Meier who derives much of her theory from meta-analyses. Based on Gallet’s 2007 interpretation of 132 studies dating back to 1945 she concludes “if the price of beer is raised by 10%, beer consumption would fall by 3.5%; if the price of wine was increased by 10%, wine consumption would fall by 6.8%; and if the price of spirits increased by 10%, spirits consumption would fall by 9.8%.”
In some ivory towered fairyland perhaps but in real life here in the UK, the leap from Gallet’s findings to “A 50p limit should cut alcohol consumption among moderate drinkers by about 3.5%, or half a unit for women and two-thirds of a unit for men” together with rest of the hyperbolic nonsense in this fabulously biased BBC article is hard to understand or justify.
I can imagine the Scottish government falling for this on the basis that any data however ridiculous is an improvement when you are used to your health secretary simply making up the numbers to support her agenda, but surely David Cameron should be better advised than this?
Reading Craig Gallet’s 2007 paper on which Meier relies heavily and being mindful of her claims with respect to targeting young drinkers, I was struck by a line in the conclusion:
“… if we are particularly concerned with teenage drinking, since we find that teens are least responsive to price, then perhaps the best approach to reducing teen alcohol consumption should involve alternatives to taxation, such as education campaigns.”
This is not the only inconsistency in this classic case of torturing the numbers to fit pre-determined policy.
By Chris Oakley. Chris has previously posted on Liberal Vision: Smokers-State Aprroved hate and Intolerance is UK policy, Alcohol is Old News – Minimum Pricing for Digestives is the “Next Logical Step” , Soviet Style Alcohol Suppression Campaign Called for By Public Health Activists , Alcohol Taxation: The truth, the whole truth and nothing but the truth , A Liberal Tolerant nation? , What hope is there for liberty if truth becomes the plaything of political lobbyists and Public Health Success?
Note from Editor
You might also find the following posts interesting:
BBC And Guardian Played Like Fools On Minimum Alcohol Pricing by Dick Puddlecote 3/10/2012
Lies, Damn lies and Sheffield University by The Pub Curmudgeon 3/10/2012