ClimateGate: the Oxburgh Review

 I normally pass up on absolutely any opportunity to comment on anything to do with climate change, but sometimes I just have to make an exception. And this is one of those occasions: Lord Oxburgh’s panel review, the second of three, of the so-called “ClimateGate affair”, the thousands of emails stolen from the Climate Research Unit of the University of East Anglia and subsequently published on the internet. Not because I endorse some of their conclusions on the general subject of Climate Change (as always I am staying out of that, and for good reason as you will see further on) but because in at least one area the panel absolutely hits the nail on the head and correctly begins to identify (at long last!) one of the two largest problems that is all-pervading in science, especially in the case of larger or “meta-studies”, but also in virtually every other study drawing heavily on “peer-reviewed published literature.

Richard Black, in a brief but excellent posting on his BBC blog site, takes three sections of the report which go to the heart of the problem and puts it together thus:

“ In essence it says: tell the whole story

"Their published work contains many cautions about the limitations of their data and their interpretation."

And a little further on:

"All of the published work was accompanied by detailed descriptions of uncertainties and accompanied by appropriate caveats. The same was true in face-to-face discussions."

And finally:

"CRU publications repeatedly emphasise the discrepancy between instrumental and tree-based proxy reconstructions of temperature during the late 20th Century, but presentations of this work by the IPCC and others have sometimes neglected to highlight this issue. While this was regrettable, we could find no such fault with the peer-reviewed papers we examined."”

I remember, many years ago, a small paper manufacturer in Kent (UK) which had a long-standing and annual contract with the Ministry of Defence to produce paper graph-recorder charts. In order to try and cut some costs, the lead paper technician, a friend of mine, proposed some infinitesimal small changes to the formula which were secretly tried without telling the client. The batch passed and was accepted by MoD, and production was continued with that small change. A few years later they did this again, another minor, trivial change, and again it was accepted. This went on three or four times over a number of years until… the batch was returned and the contract cancelled. Jobs lost and the company closed. The production run was not fit for purpose. What had happened? The last change was so minute, nobody understood why the paper was rejected. But of course it wasn’t the last change, or even each individual, any single one of the changes, but the incremental changes. Some may have added to the problem, some may have taken away from it for a bit, but the end result was eventually (but not inevitably, and that IS a problem): unacceptable.

Exactly the same problem occurs with meta-studies, studies heaping together masses of other studies and try and interpret the whole as one overarching study, using data from individual studies which on their own have lots of caveats and uncertainties spelled out, the famous or infamous widely accepted scientific benchmark of “p=<0.05” or less than a 5% chance that the conclusion is wrong. That means, as Paul Harremoes, probably and in my opinion the greatest environmental engineering guru of the mid to late 20^th century, once put it, that "of every 20 papers I read, statistically, one is likely to be wrong; problem is I don’t know which one". Add together thousands of papers and studies, and all those individual uncertainties become incremental, additive or subtracting, which may lead to something going hopelessly wrong at some point. Or maybe not. We don’t know. It all depends. The problem is we don’t really have an adequate (statistical) method for reducing the “uncertainty” of meta-studies to the same level, p=<0.05, of uncertainty (or caution) as we do for individual small studies, or even accurately determining what the level of uncertainty in a meta-study, drawing on such a variety and number of resources, including differences in methodology, as the global climate change study (but also other, smaller studies), is.

Computer models are not really helpful in that as they treat inputs, results and predictions in a fairly unique and different manner: we "calibrate" them to try and adjust initially predicted results to at least some of the observed results and then hopefully can claim "Eureka! it works". As Paul Harremoes again pointed out at the same conference in Leiden in the 1990's: "I see someone presenting a model, followed by two or three case studies showing that the outcome is as observed and therefore the model is "correct". What I don’t see is the twenty case studies where it didn’t, and which were used to "calibrate" the model". In other words what you and I would disparagingly term "fiddle factors" (which isn’t entirely fair but sometimes not far from the truth; I have written them myself like that on occasion): numerical adjustments inside the calculating part of the model, made to force the model towards the correct (i.e. observed) conclusion, but which do not necessarily have scientific value in themselves. Which is also why different models can and sometimes do come up with different, conflicting, conclusions and predictions

The problem we face immediately is not whether climate change exists or not, but that we have no adequate and rigorous methodology to "predict" with even the same degree of certainty or uncertainty the outcome of a meta study as we do with individual studies. And that is why I stay far from any giving any comment or opinion on climate change, other than: “climate change has always been happening anyway…, pass me the white wine please!”

That further simplification by the press and others doesn’t help is a given, but fundamentally the problem remains with our science and in particular meta-studies in the first instance. So that is where the blame for all the confusion and arguments lies and that is where it should be addressed.

Not by eking out small “wrong” details trying to prove “everything is hogwash”. Actually that is another peculiar but long-standing and all-pervading academic habit. As a friend of mine put it succinctly and accurately, “the prevailing attitude in academia, and especially as you move higher up the food-chain, is: “what matters is not so much whether I am right, what matters is that YOU ARE WRONG!”

And definitely not by blaming the media for our own scientific short comings and attitudes.