Fascinating, Long, Quite Technical Discussion of The Temperature Data Problem

Because ClimateAudit.org is still down (apparently a Denial of Service attack by global warming true believers), Steve McIntyre has a very long, detailed, and important essay at Watts Up With That? about the significance of the problems that NASA has now admitted with their temperature data. It is simply too long and careful to summarize simply, so I'll just excerpt a few paragraphs to encourage you to read it in full:

There’s been quite a bit of publicity about Hansen’s Y2K error and the change in the U.S. leaderboard (by which 1934 is the new warmest U.S. year) in the right-wing blogosphere. In contrast, realclimate has dismissed it a triviality and the climate blogosphere is doing its best to ignore the matter entirely.My own view has been that matter is certainly not the triviality that Gavin Schmidt would have you believe, but neither is it any magic bullet. I think that the point is significant for reasons that have mostly eluded commentators on both sides.

...

The Hansen error is far from trivial at the level of individual stations. Grand Canyon was one of the stations previously discussed at climateaudit.org in connection with Tucson urban heat island. In this case, the Hansen error was about 0.5 deg C. Some discrepancies are 1 deg C or higher.

[graphs deleted]

Not all station errors lead to positive steps. There is a bimodal distribution of errors reported earlier at CA here , with many stations having negative steps. There is a positive skew so that the impact of the step error is about 0.15 deg C according to Hansen. However, as you can see from the distribution, the impact on the majority of stations is substantially higher than 0.15 deg. For users of information regarding individual stations, the changes may be highly relevant.
GISS recognized that the error had a significant impact on individual stations and took rapid steps to revise their station data (and indeed the form of their revision seems far from ideal indicating the haste of their revision.) GISS failed to provide any explicit notice or warning on their station data webpage that the data had been changed, or an explicit notice to users who had downloaded data or graphs in the past that there had been significant changes to many U.S. series. This obligation existed regardless of any impact on world totals.
...

GISS has emphasized recently that the U.S. constitutes only 2% of global land surface, arguing that the impact of the error is negligible on the global averagel. While this may be so for users of the GISS global average, U.S. HCN stations constitute about 50% of active (with values in 2004 or later) stations in the GISS network (as shown below).

...

Now my original interest in GISS adjustments did not arise abstractly, but in the context of surface station quality. Climatological stations are supposed to meet a variety of quality standards, including the relatively undemanding requirement of being 100 feet (30 meters) from paved surfaces. Anthony Watts and volunteers of surfacestations.org have documented one defective site after another, including a weather station in a parking lot at the University of Arizona where MBH coauthor Malcolm Hughes is employed, shown below.

[picture deleted]

These revelations resulted in a variety of aggressive counter-attacks in the climate blogosphere, many of which argued that, while these individual sites may be contaminated, the “expert” software at GISS and NOAA could fix these problems, as, for example here .

they [NOAA and/or GISS] can “fix” the problem with math and adjustments to the temperature record.

or here:

This assumes that contaminating influences can’t be and aren’t being removed analytically.. I haven’t seen anyone saying such influences shouldn’t be removed from the analysis. However I do see professionals saying “we’ve done it”

“Fixing” bad data with software is by no means an easy thing to do (as witness Mann’s unreported modification of principal components methodology on tree ring networks.) The GISS adjustment schemes (despite protestations from Schmidt that they are “clearly outlined”) are not at all easy to replicate using the existing opaque descriptions. For example, there is nothing in the methodological description that hints at the change in data provenance before and after 2000 that caused the Hansen error. Because many sites are affected by climate change, a general urban heat island effect and local microsite changes, adjustment for heat island effects and local microsite changes raises some complicated statistical questions, that are nowhere discussed in the underlying references (Hansen et al 1999, 2001). In particular, the adjustment methods are not techniques that can be looked up in statistical literature, where their properties and biases might be discerned. They are rather ad hoc and local techniques that may or may not be equal to the task of “fixing” the bad data.
Making readers run the gauntlet of trying to guess the precise data sets and precise methodologies obviously makes it very difficult to achieve any assessment of the statistical properties. In order to test the GISS adjustments, I requested that GISS provide me with details on their adjustment code. They refused. Nevertheless, there are enough different versions of U.S. station data (USHCN raw, USHCN time-of-observation adjusted, USHCN adjusted, GHCN raw, GHCN adjusted) that one can compare GISS raw and GISS adjusted data to other versions to get some idea of what they did.

I've been prepared to believe that most of the scientists involved in this work were pretty honest and serious in their pursuit of truth, and that the politician sorts at the IPCC were at fault. This last paragraph I quote above makes me more and more inclined to suspect that I have been too charitable.

When you aren't prepared to share data or algorithms that form the basis of your published papers, it's usually because you know full well that others will rip your claims apart.

UPDATE: Over here is an animated GIF that claims to show the effects of this correction of NASA's U.S. temperature data. It isn't a huge difference--but you can easily see it--and from what I've read, about 50% of all the weather stations that supply the global temperature data used to prove global warming are the U.S. set. In addition, I found a very interesting comment that makes the following claim (which I don't know if it is correct or not):

A paper / article written by Vincent R. Gray updated in 2003 said "Examination of the data shows that almost all of the 1901-1996 temperature rise for Russia/Soviet Union took place in one year, 1987 to 1988." This was about 3 years before the collapse of the Soviet Union, but two years after Perestroika and Glasnost.

Gray says in another paper "Although some Russian stations have excellent records over a very long time, the service has deteriorated in recent years, together with the rest of the Russian economy. In 1988 there were 244 temperature stations, but in 1989 135 were closed; mainly the smaller ones, leaving only 109 stations. Most of the 91 5°x5° grids in Russia/Siberia in Figure 2 will be represented by single stations. Recent monthly records from Russian stations show many gaps and doubtful figures." In the same paper he notes "Monthly temperature records for the Russian stations show an extreme temperature range of around 60°C. Early measurements are likely to have been in primitive or deprived conditions. Stations would have been operated by political prisoners."

The papers (or articles) mentioned above can be found on www.john-daly.com. At one point on that same site I recall seeing an interesting hypothesis that temperature readings from Siberia pre-Soviet collapse should be suspect because officials had a strong economic incentive to report lower-than-actual temperatures: heating oil subsidies.

So here we have an interesting artifact in Russian / Soviet temperature data that presents itself as a discontinuity not unlike Steve McIntyre's "Y2K bug". In this case the discontinuity may have an economic driver behind it, not a mathematical error. Or the discontinuity may truly be due to climate change.

The paper to which this commenter refers is here:

A feature of the results is the large temperature increase in the former Imperial Russia/ Soviet Union (+1.23°C), more than double the change in Western Europe (+0.5°C) or the USA (+0.41°C). This large temperature rise in Russia/Siberia by so many stations that were regarded by Peterson et al (1999) as predominantly “rural”, casts doubt on their assumption that the effects of local heating in rural stations are negligible. Removal of the Russia/Siberia set from their analysis would surely show a significant urbanisation effect from cities in the rest of the world. This widespread local heating around surface measurement stations would explain the differences between the surface temperature record and temperature measurements in the lower troposphere by satellites., and so the major human influence on the climate.

If this Russian data is significantly wrong, it blows out a major component of the AGW claims about the Urban Heat Island (UHI) effect not biasing the data of urban weather stations.