What does consensus mean in science?       Global warming, fanatics & freedom       Future generations: climate victims or high-tech geniuses?       Let's lighten up       4 reasons to think twice                       About this website      Blog      Contact the creator of this website     HOME - back to the front page


report card     main findings     detailed findings     not-as-advertised     quality assurance     auditor list     press-release



Fairness, Quality Assurance & Minor Data Adjustments

PROCESS & PHILOSOPHY
Each of the 44 chapters in the 2007 IPCC report was audited by three individuals working independently. Auditors had no way of knowing who else was examining their particular chapter. They received a list of references cut-and-pasted from the online version of the IPCC report as an e-mail attachment. They were asked to highlight either the peer-reviewed or the non-peer-reviewed references, to count them, and to return the file.

A brief How-To Guide for this project was written by a citizen auditor who had already examined thousands of IPCC references in the course of conducting research for his blog. That guide advised citizen auditors to "Give the IPCC the benefit of the doubt." A project FAQ was authored by Donna Laframboise. Except for a couple of auditors who joined this project early on the first day, every auditor was directed to these documents (as well as to a sample chapter) when they initially signed on.

The IPCC chairman has declared that non-peer-reviewed research sources belong in the dustbin (see the last lines of this newspaper article) but this project does not necessarily take that position. Its primary goal was to determine whether the chairman's claim (frequently repeated by journalists) that this report is based only and solely on peer-reviewed literature is accurate.

UNCERTAIN REFERENCES
Citizen auditors were given the option of highlighting references about which they felt uncertain in an alternate color, thereby requesting a second opinion. Out of 132 audits, this mechanism was triggered by citizen auditors 32 times.

In the interests of consistency, Laframboise provided the second opinion in every case. Completed audits were returned in no particular order, and second-opinion assessment occurred in a similar fashion. Of the 316 references in which a second-opinion was sought, Laframboise ruled that 230 of them were peer-reviewed and 86 were not. Thus, she ruled in the IPCC's favor 73% of the time.

A note authored by Laframboise appears near the top of all audit files in which this mechanism was triggered.

MINOR DIFFERENCES IN RESULTS
Most chapters contain hundreds of references. The complete name of the publications and organizations involved is not always provided. Frequently, publication names are abbreviated in an inconsistent manner within an individual chapter - as well as across working groups. Thus, some amount of discretion was involved in decoding certain references, and in deciding whether or not they are peer-reviewed.

In five chapters, all auditors reported an identical peer-reviewed total. In 21 others, the auditors' results were within 1 or 2 percent of each other. In every case in which there was a difference of opinion, we used the number most favorable to the IPCC.

MINOR DATA ADJUSTMENTS
In six chapters out of 44, the span between the highest and lowest results initially reported by the auditors exceeded 5 percent. This prompted an examination of the audits by Laframboise who then asked six auditors to take a second look at their findings. In all but one case, Laframboise assured the auditors that the final decision was theirs. If they felt strongly that their initial result should stand, their totals would be respected.

In four cases, auditors who re-examined their audit felt the total percentage of peer-reviewed references should rise slightly. In one case, the auditor felt it should be slightly reduced. (This auditor then inquired about other files he'd worked on. On learning that there was one other instance in which there was some doubt, he re-examined his audit and adjusted his total slightly downward.)

In one final case, an auditor was asked to take a second look at his file but time constraints prevented him from doing so. Because this was the last audit outstanding in the entire project, Laframboise disregarded the results of that single anomalous audit and assigned the chapter to a fourth auditor.

 

see a spreadsheet of our findings here

    


Citizen Audit graphic                                                   IPCC graphic

   

 

 

 report card      main findings      detailed findings     not-as-advertised

     quality assurance     auditor list    press release