PROCESS & PHILOSOPHY
A brief How-To Guide for this project was written by a citizen auditor who had already examined thousands of IPCC references in the course of conducting research for his blog. That guide advised citizen auditors to "Give the IPCC the benefit of the doubt." A project FAQ was authored by Donna Laframboise. Except for a couple of auditors who joined this project early on the first day, every auditor was directed to these documents (as well as to a sample chapter) when they initially signed on.
The IPCC chairman has declared that non-peer-reviewed research sources belong in the dustbin (see the last lines of this newspaper article) but this project does not necessarily take that position. Its primary goal was to determine whether the chairman's claim (frequently repeated by journalists) that this report is based only and solely on peer-reviewed literature is accurate.
In the interests of consistency, Laframboise provided the second opinion in every case. Completed audits were returned in no particular order, and second-opinion assessment occurred in a similar fashion. Of the 316 references in which a second-opinion was sought, Laframboise ruled that 230 of them were peer-reviewed and 86 were not. Thus, she ruled in the IPCC's favor 73% of the time.
A note authored by Laframboise appears near the top of all audit files in which this mechanism was triggered.
MINOR DIFFERENCES IN RESULTS
In five chapters, all auditors reported an identical peer-reviewed total. In 21 others, the auditors' results were within 1 or 2 percent of each other. In every case in which there was a difference of opinion, we used the number most favorable to the IPCC.
In four cases, auditors who re-examined their audit felt the total percentage of peer-reviewed references should rise slightly. In one case, the auditor felt it should be slightly reduced. (This auditor then inquired about other files he'd worked on. On learning that there was one other instance in which there was some doubt, he re-examined his audit and adjusted his total slightly downward.)
In one final case, an auditor was asked to take a second look at his file but time constraints prevented him from doing so. Because this was the last audit outstanding in the entire project, Laframboise disregarded the results of that single anomalous audit and assigned the chapter to a fourth auditor.