is a required field.
This paper describes a proposed reviewing process for simulation models (and eventually other content) in the Climate CoLab. We hope this paper will stimulate discussion that leads to agreement on a process to be tried.
Background
The Climate CoLab (Malone et al, 2009) lets users from the general public create and evaluate proposals for dealing with global climate change. These proposals include various quantitatively specified actions such as reducing carbon emissions by specified percentages in various regions of the world. The proposals also include computer simulation models for assessing the likely impacts of those actions. Impacts include changes in temperature, sea level, and various economic costs.
The current version of the CoLab contains a composite model that includes the C-LEARN model as its core and several other models for predicting additional economic and physical impacts. A complete list of the models currently included is here.
In the future, we hope to include other models in the CoLab. In some cases, these models may augment the existing composite model (e.g., to further disaggregate inputs or outputs). In other cases, these models may offer alternative ways of assessing the climate change impacts or augmenting the set of impacts considered.
Goals
We would now like the get expert reviews of the models currently in the system. We hope these initial reviews will also help establish a reviewing process that can be used for other models—and other parts of the CoLab—in the future.
The primary purpose of these reviews is to offer users of the CoLab a sense of how credible the models are. More precisely, the reviews should help a broad range of users (including members of the general public) understand the strengths, weaknesses, key assumptions, and key limitations of the models.
The secondary purpose of these reviews is to help improve the quality of the models included in the CoLab. For instance, the reviewers may suggest ways the existing models could be improved, or they may suggest additional models that could be added.
It is important to note that the purpose of the reviewing process here is somewhat different from traditional peer reviewing for academic journals. With academic reviewing, a key constraint is the space available in the journal, and the key decision is whether to accept or reject the article. Here, there is only a weak constraint on how many models can be included in the system, and the key decision is how much credibility to give the different models.
Proposed process
We don’t think that we (or anyone else) knows yet how best to do this kind of reviewing, but we propose to start with one method that we believe is promising and then to evolve our processes over time based on experience.
The initial method we propose to use is essentially to let the Expert Council members write the equivalent of a Wikipedia article about each model.
More specifically:
- Each model will have associated with it a separate wiki page containing an evaluation of that model.
- Anyone will be able to see these evaluation pages, but only Expert Council members (and Moderators) will be able to edit them.
- Just as with Wikipedia pages, any Expert Council member will be able to make any changes to the pages at any time, and the pages will keep changing as long as any experts want to keep making changes.
- This means that the resulting wiki pages will, essentially, be consensus statements about the strengths, weaknesses, key assumptions, and other salient points about the models. This does not mean that all experts have to agree about all aspects of the models. It just means that all the experts who care should agree that the strengths and weaknesses of the models have been described fairly.
- To aid in this process, we suggest that experts use the Neutral Point of View (NPOV) policy articulated by Wikipedia (http://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view). This policy gives detailed guidance for how to write articles that “represent fairly, proportionately, and as far as possible without bias, all significant views that have been published by reliable sources.”
- As in any wiki, the system will keep track of who made what changes, and this detailed history of changes will be visible to anyone who looks for it.
- All members of the Expert Council will be allowed to edit the wiki evaluation pages for all models.
Of course, we don’t know for sure how well this process will work, but Wikipedia’s success, using a similar process, suggests that this is a promising approach to try. In fact, at least one member of our advisory board has already suggested something similar in print (Christy, 2010).
Potential issues
This proposed process is different from traditional academic peer reviewing, and the comparison to traditional reviewing raises two particularly important issues:
(1) Should expert reviewers be anonymous?
| Advantages | Disadvantages |
* With anonymous reviews, reviewers may give their honest opinions more freely, since they need not fear reprisals from the people whose work is being reviewed (or others). | * Reviewers have less incentive to be careful, constructive, and fair when they are doing anonymous reviews. |
* The general public has less basis for confidence in the credibility of the reviewing process when reviews are anonymous. |
On balance, we believe the disadvantages of anonymous reviewing outweigh the advantages in this situation, so we propose experimenting initially with non-anonymous reviewing. If necessary, we will experiment later with anonymous reviews, perhaps on a limited, as-needed basis.
(2) Should expert reviewers be excluded from reviewing their own work (or work that competes with theirs)?
| Advantages | Disadvantages |
* Reviewers clearly have biases about their own work and work that competes with theirs. Excluding them may make the reviews less biased. | * People who have done the work being reviewed are far more knowledgeable about the work than others. They are also usually more knowledgeable about similar or competing work compared to reviewers with no experience in the same field. |
* It seems plausible that the transparent wiki-based process proposed above will provide built-in mechanisms for removing biases. For instance, if the creator of a model makes inappropriately positive claims about his or her own work, other reviewers will be motivated to remove or modify these claims. |
On balance, we believe the disadvantages of excluding experts from reviewing their own or competing work outweigh the advantages, so we propose experimenting with including such “interested parties” in the reviewing process. If experience shows that this causes too many problems, we can revise this policy later.
Conclusion
The reviewing process proposed here takes advantage of new information technologies (specifically wiki-based websites) to do things that would have been impractical in the paper-based world for which traditional peer review processes were developed. The experience of systems like Wikipedia suggests that this new approach may work surprisingly well. We believe it is worth trying.
References
Christy, J. R. “Open debate: Wikipedia-style.” In “Opinion: IPCC: cherish it, tweak it or scrap it?” Nature, 463, 730-732 (11 February 2010). (http://www.nature.com/nature/journal/v463/n7282/full/463730a.html)
Malone, T. W., Laubacher, R., Introne, J., Klein, M., Abelson, H., Sterman, J., & Olson, G. The Climate Collaboratorium: Project Overview. MIT Center for Collective Intelligence Working Paper No. 2009-003, September 2009. (http://cci.mit.edu/publications/CCIwp2009-03.pdf)
