is a required field.
Purpose of reviews
The primary purpose of these reviews is to offer users of the Collaboratorium a sense of how credible the models are. More precisely, the reviews should help a broad range of users (including members of the general public) understand the strengths, weaknesses, key assumptions, and key limitations of the models.
The secondary purpose of these reviews is to help improve the quality of the models included in the Collaboratorium. For instance, the reviewers may suggest ways the existing models could be improved, or they may suggest additional models that could be added. While these comments are primarily targeted toward model developers, they may also be of interest to general users of the Collaboratorium.
Please note that the purpose of the reviewing process here is somewhat different from traditional peer reviewing for academic journals. Here the main decision is not whether to accept or reject the different models, but how much credibility to give them. For instance, readers of the reviews should be able to understand the uncertainty in the models’ predictions. Reviewers should feel free to be specific about how users should interpret and present the results they see (e.g. by noting that users can express “high confidence” in certain model results compared to others).
Review process
We don’t think that we (or anyone else) knows yet how best to do this kind of reviewing, but we are starting with one method that we believe is promising, and we will evolve our processes over time based on experience.
The initial reviewing method is essentially to let the Expert Council members write the equivalent of a Wikipedia article about each model.
More specifically:
- Each model has associated with it a separate wiki page containing an evaluation of that model.
- Anyone can see these evaluation pages, but only Expert Council members (and Moderators) can edit them.
- Just as with Wikipedia pages, any Expert Council member can make any changes to the pages at any time, and the pages will keep changing as long as any experts want to keep making changes.
- This means that the resulting wiki pages will, essentially, be consensus statements about the strengths, weaknesses, key assumptions, and other salient points about the models. This does not mean that all experts have to agree about all aspects of the models. It just means that all the experts who care should agree that the strengths and weaknesses of the models have been described fairly.
- To aid in this process, we suggest that experts use the Neutral Point of View (NPOV) policy articulated by Wikipedia. This policy gives detailed guidance for how to write articles that “represent fairly, proportionately, and as far as possible without bias, all significant views that have been published by reliable sources.”
- As in any wiki, the system keeps track of who made what changes, and this detailed history of changes will be visible to anyone who looks for it. This history also allows people to add short comments summarizing the contents of or reasons for specific changes. Expert Council members are encouraged to add this type of comment when they make edits (see example of an edit history).
- All members of the Expert Council will be allowed to edit the wiki evaluation pages for all models.
Expert Council members can follow this link to proceed to review tasks.
Additional information
- Model reviewing process
- Editing wiki pages
