Selecting Semi-Finalists
Semi-Finalist selection: Sector contests
First review: Regional contests
Semi-Finalist selection: Global contest
Semi-Finalist selection: Sector contests
Includes:
These steps are detailed in this 15 minute video, as well as in the text guide below. Both sources of information are complementary.
A - Screen proposals.
One or more Fellows review all contest entries and eliminate incomplete proposals or ones that do not address the contest prompt. You do this by logging into your contest page and opening the new Screening tab.
What is “incomplete”?? No text or only a few lines of text. If only a summary, it may be worth sending to Judges if it’s an interesting idea with potential.
Tip: Allow as many proposals as possible get reviewed by the Judges, while keeping it to approximately 10 proposals per Judge
When do I have to give comments? All proposals (whether they are incomplete, off-topic, advanced or not advanced), are provided comments. While there is no required length, we ask that comments are written professionally and in complete sentences. The intention is to give feedback on the proposal's strengths and/or constructive criticism on how the authors could improve their work. When giving feedback to the Semi-Finalists, let them know how they can best revise their proposals before the next round of Judging.
Please avoid harsh judgments and sharp comments – all proposal authors are passionate volunteers and we want to show respect for their time and effort.
All comments entered in the text field will be emailed to the authors as well as posted on the proposal's comment page.
B - Assign proposals to at least two Judges.
- The number next to the Judges' name will indicate how many proposals it has been assigned
- We recommend that Judges do not receive more than 10 proposals (unless they are willing to review more)
C - Before advancing Semi-Finalists, all Fellows independently rate the proposals that were sent to the Judges.
- This is for our research purposes. Your comments (but not your ratings) will be seen by other Fellows & Judges after all ratings have been submitted.
- Fellows do not need to rate proposals that were deemed incomplete or off-topic.
Once the screening is complete, the Judges will be able to log into their account and review their proposals from the contest page. Their assigned proposals will show up in red. After they have evaluated them, they will turn green.
A - Email the Judges and let them know:
- How to log into their account and see their assigned proposals (send them their username and password, if you have them)
- How to submit their ratings and comments
- A copy of the judging criteria (see below) and asking them to indicate in the comments whether or not they think the proposal should advance
- The date by when their review needs to be completed
Tip: Judges are encouraged but do not have to consider comments in the comment section and external links that detail more about the proposal. We recommend that Judges consider the proposal's content over the proposal's popularity; therefore, Judges do not need to consider the number of supporters or comments a proposal has.
Tip: Due date for the review of the proposals should be at least 2-3 days before the end of the Semi-Finalists selection period to give Fellows time to advance Semi-Finalists and provide feedback (see step 3).
B - Check that all Judges have completed reviewing the proposals. This stage will be complete when all (?)'s in the second column will be (checks).
Tip: If there is a (?) in the second column, open the Screening tab to see which Judge has not yet reviewed the proposal.
3. Advance Semi-Finalists & Provide Feedback
Go to the Judging Results tab and download the "Judges Rating as CSV" file (which can be opened in Excel) to view the average ratings and comments for all reviewed proposals. The CSV file is only a guide. Decisions to advance proposals should not be based sonely on numbers.
Tip: In the Excel file, organize Column E (“Judge”?) by average, in descending order, using the "funnel" tool in the Excel sheet.
One or more Fellows go through each reviewed proposal and submit whether or not it advances to the Semi-Finalist round. Form well-constructed feedback to each proposal author, whether or not they advance. (See more details about comments, below.)
This round of judging is complete when there are no (?)'s on the contest page. All saved emails and comments will be sent and posted simultaneously once the Climate CoLab staff confirms that the judging is complete. Emails and comments for Semi-Finalists will be posted in the Evaluation section of their proposals' webpage.
Judging Criteria
Proposals are rated on a 4-point scale on these 4 criteria:
- Feasibility of the actions proposed in the proposal. Judges with different kinds of expertise will evaluate the technical, economic, social, and political feasibility of the proposals.
- Novelty of the proposal's ideas. Innovative thinking and originality in a proposal will be valued more than encyclopedic knowledge. In addition, instead of selecting a roster of finalists that are very similar, judges will try to select a group of proposals that represent a diverse range of approaches.
- Impact on climate change (for example, for mitigation actions, the amount of greenhouse gas emission reductions or for adaptation actions, the extent to which the actions counteract the effects of climate change) and desirability of other impacts (e.g. economic, social, lifestyle)
- Presentation quality. Proposals that are well-presented will be favored over those that aren't. Presentation quality includes how well written a proposal is, how well it uses graphics or other visual elements, and how compelling are its artistic representations of possible future worlds (if any).
Timeline Guide
Fellows want to give Judges as much time as possible to review the proposals, while also giving themselves time to submit the judging results and edit the Judges' comments. You may consider a structure like this:
- Initial screening & assigning proposals - within 2 days after contest closes
- Review of proposals - 1.5 weeks
- Advancing Semi-Finalists & providing feedback - At least 2-3 days before Semi-Finalists are announced
During any part of this process, feel free to use me as your helpline. Email or call, even for small clarifying questions: 1-514-793-3300. I'm also on GChat: michael.houle3@gmail.com
Q&As
Q1. "Off-topic proposals"
If a proposal does not address your contest prompt but looks promising for another contest, please email me the link and I can move it manually. (Do not select "Advance" or "Do Not Advance".)
If it is off-topic and does not look promising for another contest, please select "Do Not Advance: off-topic" and it will not be considered for any judging on the Climate CoLab.
Q2. "Do I have to give comments?"
Comments are not required when Fellows are submitting their personal ratings.
Comments are required when communicating the decision to the author whether or not it will become a Semi-Finalist (i.e. in the text field that appears under "Comment to send to author" when the email autotext pops up.)
We also ask all Judges to submit comments when rating the proposals, so the Fellow can collect them and communicate them to the author.
Q3. "What happens to comments? Who sees them?"
The only comments seen by the author and by the public are those submitted in the text field that appears under "Comment to send to author". All other comments (from Judges and Fellows) will only be seen by other Judges and Fellows. Fellows are asked to take the Judges' comments and combine them to be sent to the authors.
Q4. "Do Judges receive an email when I assign them a proposal?"
No -- when you are ready for a Judge to review their proposals, please send her/him an email letting them know they are to log in to the site and review the proposals assigned to them (highlighted in red).
Q5. "What do I tell the Judges?"
There are two rounds of judging. The first is to select Semi-Finalists, the second is to select Finalists and the Judges Choice winner. The Fellows do an initial screening and will email the Judges when it is time for them to review their proposals. They do so by:
1 - logging into their Climate CoLab account and visiting the contest homepage
2 - reviewing every proposal assigned to them (which will be highlighted in red) and providing ratings and feedback at the bottom of the proposal page. Only the Fellows and Judges will see this feedback. Once a review is complete, the red will change to green.
(Note: the Judges do not have to review the training video to understand this process.)
After the Semi-Finalists are selected, this process repeats, however every Judge will review every Semi-Finalist proposal. They will select which become Finalists (those eligible for the Popular Choice award) and which will win the Judges Choice Award.
Q6. "My understanding is that all proposals are advanced, unless they are incomplete or off-topic. Is that correct?"
Yes. If you do this and have more than 10 proposals/Judge, you can choose to be more strict in your screening (see below).
Q7. "Do Fellows need to submit comments for proposals we are advancing?"
No. Comments are only required when communicating the decision to the author whether or not it will become a Semi-Finalist (i.e. in the text field that appears under "Comment to send to author" when the email autotext pops up).
Q8. "Our contest as 63 proposals and 6 judges. We haven't finalized which will be considered incomplete/off topic, but I am assuming we will still have more than 10 proposals per judge. Is that ok?"
As long as it is OK with your Judges. If only screening for those that are incomplete or off-topic means that there is an unworkable number of proposals per Judge, then Fellows may choose to be more strict in your screening. If you do this, select "No decision made yet" for those proposals for now and let me know -- I'll be in touch with next steps on how to mark these proposals.
First review: Regional contests
Includes:
Please note that, in addition to the 5 steps, it will be necessary to follow up with the Judges to make sure that you receive their feedback before August 7.
1.Screen Proposals- July 19
A - One or more Fellows review all contest entries and note any that are off-topic and/or empty. We hope to have as many ideas as possible reviewed by the Fellows & Judges, so please review proposals even if they are incomplete or only present a single idea.
If the proposal has some text but you are unsure if you should review it and assign it to the Judges, please email or call Michaël.
B - On July 19, Fellows post one of the two messages below on the Comment tab of the proposals that will not be reviewed by the Judges.
Below are some general messages that can be tailored for specific contests and recipients. Fellows will need to edit at least the parts highlighted in yellow to adapt the content to their contest.
| Message template for proposals that are off-topic: Thank you for your contest entry. We have looked at your proposal and found that it contained intriguing elements; however, it has not been reviewed in this first round because it does not address the contest prompt. We encourage you to review the contest resources page and to keep developing your idea before the August 31 deadline. You can also transfer your proposal to a more relevant contest or to the Proposal Workspace to re-open it, make edits, add collaborators, and even submit it into a future contest. You can do so by logging into your account, opening your proposal, selecting the Admin tab, and clicking “Move proposal”? or “Copy proposals”?. Reach out to us if you have any questions about how to complete your proposal. All the best, Additional comments from the Fellows: [COMMENTS] |
Below are some general messages that can be tailored for specific contests and recipients. Fellows will need to edit at least the parts highlighted in yellow to adapt the content to their contest.
| Message template for proposal that are incomplete: Thank you for your contest entry. We have looked at your proposal; however, it has not been reviewed in this first round because most sections are incomplete or empty. We encourage you to keep developing your idea until the submission deadline, August 31. Reach out to us if you have any questions about how to complete the proposal. All the best, Additional comments from the Fellows: [COMMENTS] |
2. Fellows’ Feedback - July 19-24
Fellows review and briefly summarize each of the sub-proposals contained in the plan proposal. They then give an overall statement to reflect on the strengths and weaknesses of the plan proposal as a whole.
The Judges will rely on these summarize to evaluate the content that is hidden behind the hyperlinks of sub-proposals.
All plan proposals are to be reviewed and summarized by at least 1 Fellow.
Steps
A - Open your contest Google folder and create a Google doc for each proposal to be reviewed
B - Copy/paste this template into each proposal's Google doc
Template link: https://docs.google.com/document/d/1bWtEwdiVsAAEoilLlNYXdyE_e1FcoUem_yY67o23Iyo/edit#heading=h.62pqpthr8q7s
C - Copy/paste the proposals' content into each new document. Be careful to keep all the sub-proposal hyperlinks in the text.
D - On the upper-right corner of each Google doc, click on "Share" and select "Anyone with the link can edit" option.
E - Fellows generate notes on each sub-proposal
Fellows highlight the link to a sub-proposal, hit the Comment button (upper-right corner of the Google document), and add the following information:
- 1-2 sentences: A short summary of the actions proposed
- 1-3 sentences: Was the sub-proposal selected as a Semi-Finalist, Finalist, or Winner in another contest? What were the Judges' impressions at the time? (You can see Judges’ comments in the sub-proposal’s Comment or the Evaluation tab)
- Your impression of this sub-proposal and how well it fits in the larger plan. If the sub-proposal has not been reviewed previously by any Judges, please review based on the 4 judging criteria.
F - Fellows write an overall statement for the entire plan proposal
- 1-paragraph (3-6 sentences): Reflect on how well the authors selected a combination of strong sub-proposals, and your impression of the full proposal. Note any overlapping or repetitive sub-proposals or concepts, and any gaps that have not been addressed.
3.Assign proposals to Judges-July 25
Fellows divide the proposals between the different Judges. Each proposal should be reviewed by at least 2 Judges, and each Judge should not receive more than 10 proposals.
Fellows email each Judge a list of proposals for them to review, with links to the Google document containing the Fellows' feedback.
Fellows will need to edit at least the parts highlighted in yellow to adapt the content to their contest.
| Dear [NAME OF THE JUDGE], We are delighted to inform you that the proposals are now ready for you to review. You have from today until August 6, midnight, Eastern Time, to make your assessments. Each proposal has its own Google document that contains the instructions, judging criteria, proposal content, and our feedback: [1. Name of proposal 1 (with hyperlink to Google doc) 2. Name of proposal 2 (with hyperlink to Google doc) ....] On August 6, we will start compiling your comments and will send them to the proposal authors. They will have the opportunity to revise their proposals before the selection of Finalists in September. If you have any questions on how to address certain proposals or questions about our feedback, please let us know. It will be our pleasure to help. We hope you will enjoy reviewing the proposals! Many thanks. Best, [SIGNATURE] |
4. Judges' Feedback-July 25 to August 6
Judges review the proposals and add to/revise the Fellows’ feedback directly on the Google document. This part of the process is all about being supportive. We will ask Judges to be more critical and start eliminating proposals during the Finalist selection.
Judges’ prompt:
- How well has the author selected strong ideas, and combined them to articulate a broad, coherent, and feasible vision for what the region in question can do about climate change?
- Improvements the author can make.
(This will be emailed to the Judges and included in the Google document template that you will use for each proposal.)
5. Post Feedback to Authors-August 7-10 (midnight)
A - One or more Fellows form well-constructed feedback for each proposal, based on the Judges’ comments.
B - Fellows post this final feedback on each proposal’s Comment tab before August 10, midnight. Please integrate the feedback to the email template below.
| Message template: Thank you for submitting your proposal to this Climate CoLab contest. Because you had submitted it before July 18th, the contest Judges were able to review your proposal and provide you with some feedback, which we have included below. We hope that you will use it to further develop your work before the August 31 deadline. On that date at midnight Eastern Time, your proposal will be locked and considered in final form. The Judges will then select which proposals will continue to the Finalists round. Finalists are eligible for the contest’s Judges Choice award, as well as for public voting to select the contest’s Popular Choice award. The Winners will receive a special invitation to attend selected sessions at MIT’s SOLVE conference and showcase their work before key constituents in a workshop the next day. A few select Climate CoLab winners will join distinguished SOLVE attendees in a highly collaborative problem-solving session. In addition, if your plan is included in one or more winning global plans, you will receive Climate CoLab Points, and the top point-getters will receive shares of a cash prize of $10,000.
Thank you for your great work and good luck! 2015 Climate CoLab Judges & Fellows
[Judges' feedback] |
Time Guideline
1. Screen proposals – July 19
2. Fellows’ Feedback – July 19-24
3. Assign proposals to Judges – July 25
4. Judges' Feedback – July 25 to August 6
5. Post feedback to authors - August 7-10 (midnight)
Judging Criteria
Feasibility of the actions proposed in the proposal. Judges with different kinds of expertise will evaluate the technical, economic, social, and political feasibility of the proposals.
Novelty of the proposal's ideas. Innovative thinking and originality in a proposal will be valued more than encyclopedic knowledge. In addition, instead of selecting a roster of finalists that are very similar, judges will try to select a group of proposals that represent a diverse range of approaches.
Impact on climate change (for example, for mitigation actions, the amount of greenhouse gas emission reductions or for adaptation actions, the extent to which the actions counteract the effects of climate change) and desirability of other impacts (e.g. economic, social, lifestyle)
Presentation quality. Proposals that are well-presented will be favored over those that aren't. Presentation quality includes how well written a proposal is, how well it uses graphics or other visual elements, and how compelling are its artistic representations of possible future worlds (if any).
During any part of this process, feel free to use me as your helpline. Email or call, even for small clarifying questions: michaelh@climatecolab.org - 1-514-793-3300
Semi-Finalist selection: Global contest
Calendar
Email Judges instructions - Oct. 15
Fellows’ feedback - Oct. 18-19
Email Judges proposals - Oct. 19
Judges’ evaluations - Oct. 19-24
Judges' call - Between Oct. 22-25
Advance Finalists & Provide Feedback - Oct. 24-25
Fellows follow up with the Judges to make sure that they receive their feedback by Oct 24.
1. Email Judges instructions - Oct. 14
Email template:
Please edit the parts highlighted in yellow
| Subject line: Preparing for the Semi-Finalist selection
Dear Judges,
The first evaluation period runs from October 19-24 (midnight EDT). This email contains a guide for this period. We know it is long, so if you have any questions, please reach out to us!
Background This global contest is part of a pilot project, in which we seek “integrated proposals”?: proposals that include a combination of singular ideas that together, articulate a plan or vision for what can be done around the world to adequately address climate change. For more detail, see https://www.climatecolab.org/community/-/blogs/now-open-build-a-global-climate-action-plan
Intention of the Semi-Finalist Selection Period To provide guidance to proposals that have potential. Any incomplete or off-topic proposals will be eliminated; all somewhat promising ideas can be advanced, with feedback.
Evaluation process The evaluation process will be conducted directly on the Climate CoLab's platform.
To evaluate the proposals, please:
*This written section is very important, because it provides the foundation for the feedback that we provide to each proposal author, whether or not they advance as a Semi-Finalist.
Evaluation criteria Semi-Finalist proposals: 1. are well presented and at least potentially feasible; 2. are especially strong in one or more of the 3 dimensions: feasibility, impact*, and novelty; 3. include a sensible combination of sub-proposals that each match the criteria above (1, 2); 4. have combined the sub-proposals (ideally regional proposals) and included additional explanations to articulate a comprehensive vision for what different actors around the whole world can collectively do about climate change.
FAQs: What if the proposal is not an integrated proposal? If a proposal is strong but does not include sub-proposals or create a comprehensive global plan, we will ask participants to revise their proposals during the revision period to fit the format.
What’s the impact tab? There is a team of MIT students (“Impact Assessment Fellows”?) who have worked with the proposal authors in some cases, who have estimated these proposals’ impact on greenhouse gas emissions. These assessments can be found on the Impact tab, and explanations will be posted by the IA Fellows on the Fellow Review tab. Judges are asked to consider these impact assessments as part of their evaluations of the potential impact of the proposals.
If you have questions about any part of this process, please ask us and cc Michael <michaelh@climatecolab.org> and Laur <lfi@mit.edu>.
We will let you know when the proposals are ready for your review!
Best, [SIGNATURE] |
In separate emails, please send the login information (username and password) to the Judges:
Please edit the parts highlighted in yellow
| Dear [NAME],
If possible, please try to log in before Oct. 16 and let us know if you are experiencing any difficulties. This way we can resolve any technical issues before the evaluation period.
Username: [Preety Bhandari: pbhandari Youba Sokona: ysokona Lin Li: linli Kelly Levin: klevin Robin Chase: rchase]
Password: The default password given to you by the Climate CoLab is climatechange (case sensitive, feel free to change).
Many thanks. Best, [SIGNATURE] |
2. Fellows’ feedback - Oct. 18-19
By Oct. 18, the Climate Colab team will:
· Screen the proposals
· Prepare the messages to be sent to the authors of incomplete or off-topic proposals
· Send Fellows the proposals that fulfill on the intention of the contest.
(We hope this will expedite the process and have the Fellows’ support the tasks for which their expertise is the most valuable.)
Oct. 18-19, by the Fellows:
· To enter your feedback in the system, please click the Fellow Review tab and enter your feedback in one or more comment boxes. Comments in this tab will only be visible by the Fellows and the Judges, and can be edited at anytime.
For each sub-proposal, please:
(1) Summarize each sub-proposal that is linked by the main proposal:
· 1-2 sentences: A short summary of the actions proposed
· 1-2 sentences: Was the sub-proposal selected as a Semi-Finalist, Finalist, or Winner in another contest? What were the Judges' impressions at the time? (You can see Judges’ comments in the sub-proposal’s Comment or the Evaluation tab) Is the sub-proposal a seed proposal (climate action plans entered by the regional Fellows)?
· 1-2 sentences: Your impression of this sub-proposal and how well it fits in the larger plan. If the sub-proposal has not been reviewed previously by any Judges, please review based on the 4 judging criteria.
Please clearly indicate to which parts of the proposal your comments relate.
(2) Write an overall statement for the entire plan proposal
· 1-paragraph (3-6 sentences): Reflect on how well the authors selected a combination of strong sub-proposals, and your impression of the full proposal. Note any overlapping or repetitive sub-proposals or concepts, and any gaps that have not been addressed.
· The structure is very important in order for the proposals to qualify for the Finalist round. If a proposal does not combine sub-proposals, please make sure to include some guidance in your feedback to the author. For example: “Your proposal does not combine a subset of proposals to form a larger strategy on how the whole world could move forward with climate change, as required for this contest. Please see the contest’s resources page for details. We strongly encourage you to bring the necessary changes to your proposal before the end of the Revision Period.”?
3. Email Judges proposals - Oct. 19
Email template:
Please edit the parts highlighted in yellow AND please send next to the instructions email sent by Gustya on Oct. 14.
| Subject line: Climate CoLab Proposals For Review
Dear Judges,
The time has come for you to evaluate the proposals, rate them, and select the Semi-Finalist(s). Semi-Finalists will have the opportunity to revise their proposals based on your feedback before the final round of evaluation (Nov. 5-16). In order to maximize the chances of having good proposals in the finalist round, we encourage you to advance any proposal that you think has a reasonable chance of being revised into a strong proposal. Please also give the authors any advice you have to help them strengthen the proposal. We very much hope that one or more of the proposals that emerge as Finalists will be good enough that knowledgeable people in the climate change field will think they are interesting and plausible.
Please carefully read the information on how to access your proposals and submit your evaluations and the evaluation criteria (see email below sent on Oct. 14). We thank you in advance for completing your reviews no later than Oct. 24. Please note that this deadline is not flexible; we are operating under a very tight schedule in order to be able to announce the Winner(s) at COP21!
Conference call: The "Semi-Finalist selection" call is scheduled for Saturday Oct. 24 at 7am Eastern Time. During this call, we will discuss the proposals you have reviewed and select the ones that should advance to the next round. If not everyone is available, some discussion can happen over email.
Please find, below, the agenda and the GoToMeeting contact information for the call.
Do not hesitate to reach out if you have any questions.
We are looking forward to the conversation! Best, [SIGNATURE]
Agenda Time: Sat, Oct 24, 2015 7:00 AM - 8:00 AM EDT Meeting led by: [Fellow] Purpose: To select the Semi-Finalists
7:00 - 7:10 : Brief welcome and introduction of the Judges on the call 7:10 - 7:50 : Discussion of the proposals strengths and weaknesses, and decision on the Semi-Finalist(s)
7:50 - 8:00 : Summarize results
Semi-Finalist Selection call - Contact information
Sat, Oct 24, 2015 7:00 AM - 8:00 AM EDT
|
4. Judges’ evaluations - Oct. 19-24
Fellows follow up with the Judges to make sure that they receive their feedback by Oct. 24.
Judges review the proposals that have been assigned to them by the Climate CoLab team.
To check the status of the Judges’ evaluation, visit the contest homepage. This stage will be complete when all (?)'s in the second column will be (checks). If there is a (?) in the second column, open the Screening tab to see which Judge has not yet reviewed the proposal. Next to the Judges' pictures, there will be either a red (X) or a green (checks).
5. Judges' call - Between Oct. 22-25
Prior to the call, download the "Judges Rating as CSV" file under the Judging Decision tab, and read the evaluations of the Judges.
Goals of call:
1. Select the Semi-Finalists*
2. Eliminate any proposals that are not to be considered as a Semi-Finalist
3. Create consensus on feedback to be given to proposals
*There is no limit to the number of Semi-Finalists selected by the Judges
Semi-Finalists will have the opportunity to revise their proposals based on the Fellows and Judges' feedback. They will then go through a second round of evaluation before being selected as Finalists. Finalists are proposals that are eligible for the Popular Choice award, which is determined by public voting, and for the Judges' Choice Award.
6. Advance Finalists & Provide Feedback - Oct. 24-25
One or more Fellows go through each proposal and submit whether or not the Judges decided to advance it to the Semi-Finalist round, by completing the following:
-
Submit Judges’ decisions: Click on the proposals’ Judges Decision tab. Select if the proposal will advance or not.
-
Provide feedback: Click on "Go To Email Template". Please copy/paste the relevant email template in the comment box under the Judging Decision tab. In addition, please form well-constructed feedback to each proposal author, whether or not this proposal advances as Semi-Finalist and add this at the end of the email template (Judges and Fellows' feedback should be presented separately). Hit the SAVE button.
This round of judging is complete when there are no (?)'s on the contest homepage.
All saved messages will be sent by email and posted (under the Evaluation tab of the proposals) simultaneously once the Climate CoLab staff confirms that the judging is complete.
Feel free to contact Michaël at anytime if you need additional support or if there is anything that slows the process.
