Use the wisdom of the MIT alumni crowd to forecast climate solutions by MIT Superforecasting
Pitch
The wisdom of crowd effect denotes the surprisingly accurate estimates that crowds can provide. The MIT crowd can be leveraged in forecasts.
Description
Summary
Since the early mathematical theorems of Condorcet and the empirical observations of Sir Francis Galton in 1906, the underlying principles of 'the wisdom of crowds' have been extensively replicated in a multitude of settings, and its dynamics and boundary conditions have been explicated (Surowiecki, 2004). Simply put, the wisdom of crowds denotes the surprisingly accurate estimates and predictions that crowds can provide, illustrating a form of collective intelligence rooted in ability and diversity (Page, 2007). Building on this spirit, Howe (2006) coined the term 'crowdsourcing' as a portemanteau combining the words 'crowd' and 'outsourcing' suggesting that problems can be outsourced to a crowd. If crowdsourcing is defined as 'an online, distributed problem-solving and production model that leverages the collective intelligence of online communities' (Braham, 2013, p.xix), then it should be possible to leverage the MIT Alumni crowd to utilize its collective intelligence to forecast sustainable and environmentally friendly technologies, business models and political initiatives to inform decision making.
Recently, research has identified the existence of so-called 'super forecasters' who are amateur forecasters that, due to a certain mindset and thought processes, are able to outperform intelligence agencies in providing intelligence forecasts (Tetlock and Gardner, 2015). It is argued that the MIT alumni network constitutes a unique 'crowd' entailing remarkable ability, intelligence and diversity. These are the building blocks of accurate group predictions (Page, 2007), potentially making the MIT alumni crowd one of the most collectively 'wise' to tap into. Hence, they may constitute a unique crowd that may be able to, collectively, predict the success of various climate friendly technologies, business models and political developments. By tapping into the MIT crowd, we can obtain accurate forecasts to inform decision making, and improve climate-related investments.
What actions do you propose?
The author of the proposal would welcome the opportunity to take an active part in the planning and execution of the proposal, and the following action plan is proposed:
- Create a forecasting tournament with MIT alumni.
- Assess the accuracy of predictions in climate related issues.
- Use forecasts in decision making within policy, technology investment and education.
The following will further detail and explicate the proposed rationale and its inherent actionable steps and sub-activities.
Novelty of the forecasting tournament
The combination of MIT alumni and forecasting to inform decisions on climate change is unique on a global basis, albeit it entails promising aspects. Hence, MIT would be at the forefront of this promising trajectory. Although several examples exist of prediction markets and forecasting tournaments, none have focused on the climate challenge to the knowledge of the proposal author. Furthermore, the author does not know of any universities that leverage their alumni networks in forecasting tournaments. Hence, MIT would be pioneers in this field, and would set the bar for a constructive use of the collective alumni base to contribute in solving global problems of urgent need of action.
MIT Alumni: One of the wisest crowds on the planet?
“Progress depends as much on our collective differences as it does on our individual IQ scores.” ~ Scott E. Page in 'The difference'.
According to Scott E. Page (2007), accurate crowds are comprised by (i) ability and (ii) diversity. When relating this scientific fact to the MIT alumni crowd, it becomes difficult to imagine a crowd that entails more ability (within technology, science and entrepreneurship) as well as diversity (in terms of nationalities and cognitive diversity). Consequently, the MIT alumni crowd should be able to predict accurately, according to the 'prediction diversity theorem' (Page, 2007). MIT alumni similarly have competencies within e.g. technology and business, which are essential domains for shaping climate solutions. Moreover, MIT has traditionally emphasized systems thinking which is pivotal in climate initiatives.
What role could MIT faculty have?
The role of relevant MIT faculty could be to pose/inform 'the correct' questions to be asked to the MIT alumni crowd. Obviously a concern could be individual biases when formulating questions. One way to circumvent this problem could be that questions needed to be double blind-reviewed similar to academic publications. Hence, anonymous reviewers (comprised by a blend of practice-based experts - academics - alumni) could review questions posed by relevant MIT faculty members in order to circumvent individual biases and validate the relevance and usefulness of the questions. As a list of e.g. 10 questions is not particularly time-consuming, it is argued that this would be a cost-effective manner of securing bias-free questions. An alternative would be having a panel of MIT faculty experts formulating relevant questions; however, group discussions always entail the risk of 'groupthink' and other social biases, so a blind review process is preferred.
What format should the forecasting tournament be?
"Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs. These habits of thought can be learned and cultivated by any intelligent, thoughtful, determined person." ~ Philip Tetlock in 'Superforecasting'.
As suggested by the above quote, ordinary individuals may become reasonable forecasters if they are considerate of certain aspects: The format of the tournament may further contribute to this trajectory, by entailing elements designed to support these processes.
The format could be either as a prediction market or a conventional forecasting tournament. The use of competitive formats has the following benefits: (i) It fosters an incentive to answer thoughtfully and reflective (ii) the gamification aspect of a forecasting market or tournament fosters intrinsic motivation and engagement.
Recent years have seen the advent of prediction markets such as 'The Hollywood Stock Exchange', in addition to corporate prediction markets at Google, Intel and Microsoft to name a few. These prediction markets take the form of both external and internal 'betting markets', where the participants invest fictional money on various scenarios. The subsequent price will then reflect the forecasted likelihood of the event. They have been used to inform decision makers in terms of e.g. the prospects of a new product, how many users a new service will have - and whether or not a project deadline will be met.
Another way of creating a forecasting tournament would be to obtain inspiration from IARPA's forecasting tournament utilizing amateur forecasters (as described in Tetlock and Gardner, 2015). In a similar vein, 'The economist' currently has an open forecasting tournament which may illustrate how this could be carried out in practice, although with a focus on climate friendly technologies and sustainable business models. The economist's forecasting tournament asks ordinary readers of the economist to forecast various political and economic questions related to international relations. Questions take the form of e.g. "Who will win the Presidential election in country x?" Or "Will there come a peace agreement in country x before May 1, 2016?".
Tapping into the MIT crowd concerning climate related issues would obviously necessitate slightly different questions. Questions could be framed as "How likely do you think it is that technology X would reduce co2 emission at the MIT campus by x% by June 2016?" with response possibilities ranging from 0 - 100%. Accuracy could then be calculated by e.g. Individual and collective brier scores, squared errors or mean absolute errors.
Forecasting accuracy
“Do not be misled by expert bravado or by an expert’s own sense of how he or she is doing. Evidence is a much better guide than an impressive self-presentation.” ~ Cass Sunstein in 'Wiser: Getting beyond groupthink to make groups wiser'.
As suggested by the above quote, we must rely on evidence to assess how we are doing. Hence, it is of paramount importance that (some) of the questions have a short to medium-term horizon, in order to asses the accuracy of the forecasts which would illustrate the collective wisdom of the MIT alumni crowd. This would constitute the main argument for utilizing their forecasts in decision making. Furthermore, prediction accuracy metrics of the MIT alumni network would in itself be interesting, as it would show (i) how 'wise' the crowd is in terms of climate-related foresight (ii) do MIT alumni over or under estimate developments? and (iii) the performance tracking could lead to individual reflection and learning among the participants, improving their decision making, forecasting abilities and interest within climate related issues. Moreover, quantifying the predictive accuracy fosters a 'gamification' element that promotes intrinsic motivation.
As previously noted, forecasting accuracy could be reported in e.g. brier scores, sum/mean squared errors, MAE etc. If calculated in sum of squared errors, the prediction diversity theorem can be filled out, and we would actually be able to put a numerical value on how much of the accuracy is due to ability - and how much is due to diversity (Page, 2007). This could, in itself, have the added benefit of uncovering inherent aspects of the MIT alumni network composition, in addition to helping solve climate issues.
How to engage 130,000 people?
The proposal predominantly relies on the participants' intrinsic motivation - a key feature of MIT students. Here, the proposal will speak to the network's intrinsic motivation as follows:
- As the proposal consists of a forecasting tournament, where participants compete to be the best forecaster, the proposal seeks to foster a 'gamification' aspect to problem-solving, which should make it fun for the participants.
- Due to the competitive aspect with objective accuracy measures and interesting questions, alumni participants will naturally learn throughout the process - which is suitable for the MIT philosophy, in addition to being a key component of intrinsic motivation.
- The competition centers on climate solutions, and hence, focuses on a larger mission of value to us all. It has often been shown that intrinsic motivation is linked to a deeper purpose - often of altruistic nature.
Although the proposal's pillars are rooted in intrinsic motivation, the competitive format has the inherent property of a prize (incorporating an extrinsic motivational element). However, this prize (monetary and/or reputational) would merely be symbolic, as the 'gamified' tournament would predominantly be an engine fueled by intrinsic motivation.
Communication plan
The communication of the project is expected to rely upon (i) official MIT and MIT alumni channels (ii) online and social media diffusion, and (iii) press coverage/PR.
Initial communication is furthermore a challenge. Here, it is proposed that invitations are e-mailed to MIT's alumni network through the relevant MIT communication channel - and it is promoted on MIT's website in addition to calls in relevant social media groups/outlets. MIT's Alumni Association might be helpful in disseminating the message of the tournament to the alumni network. A website for the competition is needed where ongoing information will be provided to the participants. Due to the novelty of the initiative, it could be possible to obtain some media coverage of the tournament, which could help increase awareness of the project.
Who should be involved in planning and execution?
The author of the proposal would, as previously noted, welcome the opportunity to plan and execute the tournament. The author has experience with long-term, large-scale forecasting tournaments to draw upon. However, it is evident that it would be necessary to collaborate closely with the MIT Alumni Association, and it would be fruitful to work with relevant researchers at MIT (e.g. Researchers within crowdsourcing - collective intelligence - predictions - climate - systems thinking - and climate friendly technology). Here, it is suggested that the set-up, in itself, relies upon the collective intelligence of MIT, the MIT alumni association and the proposal author.
Who is going to act on the forecasts?
It is proposed that in conjunction to the formulation of questions, each selected question should identify key stakeholders (mediators of change - decision makers - influencers - influenced) of the content of the question. Ideally, some of these stakeholders would be lined up before the forecasts were launched, so that they could contribute to the process, and they can continuously incorporate updated insights into decision making processes.
Who to include in this process is determined by the content of the specific questions: If the question concerns a climate initiative at MIT, relevant MIT decision makers and influencers must be lined up before the forecasts are run. If the question concerns technology investments that influence global climate change, the information could be provided to key policy decision makers at e.g. the COP summit. Local forecasts could inform local decision makers and regional businesses. Furthermore, the MIT alumni network itself is also full of essential decision makers and influencers, so the mere presence of an MIT alumni forecasting tournament could diffuse insights/forecasts from the tournament to the right outlets. Some of the questions might be formulated at the initiative of key climate stakeholders approaching MIT, which would secure decision maker involvement at the outset.
However, an essential argument for listening to these forecasts is the predictive accuracy that we would be able to assess from tracking predictions. Furthermore, a relevant vantage point could be MIT related climate forecasts where MIT decision makers could initiate the process.
Another relevant idea might be to have an advisory board connected to the tournament with relevant climate-related stakeholders/decision makers from the public domain, businesses, academia and NGOs. Their purpose can be to support and advise on how to best incorporate the forecasts in practice-based decision making on an ongoing basis.
Examples that can serve as inspiration
A multitude of examples exist which can serve as inspiration. Research has documented examples of prediction and idea markets in the following organizations - to name just a few:
- Google's internal prediction market.
- The Hollywood stock exchange predicting movie success.
- Illy Lilly predicting chances of pharmaceutical products and substances in the pipeline.
- HP predicting printer sales.
- Intel conducting demand planning.
Furthermore, the following forecasting tournaments/forecasting sites may similarly provide inspiration:
- IARPA's forecasting tournament forecasting intelligence events (more info about this can be seen at the following link: http://www.bbc.com/future/story/20121009-for-all-of-our-eyes-only ).
- The economist's forecasting tournament on political, economic and business events (this can be seen at the following link: https://www.gjopen.com/challenges/5-the-economist-s-world-in-2016 ).
- The Microsoft Prediction Lab predicting a range of political and social events.
- PredictWise, predicting various aspects from oscars to sports.
- Nate Silver's five-thirty-eight website.
The climate colab platform is, in itself, a proof-of-concept that it is possible to crowdsource ideation of solutions for the climate challenge. However, by focusing on forecasts and the MIT alumni crowd, this proposal will further add onto this promising trajectory.
Necessary conditions for the design of the tournament
“Diversity and independence are important because the best collective decisions are the product of disagreement and contest, not consensus or compromise.” ~ James Surowiecki in 'The wisdom of crowds'.
One of the main paradoxes of crowd wisdom is that participants must act as individuals rather than a crowd, in order to obtain collective wisdom. This means that the participants must provide independent forecasts. Another paradox is that the crowd average can be accurate, although the individual participants are not. This is because crowd wisdom is built upon negatively correlated prediction errors that cancel each other out - meaning that the noise cancels each other out while the signal remains. Both of these paradoxes suggest that independence is a crucial feature of wise crowds.
The MIT alumni crowd will no doubt have the needed ability and diversity to make accurate forecasts (Page, 2007), but the design must make sure that the individual forecasts are provided independently of each other. Here, the best choice might be to do a forecasting tournament without the market-based incentive (prediction markets), as it may entail the risk of fostering dependent forecasts with correlated prediction errors that may lead to information cascades that can result in flawed predictions with inflated confidence. Consequently, the format design is essential.
Walking through an actual example of the platform
A made up example of some of the steps in the tournament may be illustrative of the potential value of the idea: Imagine that you're an MIT alumni member with an interest for technology, science and climate change - which makes you curious about entering the competition. You have therefore registered on the website, and you are now looking into a list of 10 climate-related questions, which faculty members have posed, that deal with everything from technological developments, policy scenarios and assessments of the commercial implications of a novel sustainable business model. For instance, one of the questions asks: "Will the environmentally friendly technology X, which was recently developed and launched by company Y, have more than 100,000 users by June 2017?" You draw upon your insights about the technology - you relate it to similar cases of user adoption of sustainable technologies - and you start reading a bit about it, in order to make a sensible forecast. On the forecast dash board of the platform, you are able to track your own predictive accuracy, the predictive accuracy of the top 10 individuals - and the predictive accuracy of the crowd as a whole. From it, you can see that you are doing ok as a forecaster, but the crowd is the most accurate. Furthermore, you can read on the website that 2 of the forecasts have recently been cited and utilized by White House representatives to argue for actions in favor of climate technology. Both of these aspects keep you motivated to further contribute with forecasting.
Summing up the proposal
As it has been extensively explicated, there is reason to be hopeful for the implications of MIT alumni crowd forecasts in the climate domain. The main gist of the proposal can be captured in a very few sentences:
What: Tap into the collective wisdom of the MIT alumni crowd by making them forecast climate related developments.
Who: Alumni should forecast - faculty should pose questions - and question stakeholders should act.
When: Planning and preparation could start in 2016, but the competition itself should commence in 2017.
Where: Online: The platform is on a website of its own - dissemination through e.g. relevant MIT and social media outlets.
Why: The MIT alumni crowd may very well be the most wise on the planet, and collectively, they should be able to come up with accurate forecasts that can inform wise decisions concerning the climate.
How: The tournament could forecast developments - predictive accuracy is calculated - and forecasts are utilized in decision making.
Put together, these elements comprise the ingredients in a recipe for a promising and feasible journey towards improving our planet's future - with the help of MIT alumni.
By:
MIT Superforecasting
