by Samuel LeLacheur
Science is the realm of cold hard facts. The scientific method leaves little room for speculation—and intentionally so. However, even in science, and especially in the social sciences, opinions can indeed matter.
A new initiative being led by a trio of economists, Stefano DellaVigna of the University of California, Berkeley, Devin Pope of the University of Chicago, and Eva Vivalt of the Australian National University seeks incorporate expert opinions to gather predictions on what results will be before research is conducted. The premise underlying the project is that by collecting predictions, researchers can improve experimental design and better interpret results.
Their project is headquartered at the website socialscienceprediction.org, which was launched this year. It allows users to view social science research projects, mainly in economics, for which data has not yet been collected. They can then make predictions as to what they believe the conclusions of the study will be. To ensure the quality of predictions, those of experts in the field (professors, researchers, and PhD students) are collected separately from those of the general public. The goal of the site is to make it so that the barriers to prediction-making are as low as possible.
It may appear at first as though collecting predictions on research is an exercise in futility. In principle the objective analysis entailed in research will make it so that predictions have no discernable significance for results. And, could expert predictions not create inherent bias in experimental design favoring the predicted results? This does not prove to be the case, though, as DellaVigna, Pope, and Vivalt show in a recent article published in Science. Taken as a whole, the benefits of opinion-gathering prove to be both reliable and significant.
The first area of improvement DellaVigna, Pope, and Vivalt cite is that “null results” can be better appreciated. Rather than being dismissed by hindsight bias as an “obvious” result, looking at the predicted outcome of the research can shed light on to whether or not the null result was actually obvious, or is noteworthy and worth seriously considering.
Measuring and improving forecasting is another benefit associated with prediction-gathering. As the website states, collecting expert predictions can (once the results are known) “expose whether and how predictions are systematically biased or inaccurate.” This in turn would allow for better understanding on why certain forecasts may or may not be accurate, and under what conditions forecasts are most likely to stray from actual outcomes.
The final benefit cited is the improvement of experimental design. By compiling predictions, researchers can gain insights into which particular aspects of their work are of most value to fellow researchers. It then follows that they can then focus their research on what is deemed most valuable, thereby coming to more important and disciplinarily-relevant conclusions.
Prediction-gathering provides numerous advantages for the research community. The downside of this, it may appear, is that these benefits are limited only to researchers who specialize in that specific field. However, the benefits extend well beyond the specific fields and research community broadly, and apply to broader realms of society. The Science article further states that collecting predictions makes forecasts “have a practical value to policymakers needing to make a decision.”
Policymakers often have to craft policy on issues which have never been researched before, or to address developments that are too recent to have been the subject of extensive study. As a result, they rely frequently on the suggestions of experts on the subject areas on which they are working. If predictions by these experts are made erroneously, then naturally policy will not be effective at achieving its goals.
Given that the collection and testing of predictions improves the quality of future forecasts, policymakers are likely to benefit greatly from more predictions on research. They can better understand which forecasts are trustworthy, and also which particular experts in different fields are most likely to provide accurate forecasts. Policy effectiveness would improve.
Furthermore, because having predictions on research allows for results to be interpreted more meaningfully, policies can be supported by more scientifically-sound empirical evidence and written with greater specificity. This reduces reliance on generalized initiatives and vague guesswork by policymakers.
Overall, DellaVigna, Pope, and Vivalt’s initiative makes possible great innovations in the methods and rigor of social science research that will have tangible impacts that reach well beyond the purview of the research community to affect the lives of many.
Sam LeLacheur is a GW Scope staff writer and a junior humanities editor for the George Washington Undergraduate Review. He is a freshman in the Elliott School of International Affairs intending to double-major in international affairs and economics. His research interests include U.S. grand strategy, great power politics, state behavior, and international relations theory. He is not currently conducting any research, but in the past has researched the United Nations and humanitarian intervention. At GW, he is also a member of European Horizons and the Onero Institute.
Comments