Replicating research is a fundamental, yet sometimes orphaned labor, a distant second to the comparative glamor of new discovery. QuantViews however is placing the replication of financial research as its forefront focus and emphasizing the important role replication has in the scientific method and for verifying that what we hold to be true actually is true.
Amongst the many sciences, financial research in particular, where markets change over time and mountains of data invite false conclusions, is a ripe area that could benefit from a second look by a skeptical eye. QuantViews will endeavor to replicate interesting and consequential published financial research and open source the effort by using publicly available data and sharing the coding and analysis. Along the way we will verify the main results and conclusions, illuminate hidden steps and choices in the methodology and update the analysis using current data.
See the website at www.quantviews.com
The prevalence and peril of a reproducibility crisis
For the past decade, multiple scientific fields have been confronting a so-called “reproducibility crisis” as researchers have been unable to corroborate previous published research conclusions via new experiments or analysis. The strain and self-reflection has surfaced across domains as varied as psychology and artificial intelligence and is a frequent topic of discussion in preeminent multidisciplinary journals such as Nature and Science. For example, a 2016 survey of 1,500 scientists across multiple disciplines said there is a significant reproducibility crisis and the majority of researchers indicate that they themselves had been unable to reproduce a prior experiment. More recently, as of August 2020 over 30 papers on the coronavirus (COVID-19) have been retracted, highlighting the questionable quality of even some of the most topical and crucially important published research.
Unfortunately, the fields of finance and economics are not exceptional in the replication crisis. Researchers at the Federal Reserve were able to replicate only half of 67 macroeconomic papers published in well-regarded economics journals and other researchers found that 60% of replication efforts failed within experimental economics. Finance has not yet had an accounting of past research efforts but the former editor of the Journal of Finance has warned that the revelation of an “embarrassing number of false positives” may be on the horizon.
Just because research is published doesn’t mean that it’s correct.
The consequence of false, irreproducible conclusions is wasted effort within the academic community as false conclusions will inform unproductive directions for new inquiry. More practically, false conclusions can have large real world impacts, the consequences of which are obvious in medical and pharmaceutical research, but also impactful in finance and economics where bad investment strategies may yield shortfalls in an entire generation’s retirement savings and poor economic inquiry could translate into ineffectual policies that cost the public billions of dollars.
QuantViews replication: Verification and education
The mission of QuantViews is to contribute to the quality of academic and industry discourse by reviewing and replicating existing research. The main goals will be to 1) evaluate whether the original research hypothesis and conclusions are plausible in light of the replication attempt 2) continually monitor and track the analysis through time and ask is if the conclusions still hold in the face of a continuously evolving market and 3) follow a set of practices that is consistent with reproducible research so that others might easily replicate the replication and learn from the experience.
Benign causes of irreproducible research and otherwise
In discussing the goal of replicable research, it is worthwhile to touch on some of the likely roots of false conclusions and the replication crisis. Unfortunately, outright fraud or unethical misconduct in the original research likely accounts for a portion of the failure rate while at other times, simple and innocent human error like excel errors, coding errors, or multiplying by 2 when you meant to multiple by 3 are also inevitably sources of error. In other cases, the technical difficultly in wielding statistical analysis is also a possible contributing factor to the replication crisis. Finally, it is likely that the pressure to publish for career advancement incentivizes compromised research methodologies which run the gambit from data snooping, to p-hacking, to unregistered hypotheses[1].
Accessibility and the difficulties in replication
Research can also be irreproducible if the experiment or analysis itself cannot be accessed by interested parties that seek to recreate or expand on the results. Finance is notable in this regard as many datasets are proprietary or costly and cannot be effectively shared across researchers. Similarly, as well articulated any methodology may be in a formal writeup, the many design choices in an analysis remain hidden without access to the authors’ underlying code and calculations.
Replication can also be hampered by the perceived costs of openness. Even if authors were able to share data, past attempts at mandating data and code sharing in finance journals were strongly opposed due to the burden of having to support other researchers in their replication attempt and because openness it might invite unproductive nitpicking over trivial code or analytical choices.
The QuantViews approach: Open source everything
The solution to the above difficulties with data and code sharing is simply to sidestep reliance on the original authors and to recreate the research from the ground up. QuantViews will endeavor to construct its own datasets using public data sources (say from Yahoo Finance or the Federal Reserve rather than from CRSP or Factset) and write its own code for analysis. In this sense, we are not simply “rerunning the code from the original authors” (reproducible research) but rather using the outline from the original research and running our own experiment which in many ways might slightly differ from the original (replicable research).
For example, it’s likely that the datasets in a replication might differ slightly, such as by using 18 sectors defined within the large cap S&P500 universe rather than 24 sectors from a larger large cap plus midcap universe. Or it might be the case that we use a different python-based rather than an R-based machine learning algorithm. These deviations are, in effect, a form of out-of-sample testing and can illuminate the degree to which any original conclusions might be data dependent or sensitive to slight variations in analytical approach.
The goal of QuantViews is to validate the quality of financial research and educate users in the process. Replication will be the framework for discussing a financial topic in that we will seek to tread a previously laid path while also being critical and verifying prior claims and conclusions. Additionally, we will endeavor to construct useful tools and applications with special focus on user-friendly and comprehendible presentations.
[1] Data snooping, p-hacking, and unregistered hypotheses (or inventing a hypothesis after running the experiment) are arguably massive problems in finance and investment research owing both to the nature of our data and the research culture. For example, finance is for the most part not an experimental science where the creation of new data is part of the research process. Rather, thousands of studies have all data mined the same datasets, whether that be the single US GDP series or the 1,500 stocks in a so-called liquid equity universe and this data familiarity is itself a
“soft” form of data snooping. Furthermore, the research culture is arguably less rigorous than in other sciences in the sense that it is very rare to define a research hypothesis upfront. This may seem innocuous but it also invites a “hard” form of data snooping. Consider the following:
“With field studies, hypotheses usually don’t ‘come out’ on the first data run. But instead of dropping the study, a person contributes more to science by figuring out when the hypo worked and when it didn’t. This is Plan B. Perhaps your hypo worked during lunches but not dinners, or with small groups but not large groups. You don’t change your hypothesis, but you figure out where it worked and where it didn’t. Cool data contains cool discoveries.”
The line between data snooping and not is arguably subjective but, to be clear, the above is misconduct and swap in a few terms like “regime change” and I don’t find it hard to imagine it coming from the mouth of a financial researcher.
