A major concern for inventors, scientists and researchers in general is to establish the primacy of their discoveries by being the first to disclose an advance in knowledge, via either a scientific publication or a patent. This race tends to accelerate innovation, at the cost of possible adverse effects on quality.
The dilemma: quality versus speed
During the development phase, researchers and inventors face a dilemma. On the one hand, a longer maturation increases the chance of detecting an anomaly which reveals that their project is a poor lead that should be dropped (1). On the other hand, the fear of being overtaken by a rival may hasten publication or marketing decisions. The tension between these two competing forces (more learning versus the risk of being preempted) is characteristic of any innovation process, and the timing strategy ultimately chosen by scientists determines the final quality of their innovation.
The tension between a longer maturation and the risk of being preempted is characteristic of any innovation process.
The issue of the publication of negative results in science
In a world where the quality of scientific publications is increasingly challenged, the public debate has focused on the mechanisms that foster high-quality research. The question of whether or not to publish negative results, in particular, has received much attention. Recent studies show that academic journals have a marked bias for positive results (e.g., results that are significant in a statistical sense) (2).
However, the fact that negative results (for instance, results which, conversely, have insufficient statistical significance) are not made public is potentially harmful in two respects. First, a negative result is a valuable source of information for other researchers. For example, if two researchers are trying to prove the same theory, and one of them finds a counter-example that proves the theory wrong, the other researcher could benefit from knowing this information, which could spare him the cost of pursuing the same direction in vain. Second, the non-publication of negative results creates a selection bias that produces a distorted overall picture of the state of the art (3).
This is why many experts have recently spoken out in support of the systematic publication of negative results. For instance, the World Health Organization states that “negative and inconclusive as well as positive results must be published or otherwise made publicly available" (4). In line with this trend, several academic journals have decided to give greater publicity to negative results, either directly by promoting their publication or indirectly by requiring researchers to register their experimental projects in advance (5).
The publication of negative results does not necessarily increase the quality of publications
What impact should we expect from such a systematic disclosure of negative results on the quality of publications, in particular in a highly competitive research environment? We address this question in an on-going project with Catherine Bobtcheff (Paris School of Economics) and Thomas Mariotti (Toulouse School of Economics), where we model the choice made by a researcher of the duration of the learning phase in two distinct scenarios.
In the first scenario, negative results are systematically published and thus immediately made public: since the information is shared by the entire research community, each researcher learns faster than otherwise, hence an incentive to shorten the learning phase all the larger as competition is strong.
In the second scenario, negative results are never made public, and researchers respond to this by waiting longer to learn more about their projects. Indeed, they recognize that being the first to publish is now potentially bad news since a rival may have obtained a negative result and kept it private. To take up the earlier example, if a researcher who suspects that a competitor is trying to prove the same theory does not see the latter publish anything, he must take into account the possibility that the rival has found a counter-example that invalidates the theory, without this "negative result" being made public. This phenomenon well-known from the analysis of auctions is called the "winner's curse". It encourages researchers to prolong the learning period to enhance their confidence in their results.
Comparing the merits of these two scenarios highlights two opposite effects that the systematic publication of negative results has on quality. The learning process in the maturation phase is faster when the results are public (everyone benefits from more information), which is good for quality. At the same time, researchers adjust their timing strategies by shortening the learning period, which is detrimental to quality(6).
These preliminary findings already demonstrate that one should be careful before endorsing the systematic publication of negative results as a way to foster the quality of publications, especially in a highly competitive research environment.
By quantifying these two effects, our analysis shows that, when competition is strong, the second effect may be stronger, hence a lower average quality of publications when negative results are public! These preliminary findings already demonstrate that one should be careful before endorsing the systematic publication of negative results as a way to foster the quality of publications, especially in a highly competitive research environment.
(1) During this “learning phase”, a pharmaceutical firm typically performs additional tests on its new molecule to rule out the risks of side effects, or researchers carry out robustness checks on their results.
(2) See Fanelli, D. (2012), "Negative Results are Disappearing from Most Disciplines and Countries", Scientometrics, 90, 891-904.
(3) For example, a study by Turner et al. (2008), "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy”, The New England Journal of Medicine, 358, 252-260, shows that 94% of the studies on anti-depressants considered by the Food and Drug Administration (FDA) to approve marketing decisions exhibit positive results. But the percentage of positive results falls to 51% when unpublished studies not taken into account by the FDA are included in the panel.
(4) See the results here.
(5) For example, the Journal of Development Economics gives to authors the opportunity to have their potential empirical projects reviewed and approved for publication before the results are known.
(6) For instance, if a drug actually has important side effects, the firm is less likely to find out because the testing phase is shorter, so the risk that the drug is inappropriately brought to the market increases.
