Skip to main content
About HEC About HEC Faculty & Research Faculty & Research Master’s programs Master’s programs MBA Programs MBA Programs PhD Program PhD Program Executive Education Executive Education Summer School Summer School HEC Online HEC Online About HEC Overview Overview Who We Are Who We Are Egalité des chances Egalité des chances Career Center Career Center International International Campus Life Campus Life Stories Stories The HEC Foundation The HEC Foundation Faculty & Research Overview Overview Faculty Directory Faculty Directory Departments Departments Centers Centers Chairs Chairs Knowledge Knowledge Master’s programs Master in
Management Master in
Management
MSc International
Finance MSc International
Finance
Specialized
Masters Specialized
Masters
X-HEC
programs X-HEC
programs
Dual-Degree
programs Dual-Degree
programs
Visiting students Visiting students Certificates Certificates Student Life Student Life
MBA Programs MBA MBA EMBA EMBA TRIUM EMBA TRIUM EMBA PhD Program Overview Overview HEC Difference HEC Difference Program details Program details Research areas Research areas HEC Community HEC Community Placement Placement Job Market Job Market Admissions Admissions Financing Financing Executive Education Executive Masters Executive Masters Executive Certificates Executive Certificates Executive short programs Executive short programs Online Online Executive MBA Executive MBA Train your teams Train your teams Summer School Youth Leadership Initiative Youth Leadership Initiative Summer programs Summer programs Admissions Admissions FAQ FAQ HEC Online Overview Overview Degree Program Degree Program Executive certificates Executive certificates MOOCs MOOCs

Instant

Scientific research: should negative results be published?

Decision Sciences

Many call for a systematic publication of scientific negative results in order to make the production of scientific knowledge more efficient. Raphaël Lévy, Assistant Professor in the Economics and Decision Sciences Department, explains why such dissemination of knowledge may actually be beneficial, but also points to potential undesired consequences.

old man writing stokkete-AdobeStock_cover

A major concern for inventors, scientists and researchers in general is to establish the primacy of their discoveries by being the first to disclose an advance in knowledge, via either a scientific publication or a patent. This race tends to accelerate innovation, at the cost of possible adverse effects on quality.

The dilemma: quality versus speed

During the development phase, researchers and inventors face a dilemma. On the one hand, a longer maturation increases the chance of detecting an anomaly which reveals that their project is a poor lead that should be dropped (1). On the other hand, the fear of being overtaken by a rival may hasten publication or marketing decisions. The tension between these two competing forces (more learning versus the risk of being preempted) is characteristic of any innovation process, and the timing strategy ultimately chosen by scientists determines the final quality of their innovation.

The tension between a longer maturation and the risk of being preempted is characteristic of any innovation process.

The issue of the publication of negative results in science

In a world where the quality of scientific publications is increasingly challenged, the public debate has focused on the mechanisms that foster high-quality research. The question of whether or not to publish negative results, in particular, has received much attention. Recent studies show that academic journals have a marked bias for positive results (e.g., results that are significant in a statistical sense) (2).

However, the fact that negative results (for instance, results which, conversely, have insufficient statistical significance) are not made public is potentially harmful in two respects. First, a negative result is a valuable source of information for other researchers. For example, if two researchers are trying to prove the same theory, and one of them finds a counter-example that proves the theory wrong, the other researcher could benefit from knowing this information, which could spare him the cost of pursuing the same direction in vain. Second, the non-publication of negative results creates a selection bias that produces a distorted overall picture of the state of the art (3).

scientifics
©kasto-AdobeStock

This is why many experts have recently spoken out in support of the systematic publication of negative results. For instance, the World Health Organization states that “negative and inconclusive as well as positive results must be published or otherwise made publicly available" (4). In line with this trend, several academic journals have decided to give greater publicity to negative results, either directly by promoting their publication or indirectly by requiring researchers to register their experimental projects in advance (5). 

The publication of negative results does not necessarily increase the quality of publications

What impact should we expect from such a systematic disclosure of negative results on the quality of publications, in particular in a highly competitive research environment? We address this question in an on-going project with Catherine Bobtcheff (Paris School of Economics) and Thomas Mariotti (Toulouse School of Economics), where we model the choice made by a researcher of the duration of the learning phase in two distinct scenarios. 

In the first scenario, negative results are systematically published and thus immediately made public: since the information is shared by the entire research community, each researcher learns faster than otherwise, hence an incentive to shorten the learning phase all the larger as competition is strong.

In the second scenario, negative results are never made public, and researchers respond to this by waiting longer to learn more about their projects. Indeed, they recognize that being the first to publish is now potentially bad news since a rival may have obtained a negative result and kept it private. To take up the earlier example, if a researcher who suspects that a competitor is trying to prove the same theory does not see the latter publish anything, he must take into account the possibility that the rival has found a counter-example that invalidates the theory, without this "negative result" being made public. This phenomenon well-known from the analysis of auctions is called the "winner's curse". It encourages researchers to prolong the learning period to enhance their confidence in their results.

Comparing the merits of these two scenarios highlights two opposite effects that the systematic publication of negative results has on quality. The learning process in the maturation phase is faster when the results are public (everyone benefits from more information), which is good for quality. At the same time, researchers adjust their timing strategies by shortening the learning period, which is detrimental to quality (6).

These preliminary findings already demonstrate that one should be careful before endorsing the systematic publication of negative results as a way to foster the quality of publications, especially in a highly competitive research environment.

By quantifying these two effects, our analysis shows that, when competition is strong, the second effect may be stronger, hence a lower average quality of publications when negative results are public! These preliminary findings already demonstrate that one should be careful before endorsing the systematic publication of negative results as a way to foster the quality of publications, especially in a highly competitive research environment.

 

(1) During this “learning phase”, a pharmaceutical firm typically performs additional tests on its new molecule to rule out the risks of side effects, or researchers carry out robustness checks on their results.
(2) See Fanelli, D. (2012), "Negative Results are Disappearing from Most Disciplines and Countries", Scientometrics, 90, 891-904.
(3) For example, a study by Turner et al. (2008), "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy”, The New England Journal of Medicine, 358, 252-260, shows that 94% of the studies on anti-depressants considered by the Food and Drug Administration (FDA) to approve marketing decisions exhibit positive results. But the percentage of positive results falls to 51% when unpublished studies not taken into account by the FDA are included in the panel.
(4) See the results here.
(5) For example, the Journal of Development Economics gives to authors the opportunity to have their potential empirical projects reviewed and approved for publication before the results are known.
(6) For instance, if a drug actually has important side effects, the firm is less likely to find out because the testing phase is shorter, so the risk that the drug is inappropriately brought to the market increases.

Cover Photo Credits ©stokkete on AdobeStock

Related content on Decision Sciences

overconfident young business man with a phone on a skateboard - vignette
Decision Sciences

The Impact of Overconfidence and Attitudes towards Ambiguity on Market Entry

By Thomas Åstebro, Cédric Gutierrez, Tomasz Obloj

Entrepreneurship and Startups
Academic Entrepreneurship: who owns patents and businesses?
©rudall30
Decision Sciences

Decision-making: do you need a decision theorist… or a shrink?

By Itzhak Gilboa, Maria Rouziou, Olivier Sibony

Decision Making Business Strategy - Sibony - HEC Paris ©Rawpixel.com
Decision Sciences

Decision making that reflects your strategy

By Olivier Sibony

Understanding and Aiding Good Decision Making - Newsletter K@HEC - ©Fotolia - Pathdoc
Decision Sciences

Understanding and Aiding Good Decision Making

By Itzhak Gilboa, Mohammed Abdellaoui, Brian Hill, Emmanuel Kemel

The Hunt for Better Decisions
Decision Sciences

The Hunt for Better Decisions

Newsletter knowledge

A monthly brief in your email box and 3 issues of the book per year.

follow us

Insights @HECParis School of #Management

Follow Us

Support Research

Our articles are produced thanks to our reader's support