Indicators, indexes and rankings: when numbers challenge legal analysis

David Restrepo Amariles, Professor of Law - December 7th, 2015
Indicators, indexes and rankings: when numbers challenge legal analysis by David Restrepo-Amariles ©Fotolia

The legal field has experienced rapid increases in the use of indicators, which give quantitative measures of aspects such as the extent of the rule of law. A new paper takes a closer look at why and how these legal metrics have emerged and what it means for governance.

David Restrepo-Amariles

David Restrepo-Amariles is assistant professor of tax and law at HEC Paris, which he joined in 2014, and a qualified lawyer to the Colombian Bar. He holds a Ph.D in Law from the (...)

See CV

“For the rational study of the law, the black-letter man may be the man of the present, but the man of the future is the man of statistics and the master of economics,” wrote Oliver Wendell Holmes, US Supreme Court Justice, in his influential 1897 essay The Path of the Law . He seems to have been proven right. Law, once the domain of purely linguistic, argumentative, historical and qualitative reasoning found in (“black-letter”) books, increasingly relies on mathematical and statistical techniques to describe legal phenomena and theories. Specifically, the number of legal metrics in the form of indicators, rankings and indexes is growing fast. According to a recent study, in the year 2000, only six sets of transnational indicators contained information about state legal systems; today, there are more than 20. In a paper he just published on legal indicators, HEC Paris researcher and law professor David Restrepo Amariles points out this rapid growth and explores their development, their value as tools of governance, and the way they affect the behavior of individuals and institutions across the world. 

The rise of metrics

David Restrepo Amariles attributes the rise in the use of these indicators to the shift in focus towards a “policy of performance” in the 1990s on the part of international organizations such as the World Bank and the United Nations. “There was this idea that if you wanted to make progress in development, it needed to be measured,” he says. He adds that NGOs and private companies also started to collect data (“raw facts”) and organize it in the form of indicators or indexes. Law, as an important component of development, became a new field in which things were quantified. The best-known example of such indicators is the World Bank's Doing Business project, which measures the extent to which the legal system facilitates or impedes business in 189 economies. The index ranks not just regulation (quality of contract enforcement, incorporation law, and so on) but also provides a picture of what is called “law in practice.” David Restrepo Amariles explains why contrasting what actually happens on the ground with the letter of the law is useful: “For example, France, Belgium and Italy may have pretty similar labor legislation, but trade union activity is quite different in the different countries, as can be witnessed by the recent incident between Air France executives and union leaders.” 

Guillemet
 NGOs and private companies also started to collect data (“raw facts”) and organize it in the form of indicators or indexes

Information tools, but not neutral ones

Of course, the point of these indicators is not to produce rankings for their own sake, but to better inform investors and public policy-makers. The researcher says figures facilitate a stronger grasp reality – and then work to improve it – citing the example of unemployment: “If you think of unemployment as a social reality, not an individual reality, you usually think of the rate of unemployment, and that data gives us better knowledge of reality.” However, the difficulty, as with any metric, legal or other, is that the cultural and legal context plays a role in the creation of the index. “Indicators reflect the economic and political views of the issuers and, especially when they are produced by institutions such as the OECD or the World Bank, their reform priorities,” says David Restrepo Amariles. For instance, business indicators tend to promote a liberal view of employment — how cheap labor is and how easy it is to fire workers. Conceived in such a way, are these instruments relevant? “They're relevant in so far as you agree with their orientation,” concedes the researcher, adding that “they may be criticized, but they are often designed by organizations that bring together 160 countries or more and so reflect a sort of global consensus.” He also says that those who disagree can produce their own metrics to counterbalance dominant ones, like the International Trade Union Confederation, which launched a Global Rights Index measuring protection of workers' rights in 2014. 

Informing managerial governance 

What is most important is that indicators are not just information tools but have what is called in the academic paper a “performative effect,” meaning that they factually constrain the behavior of individuals and institutions. For example, sovereign credit ratings, used to set standards or demand accountability, affect the decisions of states and investors. As David Restrepo Amariles points out, in a non-sovereign global environment, the indicators are used as a tool of governance: “International organizations don't act by force, they use indicators about GDP, human development, and so on to promote their views about developing countries – and they tie financial aid to such indicators.” But the increasing use of indicators, legal and other, for auditing, benchmarking, and generally as performance measures are part of a broader trend of managerial governance. The focus on efficiency that emerged as “new public management” in sectors such as health services has brought indicators, and the corresponding managerial mindset, to the fore in the legal field. For the researcher, it's an important trend. In his paper, he highlights it as part of a broader “mathematical turn,” meaning that mathematical arguments “are becoming acceptable explanations of legal phenomena and an emerging technique of regulation.” But he is not overly worried about this shift towards numbers. “The progress towards evidence-based decision-making might seem to be to the detriment of democratic debate, but the debate must also be supported by evidence, not just by personal beliefs.” And, he concludes, that's where indicators come in, to fill in the gaps, and inform decisions. 

Based on an interview with David Restrepo Amariles and his paper “Legal indicators, global law and legal pluralism: an introduction" (Special Issue on The Journal of Legal Pluralism and Unofficial Law , June 2015). 

Implications for managers and policy-makers
Implications for managers and policy-makers

If legal information no longer comes as “a 150-page document of legal advice” but as numbers, then those on the receiving end in international organizations and corporations need to gain mathematical literacy, says David Restrepo Amariles, pointing out that managers often do have the competence, but public policy-makers perhaps less so. He also says that legal indicators being relatively new, corporations tend to work using it on a case-by-case basis, and might need to develop a protocol: “Typical professional legal knowledge used to come in books. This new type of data is disturbing and may be difficult to integrate by legal counsels.” Most importantly, says the researcher, legal indicators on their own should not be taken as “go or no-go signals,” but should be complemented by qualitative information: “For example, just because the indicator for risk of expropriation in a given country is very low or equals zero doesn't necessarily mean it's safe for private property overall.”

Methodology
Methodology

David Restrepo Amariles analyzed legal indicators in the framework of global law and governance. Following the ideas of legal pragmatism (a perspective that sees law as a practice rooted in a specific context), he attempts to describe the effects that indicators produce on the conduct of actors, including states and transnational corporations, in an increasingly globalized society. For this paper he analyzed policy documents from international organizations since the 1960s. He studied organizations like the World Bank, the OECD, the United Nations and the European Union. He also relied on an extensive literature review of recent works on statistics and indicators, management and law.