Democracy data: how do researchers measure democracy?

You can explore the data discussed in this article in our Democracy Data Explorer.


Measuring the state of democracy across the world helps us understand the extent to which people have political rights and freedoms.

But measuring how democratic a country is, comes with many challenges. People do not always agree on what characteristics define a democracy. These characteristics — such as whether an election was free and fair — even once defined, are difficult to assess. The judgement of experts is to some degree subjective and they may disagree; either about a specific characteristic, or how several characteristics can be reduced into a single measure of democracy.

So how do researchers address these challenges and identify which countries are democratic and undemocratic?

In our work on Democracy, we provide data from eight leading approaches of measuring democracy:

These approaches all measure democracy (or a closely related aspect), they cover many countries and years, and are commonly used by researchers and policymakers.

You can delve into their data — the main democracy measures, indicators of specific characteristics, and global and regional overviews —  in our Democracy Data Explorer.

Reassuringly, the approaches typically agree about big differences in countries’ political institutions: they readily distinguish between highly democratic countries, such as Chile and Norway, and highly undemocratic countries, such as North Korea and Saudi Arabia.

But they do not always agree. They come to different assessments about which of the two highly democratic countries – Chile and Norway – is more democratic, and whether Chile is more or less democratic than it was ten years ago. At times they come to strikingly different conclusions about countries that are neither highly democratic nor highly undemocratic, such as Nigeria today or the United States in the 19th century.

Why do these measures sometimes reach such different conclusions? In this article I summarize the key similarities and differences of these approaches.

How is democracy characterized?

In this and the following tables I summarize how each approach defines and scores democracy, and what coverage each approach provides.9

We see that the approaches share a basic principle of democracy: a democracy is an electoral political system in which citizens get to participate in free and fair elections. The approaches also mostly agree that democracies are liberal political systems, in which citizens have additional civil rights and are protected from the state by constraining it.

Some approaches stop there, and stick to these narrower conceptions of democracy. Others characterize democracy in broader terms, and also see it as a participatory and deliberative (citizens engage in elections, civil society, and public discourse) as well as an effective (governments can act on citizens’ behalf) political system.

Varieties of Democracy — true to its name — offers both narrow and broader characterizations, by separately adding liberal, participatory, deliberative, as well as egalitarian (economic and social resources are equally distributed) political institutions to electoral democracy.

How is democracy characterized?
Varieties of DemocracyNarrow and broader: electoral, liberal, participatory, deliberative, or egalitarian democracy
Regimes of the WorldNarrow: electoral or liberal democracy
Lexical IndexNarrow: electoral (or liberal) democracy
Boix-Miller-RosatoNarrow: electoral democracy
PolityNarrow: electoral and liberal democracy
Freedom HouseNarrow: electoral or liberal democracy
Bertelsmann Transformation IndexBroad: electoral, liberal, participatory, deliberative, and effective democracy
Economist Intelligence UnitBroad: electoral, liberal, participatory, deliberative, and effective democracy

How is democracy scored?

The approaches also differ in how they score democracy.

V-Dem treats democracy as a spectrum, with some countries being scored as more democratic than others. 

Other approaches instead treat democracy as a binary, and classify a country as either a democracy or not. 

A final group does both, using a spectrum of countries being more or less democratic, and setting thresholds above which a country is considered a democracy overall.

Approaches that classify countries into democracies and non-democracies further differ in whether all countries that are not democracies are considered autocracies or authoritarian regimes, or whether there are some countries that do not clearly belong in either group.

And while Freedom in the World identifies which countries are electoral democracies in recent years, its main classification distinguishes between free, partly-free, and not-free countries (which many treat as a proxy for liberal democracy).

Beyond these broad similarities in how the approaches characterize and score democracy, their exact definitions differ in smaller ways, too. If you are interested in the details, you can take a closer look at the specific defining characteristics at the end of this article.

How is democracy scored?
Varieties of DemocracyOn a spectrum: 0 to 1, highly undemocratic to highly democratic
Regimes of the WorldAs a classification: closed autocracy < electoral autocracy < electoral democracy < liberal democracy
Lexical IndexAs a classification: non-electoral autocracy < one-party autocracy < multi-party autocracy without elected executive < multi-party autocracy
< exclusive democracy < male democracy < electoral democracy < polyarchy
Boix-Miller-RosatoAs a classification: non-democracy < democracy
PolityOn a spectrum: -10 to 10, hereditary monarchy to consolidated democracy
classification: autocracy < anocracy < democracy
Freedom HouseAs a classification 1: not free < partly free < free
classification 2: non-democracy < electoral democracy
Bertelsmann Transformation IndexOn a spectrum: 1 to 10, highly undemocratic to highly democratic
classification: hard-line autocracy < moderate autocracy < very defective democracy < defective democracy < consolidating democracy
Economist Intelligence UnitOn a spectrum: 0 to 10, highly undemocratic to 10 highly democratic
classification: authoritarian regime < hybrid regime < flawed democracy < full democracy

What differences are captured?

How the approaches score democracy affects what differences in democracy they can capture.

Classifications tend to be coarser, and therefore cover big to medium differences in democracy: they reduce the complexity of political systems a lot and distinguish between broad types, such as the democracies of Chile and Norway on the one hand, and the non-democracies of North Korea and Saudi Arabia, on the other.

The fine-grained spectrums of other approaches meanwhile reduce political systems’ complexity a bit less, and capture both big and small differences in democracy, such as the difference in democratic quality between the democracies Chile and Norway, and the difference between autocracies North Korea and Saudi Arabia. Spectrums can also better capture small changes within political systems over time, towards or away from democracy.

While some approaches use their classifications exclusively to reduce the complexity of their spectrums, others also use theirs to clearly define what features characterize each category.

What differences are captured?
Varieties of DemocracyBig to very small differences
Regimes of the WorldBig differences, with clear meaning
Lexical IndexBig to medium differences, with very clear meaning
Boix-Miller-RosatoBig differences, with clear meaning
PolityBig to medium differences
Freedom HouseBig differences
Bertelsmann Transformation IndexBig to small differences
Economist Intelligence UnitBig to small differences

What years and countries are covered?

The approaches also differ in what years and countries they cover. 

All approaches cover the recent past, but differ in how far they go back in time. BTI and EIU begin in the mid-2000s. Freedom in the World starts in the early 1970s. The other approaches go back to the beginning of the 19th century or even the late 18th century. The Regimes of the World data we ourselves extended back from 1900.

All approaches cover most countries in the world. They differ in how comprehensive their coverage is: BTI excludes long-term members of the OECD (which it considers consolidated democracies), while all other approaches assess them. Some approaches also include very small states and territories, and some also assess many non-independent countries, usually colonies.10

What years and countries are covered?
Varieties of DemocracyYears since 1789; 202 countries, also non-independent
Regimes of the WorldYears since 1789; 202 countries, also non-independent
Lexical IndexYears since 1789; 242 countries, also non-independent and microstates
Boix-Miller-RosatoYears since 1800; 218 countries, also microstates
PolityYears 1800 — 2018; 192 countries
Freedom HouseYears since 1972; 229 countries and territories, also microentities
Bertelsmann Transformation IndexYears since 2005; 138 countries and territories, no consolidated democracies
Economist Intelligence UnitYears since 2006; 167 countries

How are democracy’s characteristics assessed?

The approaches also differ in how they go about assessing the characteristics of democracy. 

Many rely on evaluations to assess democratic characteristics that are difficult to observe, such as whether elections were competitive and people were free to express their views. Some rely on evaluations by country experts to assess whether, or to which extent, democracy’s characteristics are present (or not) in any given country and year. Others depend on evaluations by their own researchers reviewing the academic literature and news reports. And many use both country experts and their own teams. A few additionally incorporate some representative surveys of regular citizens.

The Lexical Index and the Boix-Miller-Rosato data meanwhile work to avoid difficult evaluations by either experts or researchers, and mostly have their own teams assess easy-to-observe characteristics — such as whether regular elections are held and several parties compete in them — to identify (non-)democracies.

Depending on whether they score democracy as a spectrum or classification, the approaches then aggregate the scores for specific characteristics: some average, add, and/or weigh the scores, others assess whether necessary characteristics are present, and a few do both.

How are democracy's characteristics assessed?
Varieties of DemocracyMostly through evaluations by experts; some easy-to-observe characteristics assessed by own researchers
Then weighting, adding, and multiplying scores for (sub-)characteristics
Regimes of the WorldMostly through evaluations by experts; some easy-to-observe characteristics assessed by own researchers
Then evaluating whether necessary characteristics are (not) present
Then weighting, adding, and multiplying scores for a few characteristics
Lexical IndexMostly with easy-to-observe characteristics, few evaluations by own researchers based on academic research
Then evaluating whether necessary characteristics are present or not
Boix-Miller-RosatoMostly with easy-to-observe characteristics, few evaluations by own researchers based on academic literature
Then evaluating whether necessary characteristics are present or not
PolityMostly through evaluations by own researchers based on academic literature and news reports
Then weighting and adding scores for characteristics
Freedom HouseMostly through evaluations by country and regional experts and own researchers based on different types of sources
Free countries: then adding scores for (sub-)characteristics
Electoral democracies: then adding scores and evaluating whether necessary characteristics are present or not
Bertelsmann Transformation IndexMostly through evaluations by country, regional, and general experts, some evaluations by representative surveys of regular citizens
Spectrum: then averaging of scores for (sub-)characteristics
Classification: then averaging and evaluating whether necessary characteristics are present or not
Economist Intelligence UnitMostly through evaluations by own country experts, some evaluations by representative surveys of regular citizens
Then averaging and minor weighting of scores for (sub-)characteristics

How do approaches work to make assessments valid?

The next tables summarize how the approaches address the challenges that come with measuring democracy. The first challenge is to make their assessments valid — to actually measure what they want to capture.

The approaches go about measuring democracy differently because they weigh the challenges of measurement differently. 

For those mostly relying on experts, the priority is that democracy’s characteristics are evaluated by people that know the country well. For those relying on their own researchers, the priority is that the coders know the approach’s characterization of democracy and the measurement procedures well. And for those relying on representative surveys, capturing the difficult-to-observe lived realities of regular citizens is especially important.

How do approaches work to make assessments valid?
Varieties of DemocracyExperts (often nationals or residents) know country and characteristics well, own researchers know measurement procedures well
Regimes of the WorldExperts (often nationals or residents) know country and characteristics well, own researchers know measurement procedures well
Lexical IndexOwn researchers know measurement procedures well
Boix-Miller-RosatoOwn researchers know measurement procedures well
PolityOwn researchers know measurement well
Freedom HouseExperts know country or region well, own researchers know measurement well
Bertelsmann Transformation IndexExperts (about half of them local) know country well, regular citizens know their own experiences well
Economist Intelligence UnitExperts know country or region well, regular citizens know their own experiences well

How do approaches work to make assessments precise?

The approaches are also concerned with making their assessments in a precise and reliable manner.

Expert-based approaches therefore often recruit many experts in total, several experts per country, or even several to many experts per country, year and characteristic.

Own-researcher-based approaches instead either focus more on making difficult subjective evaluation mostly unnecessary, or encourage their teams to rely on many different secondary sources, such as country-specific academic research, news reports, and personal conversations.

How do approaches work to make assessments precise?
Varieties of DemocracySeveral experts per country, year, and characteristic used (usually 5 or more since 1900, often 25 per country)
Regimes of the WorldSeveral experts per country, year, and characteristic used (usually 5 or more since 1900, often 25 per country)
Lexical IndexCharacteristics easy to understand and observe; subjective evaluation therefore mostly unnecessary
Boix-Miller-RosatoCharacteristics easy to understand and observe; subjective evaluation therefore mostly unnecessary
PolitySeveral researchers used
Freedom HouseMore than 100 experts and researchers used in total; Experts and researchers rely on academic research, news and NGO reports, personal conversations, and on-the-ground research
Bertelsmann Transformation IndexTwo experts per country and year used
Economist Intelligence UnitOne or two experts per country and year used

How do approaches work to make assessments comparable?

The approaches also face the challenge of how to make the coders’ respective assessments comparable across countries and time.

The surveys therefore ask the experts questions about specific characteristics of democracy, such as the presence or absence of election fraud, instead of making them rely on their broad impressions. They also explain the scales on which the characteristics are scored, and often all of the scales’ values.

Measuring many specific low-level characteristics also helps users understand why a country received a specific score, and it allows them to create new measures tailored to their own interests.

How do approaches work to make assessments comparable?
Varieties of DemocracyExperts answer very specific questions about sub-characteristics on completely explained scale
Experts also code hypothetical countries and many code several countries, denote own uncertainty and personal demographic information
Project investigated expert biases and found them to be limited
Regimes of the WorldExperts answer very specific questions about sub-characteristics on completely explained scale
Experts also code hypothetical examples and many code several countries, denote own uncertainty and personal attributes
Project investigated expert biases and found them to be limited
Lexical IndexResearchers answer specific questions about characteristics on explained scale
Same researcher assesses all countries and years
Boix-Miller-RosatoSame researcher assesses all countries and years
PolityExperts answer specific questions about characteristics on completely explained scale
Freedom HouseExperts answer questions about characteristics separately
Bertelsmann Transformation IndexExperts answer specific questions about sub-characteristics on explained scale
Economist Intelligence UnitExperts answer specific questions about sub-characteristics on completely explained scale

How are remaining differences dealt with?

The approaches then all work to address any remaining differences between coders, even if they do so differently.  

V-Dem and RoW work with a statistical model which uses the experts’ ratings of actual countries and hypothetical country examples, as well as the experts’ stated uncertainties and personal demographics to produce both best and upper- and lower-bound estimates of many characteristics. They thereby avoid forcing themselves to eliminate all uncertainty and thereby possibly biasing their scores, and acknowledge that its coders make errors. This also recognizes that small differences in democracy on fine-grained spectrums may actually not exist, or be reversed, because measurement is uncertain.

Most other approaches go about it differently, and have researchers and experts discuss differing scores to reconcile them. This adds an additional step to make the assessments comparable across coders, countries, and years.

And while it uses discussions, Freedom in the World still acknowledges that it refined its approach over time, which makes its scores not as readily comparable: they work best for comparing different countries at the same time, or comparing the same country over the course of a few years.

The Lexical Index and Polity meanwhile do not have several coders per country and year, but they still worked to assess coding differences by once having its researchers rate some countries independently and compare their results. Reassuringly, they found that they came to similar conclusions.

How are remaining differences dealt with?
Varieties of DemocracyMeasurement model uses main and additional information and provides estimates of remaining measurement uncertainty
Regimes of the WorldMeasurement model uses main and additional information and provides estimates of remaining measurement uncertainty
Lexical IndexOne primary coder, so no differences between coders to be reconciled
Second researcher for some countries reproduced most assessments
Boix-Miller-RosatoOne primary coder, so no differences between coders to be reconciled
For recent years discussions among researchers reconcile different standards across coders, countries, and years
PolityDiscussions among researchers reconcile different standards across coders, countries, and years
Separate researcher teams for some countries and years reproduced most assessments
Freedom HouseDiscussions among experts and researchers reconcile different standards across coders, countries, and years
Bertelsmann Transformation IndexDiscussions among regional and general experts and own researchers reconcile different standards across coders, countries, and years
Economist Intelligence UnitDiscussions among experts and researchers reconcile different standards across coders, countries, and years

How do approaches work to make data accessible and transparent?

Finally, the approaches all take steps to make their data accessible and the underlying measurement transparent. All approaches publicly release their data and almost all make the data straightforward to download and use. Most approaches release not only the overall classification and scores, but also the underlying (sub-)characteristics. V-Dem even releases the data coded by each (anonymous) expert.

Almost all release descriptions of how they characterize democracy, as well as the questions and coding procedures guiding the experts and researchers. V-Dem again stands out here for its very detailed descriptions that also discuss why it weighs, adds, and multiplies the scores for specific characteristics. Polity, Freedom in the World, and BTI meanwhile provide additional helpful information by explaining their quantitative scores in country reports that discuss influential events.

How do approaches work to make data accessible and transparent?
Varieties of DemocracyProvides data for sub-indices and several hundred specific questions by country-year, country-date, and coder
Detailed questions and coding procedures are available and easy to access
Justifies democracy characteristics and their combination in detail
Regimes of the WorldProvides data for sub-indices and several hundred specific questions by country-year, country-date, and coder
Detailed questions and coding procedures are available and easy to access
Justifies democracy characteristics and their combination
Lexical IndexProvides disaggregated data for specific questions by country-year
Questions and coding procedures are available and easy to access
Justifies in detail democracy characteristics and their combination
Boix-Miller-RosatoProvides data by country-year
Questions and coding procedures are available and easy to access
Justifies democracy characteristics and their combination
PolityProvides disaggregated data for sub-indices and specific questions by country-year
Detailed questions and coding procedures are available and easy to access
Explains scores with country reports
Freedom HouseProvides recent disaggregated data for sub-indices and specific questions by country-year
Questions and coding procedures are available and easy to access
Justifies democracy characteristics
Explains scores with country reports
Bertelsmann Transformation IndexProvides disaggregated data for sub-indices and specific questions by country-year
Detailed questions and coding procedures are available and easy to access
Justifies democracy characteristics and their combination
Explains scores with country reports
Economist Intelligence UnitProvides disaggregated data for sub-indices by country-year
Questions and coding procedures are available
Justifies democracy characteristics

The best democracy measure depends on our questions

There is no single ‘best’ approach to measuring democracy. Conceptions of democracy are too different, and the challenges of measurement are too diverse for that. All of the approaches put a lot of effort into measuring democracy in ways that are useful to researchers, policymakers, and interested citizens.

The most appropriate democracy measure depends on what question we want to answer. It is the one that captures the characteristics of democracy and the countries and years we are interested in.

This means that having several approaches to measuring democracy is not a flaw, but a strength: it gives us different tools to understand the past spread, current state, and possible future of democracy around the world.

If you want to explore and compare the data that each of these datasets produce, you can do so in our Democracy Data Explorer.


Keep reading at Our World in Data
Acknowledgements

I thank Daniel Bachler, Hauke Hartmann, Joan Hoey, Staffan Lindberg, Michael K. Miller, Hannah Ritchie, Max Roser, and Svend-Erik Skaaning for reading drafts of this text and their very helpful comments.

What are democracy’s specific characteristics?