By Kabira Namit
1. INTRODUCTION
We need credible data to design evidence-based policies, measure national progress and compare development strategies across nations. However, government officials and donor partners agree that the current sets of development statistics being published by national statistics offices are largely unreliable. The statistics are derived after making a series of simplifying assumptions, generalizations and hasty guesswork (Jerven, 2013). This lack of quality data is a major challenge for the government as they are unable to measure the success of their projects and policies. With no real insight into the impact of current policies, relatively simple interventions that could improve outcomes of ongoing policies fail to be conceptualized and implemented. Any new policies are based on ‘guestimates’ and anecdotal evidence. Similarly, the lack of credible information is a challenge for donors as they monitor their projects and decide which programs to support. It will be a particularly major challenge over the next fifteen years, as the Sustainable Development Goals become the cornerstone of foreign aid. The SDGs require a set of measures and indicators that are consistent and comparable across nations. Currently, the SDGs contain 17 goals and 169 targets. All of these 169 targets need to be measured periodically. This paper highlights the scope of data problems and examines the key bottlenecks that are responsible for the poor quality of the data in the developing world. Subsequently, it surveys the efficacy of policy responses to this ‘statistical tragedy’ (Devarajan, 2011). The bottlenecks and policy recommendations presented here are based on a series of informational interviews conducted with government officials and development practitioners who are currently based in Ghana and Malawi.
2. SCOPE OF THE PROBLEM
We have a natural inclination to treat numbers as factual absolutes. Economists routinely use development statistics to run cross – country regressions and assess the merits of policy alternatives. Economic historians write about the success or failure of development trajectories by comparing historical GDP growth rates. Yet, no matter how carefully the regression analysis is run or how convincing the GDP tables may be, the key assumption is that the underlying numbers are reasonably accurate and don’t suffer from measurement error. However, Jerven (2013) argues that it might be better to stop thinking of development statistics as facts and start considering any statistical estimate to be a product, whose production is subject to economic, political and logistical constraints. An example can help illustrate the magnitude of the problem. In November 2010, Ghana revised its GDP estimates upwards by $13 billion dollars (approximately 60% of its GDP) after rebasing the base year from 1993 to 2006. Overnight, Ghana moved from a low-income country to a lower-middle-income country. Sydney Casely Hayford, a financial analyst from Ghana, argues that these GDP figures could still be out by twenty percent (from Gray, 2010). Similarly, Nigeria’s GDP saw an 89% increase in 2013 to $510 billion after it updated its base year to 2010. Instantly, Nigeria’s economy rose to 24th place in the list of world’s large economies, ahead of Belgium and Taiwan and became the largest economy in Africa. Thus, till the day prior to Ghana and Nigeria’s announcements, all international organizations were reporting GDP statistics on their websites that understated the national GDP figures for these two countries by 62% and 89% respectively. Rebasing isn’t the only problem with GDP estimates. Devarajan (2011) states that only 35% of Africa’s population lives in countries that use the revised 1993 UN System of National Accounts. Some countries are using accounting systems that date back to the 1960s. Poverty estimates are similarly unreliable. Though poverty across countries in Sub Saharan Africa declined by 9 percentage points from 1995 to 2005 (from 59% to 50%), the statistics represent data from only thirty-nine countries for which internationally comparable estimates are available. For all the others, the 2005 poverty estimate was merely an extrapolation. In the case of Botswana, the extrapolation was done on the basis of 1993 data. The primary challenges in the measurement of GDP and poverty is the paucity of source data. Detailed household, enterprise and agriculture surveys are conducted infrequency. When new results become available, all estimates made during the intermittent period have to be substantially revised. The Africa Development Bank Report (2014) shows that most countries are using surveys for all their national estimates that are at least half a decade old. Even key social sectors like health and education where census surveys are conducted every year have unreliable estimates. As shown in the subsequent section, key enrollment numbers in the education sector are overestimated by as much as 21.4% for some countries and at least 3.1% on average across Sub Saharan Africa. Thus, there are substantial discrepancies and alarming gaps in statistical knowledge about GDP, poverty, productivity, income distribution and population for most developing country economies. These estimates are even worse in poorer countries, as statistical capacity there is lower. Annex 1 shows the Statistical Capability Index of all countries ranked by lowest performing countries. Somalia, Libya and South Sudan have the lowest capacity according to this World Bank indicator. The next section examines the key bottlenecks impairing the ability of nations to produce quality statistics.
3. KEY BOTTLENECKS
a) Lack of Resources According to Shanta Devarajan, the World Bank Chief Economist for Africa and the Middle East, “In many countries the statistical office is like an orphan. I’ve encountered cases where the Minister is not even aware that the Statistical Office is under his Ministry.1 ” Jerven’s Poor Numbers (2013) traces the history of this decline in the prestige of statistical agencies. In the late colonial period and the brief period from independence to the economic crisis of the 1970s, the statistical capacity of African states benefitted from significant expansion. However, since the structural adjustment took place in the 1980s and the 1990s, statistical offices were neglected. In an era when public spending was declining and the informal economy was growing across Sub Saharan Africa, these statistical offices were asked to produce more statistics in less time and with fewer resources. Jerven (2013) paints a despairing picture of the Ugandan national statistical office in the 1990s with a derelict building, a singular roadworthy vehicle and no computers, so all statistics had to be tabulated using desk calculators. 1 Quoted in Gray (2012) Even now, most statistical offices lack the requisite infrastructure to conduct analysis and do not benefit from the latest technology. Moreover, in most countries, statistical offices are geographically separated from frontline ministries. For example, in Malawi, almost all government ministries are on Capitol Hill in Lilongwe, the national capital. However, the National Statistical Organization is based in Zomba, 210 miles (or about six hours) away. In countries like Malawi where the Internet is still slow and erratic, such distances create a communication barrier that makes the collection and dissemination of data all the more difficult.
b) Human Capital Constraints Talented recruits stay within the public sector for only a brief period of time before they are poached by donor agencies or the private sector. Thus, high rates of turnover lead to governments having to train new staff every year. A good example may be Ghana’s Ministry of Education. Ghana has a decentralized budgetary process where staff from Headquarters annually collates data from the two hundred and sixteen districts in the country. Every year, the district-level staff has to be retrained in completing budgetary forms and submitting funding requests as most of the district budget officers trained in the previous year have moved on. At the national level, long-term employees get headhunted by donor agencies like the World Bank to work as Project Coordinators when a new project becomes active. International organizations offer a significantly higher remuneration to experienced employees, leading to a loss in institutional memory.
c) Political Pressure The National Statistical Agencies in most developing countries are not autonomous. Government Statisticians are political appointees and serve at the pleasure of the President. Thus, the objectivity of the data analysis is often called into question. In cases where the government statisticians and the ruling government disagree, the Chief Statistician may be removed from office. In January 2012, Dr. Grace Bediako, the Chief Statistician of Ghana was asked to go on indefinite leave while the National Census results were being tabulated. Subsequently, reports of disagreements between her and the ruling National Democratic Party (NDC) regarding Census estimates emerged in the press. In particular, census data, poverty estimates and inflation figures are under constant public scrutiny. As potential electoral “hot buttons,” these statistics are frequently politicized. Devarajan (2011) points out that during election years, there is a strong tendency on behalf of the host government to delay the publication of unflattering results that may reflect poorly on the incumbent administration. When the results are finally published, the raw data is never made publicly available. Thus, it becomes impossible for researchers to replicate and validate the reported results. Moreover, Jerven (2013) finds that statisticians often ‘self – censor.’ If the data shows something controversial, they may be called up in front of their superiors. Statisticians prefer to omit uncomfortable estimates rather than get mired in any controversy.
d) Donor Demands The Nationals Statistical Offices are inundated with requests from donor agencies and the NGO sector for customized reports for each of their projects. In spite of the Paris – Accra – Busan declarations on aid effectiveness, monitoring templates aren’t harmonized across aid agencies. In fact, agencies like USAID have different monitoring templates and requirements for every project. Project funding is contingent on the completion of detailed reports with statistical analysis. Moreover, statisticians receive a generous per diem allowance to collect data and draft reports for donor agencies. Thus, they prioritize the completion of these reports over national priorities. Instead of building up statistical capacity, donor intervention distorts data production. Also, if donor agencies require data urgently for the purposes of their own reports and publication, they habitually bypass the national statistical agencies and use a parallel structure for gathering data (like international impact evaluation firms). Such parallel structures do little to help strengthen the countries’ statistical capacities.
e) Principal Agent Problems In ‘The Political Economy of Bad Data,’ Sandefur and Glassman (2014) show that misrepresentation of national statistics does not occur only due to a lack of resources or poor human resource capacity. The incentives provided by the governance and funding structures lead data producers to overstate development progress. In their paper, they discuss two principal agent problems that lead to a systemic upward bias in the statistical indicators. Firstly, national governments can be seen as agents of international aid donors. When donors link explicit performance incentives to administrative data, the statistical data shows a policy induced upward bias. For example, in 2000, the Global Alliance for Vaccines and Immunization (GAVI) offered to pay eligible countries a fixed payment for every additional child immunized against Diphtheria-tetanus-pertussis (DTP3). Payments were based on data collected by national health ministries. Sandefur and Glassman show that this incentive structure led to a 5% overestimate of coverage rates across forty-one African countries. The second principal – agent problem arises between the national governments and the civil servants in their district and regional offices. When allocating resources to the district or regional level, on most occasions, the Ministry depends on self- reported administrative data from the local district offices. For example, after school fees were abolished for primary education in most of the Sub Saharan Africa region, the Ministry of Education began providing resources to the district based on a head count of pupils. The public schools have no incentive to report accurate student figures to the district offices, the district offices have no incentive to report accurate student figures to the regional offices and the regional offices has no incentive to report accurate enrolment rates to the Ministry. By the time the “Census figure” reaches the Ministry of Education, the “degree of optimism” in the data is very high. In Ghana, the practice is colloquially known as ‘massaging the data.’ Interview respondents from Ghana estimated the upward bias to be at least 5%. Sandefur and Glassman compare administrative data with survey data to show the average gap across Sub Saharan Africa (for 21 countries for which data is available) to be 3.1%. In Kenya, the gap was reported to be 21.4%. Kenya’s administrative data has shown a steady rise in enrollment figures following the introduction of free primary education. Yet, two household surveys have shown almost no change in enrollment rates. The systematic misreporting undermines the ability of the state to design evidence-based policies or manage their public resources. It is also relevant in assessing the progress of nations towards achieving the Millennium Development Goals and the upcoming Sustainable Development Goals. For example, Tanzania’s Education Management Information System (EMIS – the Ministry of Education administrative database) suggests that the country is on course towards reaching the universal primary enrollment Millennium Development Goal. However, household surveys suggest that 17% of children between the ages of 7 and 13 are not in school (Morisset and Wane, 2012).
4. POLICY RESPONSES
It is difficult for countries to give importance to the proper collection of national statistics when budgets are scarcely adequate to pay for essential public sector human resources – like teachers and nurses. Yet, governments and donors must prioritize the funding for statistical agencies so that they may attract and retain high quality employees. Earmarked funds over a three-year rolling period can help reduce the volatility and unpredictability of funding. Assured funding will also prevent officers from prioritizing donor projects (with guaranteed funding) over national priorities and core statistical activities. Donor funding should not be tied to specific agency projects and reports but must fund core statistical agency activities. In keeping with best practices, household, enterprise and agriculture surveys must be conducted at least every five years. The sample size may be relatively small but as Jervens (2013) argues, it is better to survey fifty minibuses a year and get a relatively accurate estimate of earnings and services in the small scale transport sector, rather than conducting an exhaustive transport survey every three decades. These surveys can be used to periodically update GDP, poverty estimates and other statistical indicators. In keeping with the Paris – Accra – Busan declarations, monitoring templates for donor projects should be simple, uniform and require preapproval from the local government. This will ease the burden on statistical officers at the local and national level. Detailed data and metadata (details about the design and specification of data) must be made available online (in a readable format like excel or access) on prearranged timelines. This would help reduce political pressure on statisticians to ‘self censor.’ Kenya has recently made strides in this regard by making all its data available online. Technology can and should be used for effective data collection. Mobile phones, in particular, can minimize costs, accelerate the speed of data collection and make real time data available to the Ministry and the public with relative speed. Ghana is piloting the use of mobile phones for citizen reporting for its World Bank funded Secondary Education Improvement Plan (SEIP) project. Finally, in order to prevent the principal agent problems, statistical officers must be trained to validate data using spot checks and household surveys. Moreover, surveys could be designed to complement and correct rather than substitute administrative data (Sandefur and Glassman, 2014). Free public access to data online should also discourage the inflation of numbers along the data production line. Making these changes will ensure that discrepancies are noticed and addressed early.
5. CONCLUSION
This paper highlights the fact that there are significant gaps in our knowledge about the real situation ‘on the ground.’ We don’t really know the number of children who are attending schools in South Sudan or the number of patients getting HIV treatment in Mali. From a foreign aid policy perspective, the effectiveness of our $134.8 billion official development assistance cannot be assessed in a satisfactory manner. Understanding the failings of our development statistics and the bottlenecks that may be causing them is the first step towards solving the problem. Moreover, wider knowledge of the problem should encourage academicians and development practitioners to proceed with caution when disseminating results. Financing the SDGs must take these challenges into account so that we can develop a more robust framework for data collection, analysis and dissemination. Good data will be essential for monitoring and achieving the SDGs.
6. BIBLIOGRAPHY
ECON+ 