House of Commons - DFID Annual Report 2008 - International Development Committee

DFID Annual Report 2008 - International Development Committee Contents

5 What is DFID's expenditure achieving?

46. In our Report on the 2007 DFID Annual Report we said:

We are concerned that DFID continues to emphasise inputs rather than outcomes—it focuses too much on how much it spends on aid rather than measuring the effects of its aid spending on poverty.[64]

We quoted the Cabinet Office's Capability Review of the Department which found that: "Evaluation of DFID's outcomes is not sufficiently independent. Evaluation of impacts and outcomes needs to be strengthened further to inform future policy making."[65]DFID acknowledges that there are problems in conducting effective evaluation. In response to our 2008 report on aid effectiveness, the Department said that it was:

[…] working to ensure a stronger focus on rigorous evidence of impact […] We recognise that there is an international 'evaluation gap' ie insufficient high quality, rigorous and policy-relevant impact evaluations funded by the international development agencies as a whole. DFID is playing a key role internationally to fill that gap, through a number of complementary activities.[66]

47. The aim of DFID's internal Evaluation Department (EVD) is "to determine relevance and fulfilment of objectives and assess developmental efficiency, effectiveness, impact and sustainability" by carrying out:

formative studies addressing new aid instruments and emerging themes
ex-post evaluations
policy-oriented studies
country programme level studies and examination of themes (eg gender)
studies of sectors (eg basic education) and
performance management quality assurance reviews.[67]

48. DFID's key initiative to improve the quality of its evaluation is the Independent Advisory Committee on Development Impact (IACDI). We took evidence from the Chair of IACDI and two of its members in July 2008. Below we explore its plans for tackling the main challenges to effective evaluation of the Department's work which include changes to the way the Evaluation Department operates.

Independent Advisory Committee on Development Impact

49. In announcing the establishment of IACDI, Hilary Benn, the then Secretary of State said:

The Government is committed to ensuring that the UK's aid budget is used effectively to make a difference to the lives of the world's poorest people and that it represents value for money. As we increase levels of overseas aid, we must be rigorous in assessing the impact of that aid in helping to reduce poverty and change the lives of poor people for the better. [68]

IACDI's terms of reference specify that it has been established to: "Assure the independence of the evaluation function in DFID and the use of evaluation results to enhance the delivery and impact of UK development assistance." IACDI does not commission or engage directly in individual evaluation reports. It is not permitted to become involved in DFID's operational decisions. Its work is intended to complement that of the National Audit Office. The Chair determines the work programme and is responsible for the appointment of members and the organisation of meetings.[69] IACDI is now responsible for agreeing the workplan of DFID's Evaluation Department and the Department's overall evaluation priorities. From this year, IACDI will agree a three-year rolling work-plan for the Evaluation Department. A draft Evaluation Policy was put out to public consultation in December 2008.

50. The Chair of IACDI reports to the Secretary of State annually in writing giving an overview of evaluation work and making recommendations. The terms of reference specify that this letter will be copied to the International Development Committee and will be published. The advice and recommendations are not binding on the Secretary of State but he is required to respond to them. We received the first of these letters on 1 December 2008 and a copy of the Secretary of State's reply on 11 December.[70]

IACDI'S PLANS FOR IMPROVING DFID'S EVALUATION WORK

51. In his first annual letter to the Secretary of State, the Chair of IACDI highlighted the importance of proper evaluation of DFID's work:

We are conscious that high quality, independent evaluation is particularly important for the management and accountability of spending programmes like DFID's that are expanding rapidly and where the links between departmental policies and expenditures and their impact—notably on poverty and achievement of the Millennium Development Goals—are usually indirect and complex. High quality evaluation is needed both to provide assurance that DFID's money is being well spent and so that lessons from experience are learned and applied in future.[71]

The letter goes on to say that IACDI's assessment was that the quality and independence of the work carried out by DFID's Evaluation Department were comparable with other bilateral aid agencies but that it believed "DFID can and should aim for a higher standard".[72] The letter sets out the steps which are already being implemented to address the identified weaknesses, those which will be implemented in the immediate future, and those planned for the next two to three years.

RESOURCES AVAILABLE FOR EVALUATION

52. The Chair of IACDI told us in oral evidence that "there is not a very strong culture of self-evaluation in DFID".[73] His colleague Tony Killick reinforced this: "DFID as an institution is not collecting the information and does not have the systems in place to enable it to reach firm judgements about cost-effectiveness".[74]

53. A comparative study of evaluation policies and practices in a range of development organisations, including DFID, was published in December 2007 by researchers from the Overseas Development Institute.[75] It found that, although impact evaluation was an area of increasing interest for development agencies, there were a number of challenges to carrying it out effectively. It identified a disconnect "between the rhetoric on the strategic and growing importance of development evaluation, and evaluation practice in development agencies." It said that more investment was needed for evaluation, both financial, and more importantly, in human resources and capacity. There were too few staff involved in evaluation; they sometimes had no evaluation experience; and attracting high calibre staff and limiting turnover presented a challenge.[76]IACDI witnesses told us that less than 0.1% of DFID's total programme budget, or 2% of the administrative budget, was spent on evaluation.[77]

54. In the Chair's annual letter, IACDI differentiated between the work being carried out within DFID's Evaluation Department (EVD) and that undertaken across the organisation as a whole. It stated that work carried out outside the EVD needed to be strengthened. It believed that the EVD should take responsibility for oversight, quality assurance and guidance on all evaluation work, for which it will require greater resources.[78] The Chair's letter highlighted a concern to ensure that the "level of administrative resources for evaluation in DFID both inside and outside EVD is adequate for the task" and proposed that IACDI should have a central role in deciding the future budget of the Evaluation Department. It also proposed that more staff should be recruited from outside DFID and that the balance should change so that external consultants played less of a role in evaluation work.[79] The Secretary of State's response points out that the Evaluation Department's funding has increased from £3.6 million in 2007-08 to £5.1 million for 2009-10 and that the number of staff has "nearly doubled since 2004".[80]

55. We agree with the Independent Advisory Committee on Development Impact that, if evaluation is to be carried out effectively within DFID, sufficient administrative and financial resources need to be allocated to this function. We accept that increases have been made in the financial and staffing resources available for evaluation but these remain tiny as a proportion of DFID's overall and increasing budget. We believe that an objective and independent assessment is required to determine the appropriate level of resources. We therefore recommend that IACDI's future role should include advising on the budget and staffing requirements of DFID's Evaluation Department.

INDEPENDENCE OF EVALUATIONS

56. The Comparative Evaluation Study found that "independence is one of the cornerstones of the evaluation function in aid and development agencies" and it broke this down into three components: organisational independence; behavioural independence; and protection from external influence.[81] It noted in relation to the first component that in DFID, the Evaluation Department "sits within the line management structure" of the Department: it is an integral part of the departmental structure, rather than being at arm's length and its Head reports to the DFID management. Other agencies have increased the organisational independence of their evaluation functions by setting them up as external and separate organisations (for example, Sweden) or by them reporting to a separate board, or directly to ministers.

57. One of IACDI's initial focus points has been strengthening the independence of evaluation, although it cautions that this "should not mean isolation from decision makers and there needs to be a balance between independence and engagement".[82] A change to be implemented in the immediate future is that IACDI will have a formal role in agreeing the job description, protocols and performance review arrangements for the Head of the Evaluation Department. Proposals for the longer term include a more senior grade for the post to improve its visibility and "clout"; and that the reporting line should be direct to the Permanent Secretary.[83]

58. As a step towards this, DFID has already made the post accountable to the Director General of Corporate Performance.[84] We also discussed with IACDI in oral evidence making the Head of the Evaluation Department an "end of career post" which may allow the incumbent greater freedom to be critical of the organisation, or appointing an individual from outside DFID to the post.[85] One of IACDI's longer term proposals is that future heads of evaluation should be appointed "on an understanding precluding employment elsewhere in DFID, and the contract should be for a fixed term".[86]

CONSULTANTS

59. In common with most development agencies, many of DFID's evaluations are carried out by consultants. The Comparative Evaluation Study pointed out that this does not ensure independence as "consultants who rely on regular commissions from a particular agency can be under considerable pressure not to be too critical in their analysis" and that, in order to secure contracts, consultants "are not always incentivised to provide objective and critical views of agency actions".[87]

60. We explored the role of consultants in evaluation in our oral evidence session with the IACDI representatives. They agreed that using consultants in no way guaranteed independence and that the quality of the work produced was also an issue. The Chair advocated at that time bringing more DFID evaluation work in-house and using consultants less.[88] This recommendation was included as one of the longer term proposals in the Chair's letter to the Secretary of State.[89]

IMPLEMENTATION OF LESSONS LEARNED

61. In oral evidence, Robert Picciotto, one of the IACDI members, told us that an effective evaluation department ought to be about "learning what works and what does not work".[90] The Chair emphasised that although part of the function of evaluation was to assess whether work had been carried out effectively, the equally important second half was ensuring lessons were learned and recommendations were followed up and that this had been one of the motivations for the establishment of IACDI.[91]

62. The Comparative Evaluation Study noted that communicating, disseminating and following-up evaluations were key functions but ones which were not always carried out effectively. A tension was identified between the need for proper independence of evaluation and the desirability of its integration with the rest of an organisation's activities so that findings can be properly used. In DFID's case it suggested that there was a danger of the Evaluation Department being isolated from the other parts of the Department.[92] It found that DFID performed well in relation to publishing evaluations and using seminars and other methods to ensure evaluation findings were made available to a wider audience. What was less clear was the extent to which the findings from evaluations were implemented and whether evaluation findings led to practices and policies being changed.[93]

63. In an attempt to address deficiencies in ensuring that lessons are learned from evaluations, a number of IACDI's recommendations were implemented immediately. DFID has agreed that a departmental director should now be made responsible for following up each evaluation report,[94] and DFID released a note on follow-up action taken as a result of the evaluations carried out in 2006-07 to coincide with publication of the letter from the IACDI Chair to the Secretary of State.[95] The Head of the Evaluation Department also published the first of his new annual reports drawing lessons and commons themes from EVD reports completed during the year.[96]

64. Lesson-learning is an essential function of evaluation. We welcome the publication of new annual reports to show the extent to which DFID has implemented evaluation findings from previous years and that DFID has agreed to make departmental directors responsible for following up evaluation reports. At a time when DFID's budget is increasing, it is important that spending decisions are taken on the basis of evidence of what works, so that money is used cost-effectively. We expect DFID to continue to build on the progress it has made in implementing these initial recommendations from the Independent Advisory Committee on Development Impact.

EVALUATION OF THE IMPACT OF BUDGET SUPPORT

65. A substantial amount of DFID's funding is now provided direct to partner governments and spent by those governments according to priorities which it agrees with DFID. This form of providing development assistance is known as budget support. In 2006-07 DFID provided £461 million in budget support to 13 countries which represented nearly 20% of bilateral expenditure.[97]Our colleagues on the Public Accounts Committee (PAC) have highlighted the problems associated with assessing the impact of DFID funding provided through budget support to partner governments. Their Report points to some of the benefits of budget support: increased services to benefit the poor in six out of nine countries assessed; increases in the quantity of services delivered; and supporting partner countries to make education and health services free. However, the PAC found that countries which did not receive budget support had seen similar achievements. Moreover, despite using budget support for a number of years, DFID has not yet established whether it is a cost-effective mechanism.[98]

66. The PAC Report pointed to lack of data and technical capacity in developing countries as key factors hindering effective evaluation of budget support. The Comparative Evaluation Study also highlighted that capacity of partner governments was a limiting factor in effective evaluation of impact.[99]

67. When we questioned IACDI about effective evaluation of budget support the Chair told us that this had not formed part of IACDI's initial work programme but that evaluation of budget support would be included in the consultation on the future work plan of the Evaluation Department.[100] We are very pleased to see that budget support has been included on the shortlist of possible topics for evaluation as part of DFID's current consultation on its evaluation policy.[101] We recommend that evaluation of the impact of budget support be given a high priority and be undertaken as early as possible in the new three-year evaluation period which begins in 2009-10.

DFID's analysis of its impact on poverty reduction

68. DFID's Annual Report claims that the Department helps 'to lift at least three million people permanently out of poverty every year'.[102]The Report gives the source of this observation as 'the Collier-Dollar Model of Impact Aid'. This is a model used to calculate the impact of aid levels on poverty reduction in developing countries. The original research was undertaken for the World Bank in 1999,[103] and the DFID-established Economics & Statistics Analysis Unit reviewed and updated that work in 2003.[104]

69. The 1999 World Bank research examined the correlation between aid levels and economic growth, and between economic growth and poverty reduction. In doing so it factored in a number of variables including the aid-recipient country's starting level of income, its distribution of incomes and its policy environment (which makes aid effective in-country). It found that at the levels and distributions of aid at that time, the average cost globally of permanently lifting a person out of poverty was $1,205, but that if aid were optimally allocated between recipient countries the cost could have been only $445. The focus of the analysis, however, was on the marginal effectiveness of aid, which was $1,502 per person lifted out of poverty (marginal costs are higher than average costs because of diminishing returns as aid levels increase), because the purpose of the model was to guide how donors might best allocate additional aid spending.

70. The 2003 analysis by the Economics & Statistics Analysis Unit updated the earlier research and also considered how performance varied between recipient countries and between aid donors. Globally, the 'marginal effectiveness' of aid had nearly tripled between 1990 and 1998 (from 105 people lifted out of poverty per $1 million to 284 people.) However, the research found that that increase in marginal aid effectiveness had been the result of reduced aid volumes during the 1990s decreasing the effect of diminishing returns and a change in the way the local policy environment was scored, rather than as a result of better aid allocations between recipient countries. The UK's marginal effectiveness rate—387 people per $1 million in 1999—was the highest amongst the major bilateral donors. It was also higher than DAC donors overall (275 people per $1 million) and multilateral donors taken together (311 people) (but not as high as some individual multilateral donors—for example the UN Development Programme (443 people)). However, once adjusted for aid volume and policy scoring changes, the UK's marginal effectiveness rate was no higher in 1999 than in 1990 (see Figure 1): 97 people per $1 million in 1990, compared with 84 people in 1999 once adjusted for the reduced volume of aid and changes in policy scores.[105]

71. Such figures are of course a decade old. On the basis of the Government's planned ODA expenditure, a rough calculation suggests that, at the UK's 1999 aid effectiveness rate, it could be lifting 3.6 million people out of poverty in 2008-09.[106]But such a calculation takes no account of: any improvements in the allocative efficiency of UK aid; the effectiveness of its aid projects over the last decade; the likelihood that the average effectiveness is higher than the marginal effectiveness used in such a calculation (factors which would produce a higher figure); nor the greatly increased volume of ODA expenditure (which would tend to reduce the effectiveness figure through the effect of diminishing marginal returns).

72. We therefore asked the Department to provide us with its detailed calculations for the figure of three million people lifted out of poverty, cited in the Annual Report. The Department told us that they used the original Collier-Dollar model to provide more up to date marginal and average effectiveness figures, summing the numbers lifted out of poverty in each recipient country covered by its bilateral and multilateral programmes. It gave us figures for the range of effectiveness levels across recipient countries, but it did not give us the "detailed calculations" we had sought.[107] Instead, DFID told us that the three million people statistic was generated in 2006, based on estimates made then of bilateral and multilateral aid allocations for 2007-08. The model suggested, we were told, that the Department's allocations in that year would help to lift 3.45 million people out of poverty—a figure conservatively rounded down to 3 million for the Annual Report.[108]The Department explained that there would be margins of error in using the model to infer the actual impact of aid on individual countries.[109] The Permanent Secretary told us that, while the Collier-Dollar model was "a well respected piece of research", her "confidence in the estimate gets less the more disaggregated it becomes" [110]^.

Figure 1: Marginal effectiveness of aid (people lifted out of poverty per $1m) by donor 1999-2000

Source: Poverty Efficient Aid Allocations—Collier/Dollar Revisited, Jonathan Beynon, Economic & Statistics Analysis Unit, November 2003, Figures 4.2, A4.4

Note: The Economic & Statistics Analysis Unit paper attributes negative marginal effectiveness levels to the high levels of aid given by some donors to particular countries with low policy scores.

73. In the absence of the Department's detailed calculations, we are left with a reasonably recent but aggregate figure on the one hand, and the original detailed data on the other, which is now a decade old. It is important that the Department has the capacity to make an accurate assessment of the impact of its work and of the UK's ODA expenditure. The Department will need to maintain pressure on the Treasury in the next Spending Review to ensure that it secures a budget that will allow it to deliver its MDG commitments; being clear about what that expenditure produces should be an important lever in those negotiations. Indeed, the Department told us that it produced its assessment of three million people lifted out of poverty for its Comprehensive Spending Review bid. In making that assessment, the Department used the original Collier-Dollar methodology. For future exercises, it said it might produce new estimates using more recent global aid data, and updated poverty and GNI per capita data and that the Chief Economist was preparing a review of research on alternatives to the Collier-Dollar allocation model.[111]

74. Any such new assessment needs to address the likely impact of the current financial crisis and the low value of the pound, both for the Department and for partner countries. The Secretary of State recently described to us in oral evidence how the financial crisis is adversely affecting the economies of recipient countries, which means more development funding will be needed to achieve the same poverty reduction outcomes. He also acknowledged that the purchasing power of DFID's sterling-based budget is being diminished while the value of the pound remains low because aid expenditure is incurred in local currencies, and DFID's commitments to the European Commission's programmes are set in euros.[112] These factors mean that DFID's budget will need to be properly supported, to ensure that the number of people lifted out of poverty does not start to fall. We are examining these issues in greater detail in our current inquiry into Aid Under Pressure.

75. We understand and support DFID's desire to publicise the many benefits which its assistance brings. It is essential that it can accurately assess the impact of its expenditure on poverty reduction, not least to enable it to support its bids to the Treasury for funding under the next Spending Review. It is particularly important during an economic downturn that DFID is able to demonstrate to taxpayers what is being achieved from UK expenditure on development assistance so that public support can be maintained.

76. Our inquiries have shown that the rather bald claim made in the Annual Report, that DFID lifts three million people out of poverty each year, can be substantiated but we would caution that statistics should be used accurately and with proper evidence. Our investigations have shown that this claim is possibly an under-estimate of the Department's true contribution to poverty reduction. We might have been assisted in coming to a judgement if we had been given the information we requested. In its response to this Report, we call again for the Department to provide the detailed calculation for its assessment of the number of people it lifts out of poverty each year. In addition, we intend to examine in detail the results of DFID's planned further research on its impact on poverty reduction.

DFID staffing levels

77. In order to meet Government efficiency targets, the number of DFID home civil service posts has reduced from 1,907 in March 2004 to 1,612 in March 2008.[113] DFID is however unusual in that it has an increasing budget. It is therefore presented with the challenge of using rising levels of funding effectively when it has fewer people to deploy. In our report on the DFID Annual Report 2007, we said:

We accept that DFID cannot be exempt from efficiency targets set for the whole of Government. The Department has made good progress in reducing administrative costs, albeit predominantly in the less tangible form of non-cashable rather than cashable savings. We are concerned, however, that the need to reduce headcount and to make administrative efficiencies, and under the Comprehensive Spending Review settlement to meet a significantly higher cash-releasing efficiency target, will act as a constraint on DFID working in the parts of the world where its assistance is most needed: the poorest countries, often fragile states, which have so far failed to benefit from the vast volumes of international aid. DFID therefore needs to make some very difficult choices about withdrawing from some countries, or some sectors, so that it can focus development assistance where it will have the greatest effect on poverty reduction.[114]

78. World Vision told us that it was "concerned about DFID headcount restrictions in UK and field offices, and its potential impact on programme quality." It believed that this had impaired DFID's capacity to engage in policy dialogue with recipient governments, civil society and other donors, particularly in conflict-affected and fragile states.[115] Similar points were made in our recent inquiry into DFID's new HIV/AIDS Strategy. Witnesses expressed concern that implementation of the Strategy might be hindered by DFID having insufficient staff to take on the necessary tasks and that this was a factor in DFID reducing its direct engagement with civil society organisations working on HIV/AIDS. It was also suggested that DFID staffing constraints affected its ability to collect the data required for evaluations.[116] Collection and evaluation of data is particularly important where DFID is providing direct budget support to partner governments.

79. We concluded in our AIDS Report that:

[…] in the specific context of the new HIV/AIDS Strategy, […] staff reductions at DFID may have reached the point where they risk adversely affecting the Department's ability to deliver its objectives in vital fields such as health and social care.[117]

In our Maternal Health report, we expressed concern that the capacity within DFID's human resources to drive the maternal health agenda was constrained, both in-country and within DFID headquarters. We reiterated, in this specific context, our earlier recommendation quoted above, that DFID needed to take some difficult decisions about withdrawing from some countries or sectors if its funding was to achieve the impact on poverty reduction set out in the Department's objectives.[118]

80. When we explored staff constraints in oral evidence with the Permanent Secretary, she was frank in her response: "Our staff are very pressed, they are working very, very hard. […] we are coping but we are struggling".[119]

81. The impact of staff reductions on DFID's ability to deliver its objectives has been a recurrent theme in our reports. We have consistently praised DFID staff for the dedication and professionalism which they bring to their work and this has not changed. We are, however, alarmed that even the Permanent Secretary is now prepared to acknowledge that the Department is struggling to fulfil its commitments. It would be perverse if the administrative savings achieved by reducing staff numbers led to DFID's rising budget being spent less effectively and with less accountability. We have previously accepted that DFID cannot be exempt from Government efficiency targets but we believe the situation has changed. Our concern now is that DFID no longer has sufficient staff in place to ensure its increasing budget is used most effectively in support of poverty reduction and achieving the Millennium Development Goals. We recommend that the Government urgently reassess the staffing level DFID requires to deliver the objectives which it has assigned to the Department under its Public Service Agreements.

64 First Report of Session 2007-08, Department for International Development Annual Report 2007, HC 64-I, paragraph 11 Back

65 Civil Service Capability Review of the Department for International Development, March 2007, p 20 Back

66 Seventh Special Report of Session 2007-08, Working Together to Make Aid More Effective: Government Response to the Committee's Ninth Report of Session 2007-08, HC 1065, p 15 [Response to recommendation in para 73] Back

67 See DFID website at http://www.dfid.gov.uk/aboutdfid/evaluation.asp Back

68 DFID Press Release, 9 May 2007 Back

69 See IACDI terms of reference available on its website at http://iacdi.independent.gov.uk Back

70 Ev 88 Back

71 Ev 88 Back

72 Ev 90 Back

73 Q 4 Back

74 Q 10 Back

75 A Comparative Study of Evaluation Policies and Practices in Development Agencies, December 2007. The Report author was Mara Foresti, Research Fellow, Poverty and Public Policy Group, ODI. The organisations studied were: DFID, SIDA, KfW, Danida, the EU, the World Bank, IMF, the African Development Bank and Oxfam. Back