House of Commons - Science and Technology

Hansard

Archives

Research

HOC Publications

HOL Publications

Committees

Select Committee on Science and Technology Sixth Report

3 Objectives and performance measures

24. The OSI is responsible for meeting one of the DTI's Public Service Agreement (PSA) targets as amended in the Spending Review 2004 settlement. PSA 2 requires the department to:

Improve the relative international performance of the UK research base and improve the overall innovation performance of the UK economy including through effective knowledge transfer amongst universities, research institutions and business.

The DTI state that as a consequence of the responsibility of the OSI for "the health and standing of the UK research base as a whole", the OSI "has a key role in aligning Science Budget priorities with those of other funders to deliver two broad outputs that derive directly from this PSA target".[45] These outputs are:

Output 1: A healthy UK research base
Aim—to maximise the impact (of the Research Councils' and Academies' investment of the Science Budget) on maintaining and improving the UK's Research Base
Output 2: Better exploitation
Aim—to increase the contribution made to improving exploitation of the Research Base to meet national economic and public service objectives.

Beneath this are two further levels of outputs aimed at the Science Budget and then at the individual Research Councils and associated bodies. Examples of principal outputs to be measured in the case of output 1 include the number of publications, citations and PhD students; and for output 2, collaborative research and co-operative training. The indicators used in connection with these outputs are derived from the work carried out by Evidence Ltd for the DTI, which was commended in reports by our predecessor Committee on OSI scrutiny, who also expressed doubts that the new measures introduced as part of the Science and Innovation Investment Framework 2004-2014 would be "as clear or as useful" (see paragraph 28 below).[46]

The OSI performance management system

25. In the last two years there has been a major change in monitoring the contribution to the PSA targets made by the Research Councils and Academies with the development of a new performance management system. This was introduced by the OSI in order to provide "a firm, evidence-based foundation" for the long-term planning demanded by the Ten year Framework for Science and Innovation[47] and was outlined in the DTI Science Budget Allocations 2005-06 to 2007-08, published in May 2005. It was intended to provide "a more robust mechanism for translating the strategic priorities for the research base into specific aims and objectives for the Research Councils and Academies",[48] offering the opportunity to "allow the balance of investment across the Science Budget to be adjusted in response to a more strategic view of new priorities".[49] The system involves measuring performance against a number of indicators derived from the 2004-2014 Framework aim of making the UK the most attractive place for science and innovation. It has three elements: Delivery Plans for each Research Council and for RCUK, setting out the key deliverables in the next three years; a scorecard recording the deliverables, targets and milestones to be achieved; and output frameworks which take the outputs and measure them against "characteristics of interest" (scale, quality, sustainability, agility, productivity and user focus, in the case of output 1; scale and quality for output 2). Data is collected and published on an annual basis.

26. The system has been in operation since 1 April 2005. In September 2006 the OSI published the first Annual Report on Output Frameworks which aimed "to outline the baseline data, comment on any particular aspects for future monitoring and summarise the current Research Council position".[50] It did this by presenting OSI data in a matrix and by then commenting on the data provided by the Research Councils. It is notable that in several cases metrics are still under development or the OSI has concerns about their usefulness. For examples, metrics are not yet developed for the rate of change in publication numbers and in PhDs awarded in relation to identified priorities. In the case of joint publications between researchers and industry, the OSI is not collecting the data because research has indicated that to do so "would involve significant costs and would not be a reliable indicator of research base-business interactions".[51] A similar argument is made for indicators for the level of business confidence in university knowledge transfer activities where the possibility examined by OSI has left the OSI with "serious concerns over how meaningful or robust indicators would be" and with work to do in identifying alternative indicators.[52]

27. The OSI stresses in the introduction to the Annual Report on Output Frameworks that "the output frameworks are not static documents, as the metrics are evolving and open to revisions and modifications".[53] We accept that it is necessary to adapt the output frameworks used to measure the performance of the Research Councils as it comes fully into use in order that it remains a practical and useful document. We are concerned, however, that the measures should be firmly established in order to allow comparison from year to year. There must not be too many changes once the system has been allowed to bed down. It would be helpful for the OSI to highlight in the introduction to future output frameworks reports exactly which metrics have been amended and which are causing difficulties in terms of data collection or meaningfulness.

The DTI Performance Report

28. In addition to the output frameworks, the OSI's performance against the PSA target is assessed in the DTI Autumn Performance Report against 25 indicators across five attributes of science and innovation: world-class research at the strongest centres of excellence in the UK; sustainable and financially robust universities and public research institutes; greater responsiveness of the research base to the needs of the economy and public services; increasing business investment in R&D and increased business engagement; and a more responsive supply of science, technology, engineering and maths skills to the economy.[54] A footnote to the Autumn Performance Report 2006 notes that "These attributes also provide indicators of progress against the measures set out in the ten-year Science and Innovation Investment Framework".[55] There is no obvious other indication in either the Performance Report or the Output Framework document of the relationship between the two documents. Nor is there a readily-accessible list of targets and output performance measures on the OSI website. There is room for improvement in the transparency and presentation of information about how the performance of the OSI and its associated bodies is to be assessed across the board and we urge the DTI and OSI to address this.

29. The overall assessment of progress against SR04 PSA 2 is that it "remains on course".[56] Significant challenges, however, are identified in the areas of "raising business R&D and increasing the supply of science, engineering and mathematics skills available to the economy".[57] Both these areas were highlighted in the Science and Innovation Investment Framework 2004-2014 which established a headline ambition that public and private investment in R&D should reach 2.5% of GDP by 2014 and placed great emphasis on measures to improve the teaching and learning of science, technology, engineering and mathematics (STEM) subjects at all levels.

30. There are difficulties in measuring both these outputs. For the indicators used to measure the supply of skills to the economy, the most recently available data for one of them is for 2003 and alternative methods of collecting data for the other are being explored as the dataset stopped in 2002. There is therefore no effective way of assessing the up-to-date position of the UK in relation to the supply of skills. The most recent data for business R&D as a share of GDP is set out in the table below:Table 1: Business R&D as share of GDP

Country 1988 1993 1998 1999 2000 2001 2002 2003 2004 2005

Canada 0.77 0.9 1.08 1.08 1.16 1.27 1.09 1.03 0.98 1.03

France 1.33 1.48 1.35 1.38 1.36 1.41 1.43 1.36 1.36 N/A

Germany 2.02 1.65 1.57 1.7 1.75 1.75 1.75 1.73 1.75 N/A

Italy 0.7 0.6 0.52 0.51 0.53 0.55 0.54 0.55 0.55 0.56

Japan 1.9 1.87 2.1 2.1 2.12 2.26 2.32 2.36 2.37 N/A

UK 1.40 1.33 1.17 1.23 1.20 1.19 1.17 1.12 1.08 1.08

USA 1.92 1.78 1.95 1.98 2.04 1.99 1.87 1.81 1.79 N/A

OECD average 1.55 1.42 1.49 1.52 1.56 1.58 1.54 1.51 1.53 N/A

Source: ONS for UK, OECD remainder, Annual Performance Report 2006, p 15

These figures are worse than expected because they were revised by the Office of National Statistics "in the light of new information".[58] Previous data had suggested that there had been an improvement in the last two years but the new figures show that business enterprise research and development expenditure (BERD) has been falling steadily since 1999, although we note the time lag in these figures of up to a year which makes it hard to be precise about the current state of R&D investment and the effect of Government policies.[59]

31. In addition, there are concerns both that the BERD figures are not capturing the full extent of R&D in the UK and that they are in any case not sufficient in themselves to give a rounded picture of innovation. The Minister of Science suggested that there was too much emphasis in the measurement on the manufacturing sector and that the R&D target could be widened to capture R&D conducted in services.[60] Sir Keith O'Nions argued more generally that BERD was just "a single target which can only capture a very small part of the performance across the landscape" and that "there is a lot more thinking to be done as to what a broader range and basket of measures should be in relation to productivity and innovation performance".[61] He elaborated in discussing the effectiveness of knowledge transfer efforts that:

32. It is common sense that performance measures have to be properly focussed on the right attributes in order that they may show a true picture of the state of innovation in the UK. The challenge of devising suitable measures can also be seen in the gaps in the current systems of measurement. There is clearly no point in having indicators which cannot be measured, nor outputs which focus on narrow issues to the detriment of significant ones. Sir Keith O'Nions told us that, whilst there was no formal Treasury review of output targets, the OSI was trying to address the recognised difficulties by "looking at a whole number of possible, sensible performance measures in close discussion with the Treasury, particularly around that area [of the percentage of R&D expenditure] whilst reviewing all of them".[63] He expected this "will most certainly come to a head at the time of the allocations of the budget, probably next October".[64] We welcome the work going on to identify meaningful and accessible output measures for the OSI and the Research Councils. We expect to receive updates as the work progresses in this area and before the changes have been finalised.

THE 2.5% TARGET FOR R&D

33. Whatever measures are decided upon to assess progress, the Government is, so far, not on course to achieve its 2.5% target for R&D as a percentage of GDP by 2014. This target is central to the ten year science and innovation strategy and the Minister for Science told us that "it remains a priority".[65] Indeed, it is highlighted again in the Budget Report 2007 where it is described as a "commitment".[66] While there are other ways in which innovation and productivity growth might be measured and a single target such as this might not be sufficient on its own, it would be a highly significant for the 2.5% target to be missed. In its response to the OSI Scrutiny 2004 report, the Government told our predecessors that raising R&D to 2.5% "is a challenging target for the UK, but one which is feasible given the UK's starting point and the strengths and structures of its science and industrial bases".[67] Moreover, it was "arrived at following consultation with business and has been supported by business subsequently as a helpful guiding ambition to focus public and private investments in future years."[68] We should like to be more confident of the likelihood of achieving this target. We recommend that the OSI, in its response to this Report, make a clear statement on how it intends to meet the target of 2.5% as a percentage of GDP within its current strategy. In addition, although we appreciate the logic of including service sector R&D within the scope of business R&D, we believe that the 2.5% target would need to be revisited if the basis of measurement were changed.