WEDNESDAY 17 JANUARY 2001

MR JOHN RANDALL and MRS JULIE SWAN

  121. Good morning. May I welcome John Randall and Julie Swan of QAA and say thank you for responding at pretty short notice to our invitation to come before the Committee. I am afraid, as I said informally before the start of the meeting, we only have a strict hour, so we are going to crack on. As you know, we have almost completed—hopefully today we will have completed—the inquiry into access into higher education; that will hopefully be available very early in February. But today is part of the inquiry into retention, which we believe is a manageable part of the higher education inquiry which we now complete on a good timetable. It is very informal here—and we are not being televised anyway today. It is interesting that sometimes when we get inspectorates of different kinds we do have television!—but not today. We are concerned about this question of retention. It becomes more important, as you know, as we expand higher education. Can you give us something of a backdrop to retention, how much this concerns the QAA. Is it a problem that has got worse? The committee knows that our retention levels are much better than many of the other countries that have a highly developed higher education system. Are there any general concerns about retention—trends, that you have identified—or are you pretty much happy about where we are?
  (Mr Randall) I think it would be complacent to say that one could be happy about it. The figures that were given to you by Professor Mantz Yorke suggest about a 17 per cent non-completion rate overall. I recall from my time as an undergraduate 30-odd years ago that the non-completion rate then was probably about 12 per cent. If you look at the growth that has taken place in higher education in the intervening period, that suggests to me that we are actually not doing too badly, if we are only getting a non-completion rate of 17 per cent from an entry that clearly goes beyond the sort of very top end of the ability range that would have been going into higher education 30 years ago. So, from that point of view, it is quite a good news story if the wastage is that low; on the other hand, any wastage is bad news because it represents a loss in terms of the investment of both public and private resource, in the development of the individual, and, in some cases, if a person from a group that is currently under-represented in higher education has an experience that they would characterise as one of rejection and failure, then that is going to spread amongst other members of their peer group (be it a social-class based group or an ethnically based group), that will do damage to any strategy of widening participation from able people who are not currently represented or who are disproportionately under-represented. From our work we can identify a number of facts that we think will help improve retention. I do not think there is any rocket science in this. It starts with providing students with clear information about what is involved in programmes of study so that they can make informed choices; it continues with an admissions' policy that takes care in trying to match the aptitudes and abilities of potential students to the demands of programmes that are on offer; and, once the student is in higher education, it depends a great deal on the quality of the academic support that is provided, particularly in terms of the feed-back that is given to students from assessed work. Our reports on subject provision in universities and colleges suggest that one of the main complaints from students is about the adequacy and timeliness of feedback from work that they have submitted for assessment. If students are not given good feedback on how well they are doing, then they will be a bit without a compass in the jungle, and any subsequent under-performance or failure, if it comes as a bit of a shock, could be rather demoralising and could be something that would trigger a person dropping out, whereas, on the other hand, if they had been receiving good guidance and good feedback for work that they had submitted, they would be able to address any weaknesses and try to put them right. So I think, overall, our view is that the performance of higher education in this area is not bad, certainly by international comparisons; there is scope for people to do better; and there are things that I hope we are doing, in all of those areas which we have covered in the written evidence, that will help higher education institutions improve their performance.

  122. But are there institutions that are doing better than others of a similar kind? I do not want you to compare a very small college to a very large university, obviously, but, of similar kinds, are there institutions that seem to have procedures and methods that are significantly better practice and more effective? We visited two universities yesterday which had non-completion rates of nine and ten per cent, and some of the findings suggested that half of those were going elsewhere, they were not dropping out of HE. Is there good practice out there that you have identified and are trying to spread across the sector?
  (Mr Randall) Yes, there is. I would be hesitant to characterise it by institution, because institutions are very large and complex organisations and you can have different practices in different departments and different departments within the same institution may have cohorts of students with different characteristics and a sweeping generalisation to say "The University of A is better than the University of B" is probably not very sensible. But the good practice that I would point to would be in providing clear information to students about what the programme involves; about good induction of students in terms of identifying any particular learning needs that they may have; and prompt feedback from assessment. One example of good practice that we have mentioned in the written evidence is something that is happening at De Montfort University, with sponsorship from the DfEE for the written materials; that is, self-analysis kits for students, dealing with the key skills that they would be using in higher education, to establish their degree of confidence in numeracy, in written and oral communication, so that anything that needed to be addressed could be addressed at an early stage. The student who was going in with, perhaps, less confidence than might be needed in terms of presentation of written work or holding their own in seminar discussion, could be helped with that in time for the help to be of benefit to them at an early stage in their studies. That sort of good practice needs to be encouraged.

  123. There is a voice out there, perhaps a voice like Alan Ryan's, that would say that you are part of the problem, that university teachers are so worried about the time and expense and disruption caused by the QAA that they have hardly got time to provide quality education for their first year students. He would rather like to get rid of you, would he not? He says the CVCP would have liked to have got rid of you. Are you a burden, distracting teachers from actually giving quality education, and, ironically, giving less time to the students?
  (Mr Randall) No, I do not think we are. I do not think it is the CVCP wanting to get rid of us, since the CVCP are among the members of the company that constitute us.

  124. I am quoting from Alan Ryan.
  (Mr Randall) Yes. I believe that Alan Ryan said that he thought the CVCP ought to use that position they have to try to get rid of us. That is not, as I understand it, the view of the CVCP. I will ask my colleague Julie Swan to say a little bit about this in a moment, about the feedback that we get from individual academics about the work that we have done that we believe is enhancing their performance. We have produced a large number of codes of practice, which are not things which have been imposed from outside but are distillations of good practice, worked on by academics either by subject (in the case of our subject benchmark statements) or in areas like dealing with assessment, dealing with external examining and so on, where we have had a lot of quiet feedback from people saying, "Actually, this is rather helpful." It is fashionable to go and bash those who appear to be "the demons from outside coming in to make our lives a misery" and anybody would be likely to say that they would sooner spend their time doing something other than dealing with a review team from the Agency, but we also know, from some of the quieter feedback that we get, that, because we place a very strong emphasis on self-evaluation as the cornerstone of the processes that we operate, people find that being prompted to undertake that degree of self-reflection can bring about change that is genuinely owned by those who have to make it work and, because it is their change, prompted by that process of self-reflection, it is likely to be effective. I think that underneath some of the sound and fury that an organisation like ours is always going to attract, there is actually a quieter voice saying, "We think that what is going on is actually helping us." Perhaps I could ask Julie to say something on the feedback we have had of that sort on the sections of our code of practice.
  (Mrs Swan) Perhaps I could just illustrate that fact by reference to a small article which we came across in a journal which appeared towards the end of last year, focusing particularly on the needs of disabled students—an entirely unsolicited article, written by somebody within an institution who had taken one of our small codes of practice on students with disabilities and had found this a very useful tool for talking with colleagues across the departments within the institution about the support they were offering to their disabled students—entirely unsolicited, very, very positive, offering people within institutions a way, perhaps, to cut through some of the bureaucracy with which maybe they are from time to time confronted and to focus on the important questions which can actually make a real difference to individual students' experiences. We are picking up, as my colleague has said, a lot of quiet feedback of that type on the usefulness of the codes of practice in particular.

  125. I would like to come back specifically to completion and your reports. Have you any idea what proportion of your reports actually picks up completion as an important issue? Secondly, have you been able to analyse from the whole range of reports you have done whether, apart from the good practice which John has already indicated, there are specific across-the-board factors. We are told that we need more doctors in hospital for mortality levels; what do we need in universities? Is it, again, the staff:student ratio? Is it increasingly important in the first year? Are there any generic factors which are part of that good practice?
  (Mr Randall) The last appendix to our evidence describes briefly the way in which we currently report, and that does not directly address completion rates. Data on that is collected by the funding council as part of the performance indicators. We are looking at a snapshot in time of how an individual academic department is performing and we will look at the factors that would affect completion under several headings, including assessment, which would deal with the adequacy of feedback that is given, the general support that is given to students, both academic and pastoral, and also the actual achievement of students in terms of the results that they get. That means that it is difficult to pull from those different strands a consistent picture that you could turn into hard numbers. The new method of review—that we have started using in Scotland this year and which we will use throughout the United Kingdom from this autumn—deals with progression as a separately identified issue, and it picks up all of the factors that I have mentioned from the point of admission onwards. That will mean that in future we will have far more focused reporting on progression and its effects on retention. But, ultimately, we do not do the statistical side of the completion figures; that is done by the Higher Education Statistics Agency and by HEFCE. In terms of looking at the across-the-board issues, I am always hesitant to say that everything will get better if only you throw a bit more resource at it. Very often you can make things get quite a lot better by being a little bit more clever about the way in which the resource is used. Clearly, if you have very bad staff:student ratios then it is going to be hard to give the degree of personal attention to students that is needed. But I think that within any staff:student ratio there will always be the opportunity for people to give priority to the things that are most likely to support students, and I come back to this critical question of giving students feedback on their progress through assessment. There can be a lead from the top, given by heads of department.

  126. May I just intervene. You have mentioned that before and I accept that that is one of the key things you are coming out with. On reflection, from the comment you have just made, however, surely the QAA is in that position, having gone into institution after institution, to give that lead, to give that information to the different institutions as to what they should be doing and what level of staff:student ratio is significant. My colleagues will come on much later to differentiate or to try and pull out whether a snapshot, as opposed to progression for your work, let alone the institution's, is the best approach, but can I just stick at the moment with this specific of, should you not be the body giving that guidance to the institution?
  (Mr Randall) I do not think we should be giving guidance on numbers, on resource, from that point of view. We are looking at how effectively an institution is performing.

  127. And then you walk away.
  (Mr Randall) No, we report.

  128. I think what Valerie is trying to get as is: Are you looking at how the institution performs or are you doing it much more selectively, department by department?—so you do not have a holistic view.
  (Mr Randall) No, we do both. We carry out two types of review. We review subject by subject, under commission from the funding councils, who have a statutory responsibility to assess the quality of the provision they fund—and they contract with us to do that. We also carry out reviews of the effectiveness of the management of quality and standards overall at a holistic level, for the institution as a whole. One of the reasons that I am very cautious, when people ask us "Should we not be saying that the university should have £X million more or so many hundred staff more?" is that I do not want to be in a position where every time we find a weakness the excuse is given, "Well, we could do better if only we had more resources. It's not our fault." In most cases there are things that can be done to use resources effectively. We are looking very much at the effective use of the resources that an institution has and reporting on best practice. In terms of disseminating best practice, that is the purpose of the codes of practice that we produce, which highlight good practice in areas like external examining, assessment, programme design and so on. They are the instrument for getting out the good practice to the higher education sector as a whole, so that others can learn from it. The example that Julie gave, that is one of a number that suggests that people are taking notice of the findings that we have got.

  129. Mr Randall, can a shortage of funds and a fall in the unit of funding per student affect the quality of teaching provision?
  (Mr Randall) Of course it can, but the impact that will have will depend on how well and how intelligently it is managed by the institution. In all walks of life we are all confronted from time to time with people saying, "Do it for less." You know, governments will do the same sort of thing: the Treasury will have its limits that are imposed on the public service. There are priorities that are properly taken by government as to where the money gets spent. Once those decisions have been taken, there is then a need to make sure that resources are being used most effectively. There are a lot of developments that have taken place over the last few years that I think enable institutions to operate to a high degree of effectiveness. Every developed and developing country in the world is coping with the transition from a higher education system based on coping with a small elite to one based on mass participation. That means that the resource is being spread a lot more thinly. The response to that is usually a greater emphasis placed on student learning, rather than putting everybody in a big lecture theatre and lecturing to them, and on the use of electronic technologies for accessing data, and those types of efficient use of the time of academics, so that the time they spend with individual students is high quality time.

  130. Even if it has reduced.
  (Mr Randall) Yes. If you think back—and I can probably think back a little further than you to when I was an under-graduate—

  Dr Harris: I am sorry, can I just clarify something?

  Chairman: Give Mr Randall a chance to finish.

  131. I need to clarify something because that was quite a long answer. Presumably you have more opportunity of getting high quality time the more time you have between staff and student.
  (Mr Randall) Every academic job has to balance the teaching responsibilities, research responsibilities and the contribution that academics make to the administration of the department. Their first personal time management task is going to be between those three things. Within the time that they have available for teaching, they would need to ensure that they are using that to the greatest effect. It is self-evident that the more time they have, the more time potentially they have to spend with a student, but I am working, I hope, in a fairly hard-headed way and saying that, if there is a limit to the resource available, our job is to make sure it is used most effectively.

  132. There is an argument that if the funding per student drops, then somewhere, in some remote department in some university, the quality is going to fall. It might still continue to rise elsewhere, for the reasons you have given. Would it not be a freak if nowhere was this quality to drop as a result of funding? But will universities admit that their quality is dropping? Or is there a fear that the first university to admit that will be in a real problem? Do you have the mechanisms to detect and report that it is continuing falls of funding or lack of teaching resources/library staff that is affecting quality?
  (Mr Randall) There is always the risk that that will happen and that is one of the reasons that we are in existence, so that institutions know that we will be coming round and looking at every subject field in which they teach to make an assessment of the quality and to ensure that the standards are not falling. Now, when we report we will sometimes say that the problem is due to a lack of staff or a lack of resource (such as library). If it is clear to our assessors when they undertake the review that there are insufficient staff or insufficient staff with appropriate qualification to sustain a programme effectively, then that will appear in our published report.

  133. My final point on this line is that, clearly, those institutions that offer higher education courses in further education colleges are, by dint of funding and history, unlikely to have the sort of learning and teaching facilities that new universities and, more so, even the old universities will have by accumulation and accretion from the time they existed when funding days were better. Is it reasonable to have a system that reports poor quality, which appears really to say that some of the quality of teaching in some of those institutions is more likely to be due to lack of teaching resources? Is it fair to report that without headlining that it is likely to be due to the lack of resources?
  (Mr Randall) I try to look at this from the point of view of the student or, indeed, from the point of view of the parent. If you are signing up to a degree programme, no matter where you go you should have a reasonable expectation that that would reach a minimum standard and that there would be the resources there to enable that to happen. If it does not, then you need to be warned, and that is a reason for having published reports from us. The further education colleges that provide higher education courses in some cases do it extremely well—indeed, there was one further education colleague that managed to get a complete 24 out of 24 score on one of our reviews, so it can be done—but I am afraid there are more further education colleges that have not done so well. That, I think, has as much to do with the culture of the institution and the critical mass of higher education teachers. Higher education normally takes place in an environment that is informed to an extent by research, where there is a style of scholarship that involves interaction between a number of higher education teachers, often across disciplines, and there is not always the critical mass in a further education college to sustain that.

  Chairman: I think we have to switch questioning a little now.

  134. You have said a little about the different completion rates between institutions but what I would like you to tell us about is what you find about the factors leading to non-completion amongst different groups of students either in your subject reviews or in your institution reviews. We now have the traditional students of 18 to 21, we have mature students and we have part-time students. Have you found different levels of non-completion amongst those groups, and, if so, do you know any reasons why?
  (Mr Randall) We do not have data from which I would feel comfortable in trying to draw reliable conclusions, because what we are looking at when we go into an institution is the performance of the institution, not the performance of the students. We are not examining the students, we are not assessing their work; we are assessing the institution itself, either at the whole institution level or the level of the academic department, and so what we are reporting on is how well the department is doing. The performance of students clearly is a part of that, but we do not do an analysis that is based on the nature of the student population, although it would be quite likely to be commented on in the narrative of a report if it was a significant factor.

  135. You do have data on non-completion, presumably. Is it possible to relate that data to the types of students with whom you are dealing?
  (Mr Randall) Not from the data that we have. The last appendix to our written evidence discusses briefly the scope and limitations of correlating the data that we have from our current method of subject review with the data collected by the statistics agency on completion.

  136. Do you not think it is a factor in assessing how a department performs to look at how it deals with different types of students? Because there are institutions which have been used to dealing with traditional students which are now dealing with a much wider variety of students from different backgrounds. Surely it is part of that department's teaching effectiveness to look at how it deals with, how it supports, how it manages to move through the system to completion for those different types of students. It is not enough to say, "Sorry, we wash our hands of those."
  (Mr Randall) No, and we do not say that. The basis on which the review proceeds starts from the self-evaluation carried out by the institution and the judgment is made against the objectives that the institution has said that it is seeking to secure. That will be in the context of the overall mission of an institution. Where an institution is setting out to pursue a wider participation agenda, where it has a policy that is seeking to give access to higher education, to groups other than the traditional three `A'-level school leaver, they will design their programmes and their support mechanisms accordingly and we will be making judgments on how effective those are. We will do those in relation to the objectives set by the institution, which, in turn, will reflect what they are seeking to do with the type of student cohort which they have.

  137. I understand that, Mr Randall, but does it not then follow that, if an institution has not set itself the objective of widening access, that is not then part of your assessment? That is what you are telling us, effectively, is it not? If it is not part of their mission statement, you do not assess it.
  (Mr Randall) We are dealing with institutions that have a substantial degree of autonomy, which have a freedom to determine the nature of the teaching that they will provide, and there is not any central diktat that says every institution must do certain things—whether it is every institution must pursue a particular type of access mission or every institution should be a research-led institution. I think, currently, most institutions would regard themselves as having some responsibility to help implement a widening participation agenda—and there are funding mechanisms that the funding councils have put in place to try to promote that—so, clearly, we will be having regard to the nature of the student cohort that is entering in all institutions in coming to the judgments and writing our reports.

  138. If we can move on from that point, John, I want to look at something which has fairly traditionally been central to your role and that is looking at teaching structures and strategies themselves and how they affect the issue of completion. In the written evidence that you have given to us, you have given, I suppose, two cheers for modularisation, because you say it could bring many benefits, but you are concerned about the amount of time from the student perspective that preparation and assessment of the module may give as opposed to reflection. As part of your assessment, do you look at the overall structure of courses in terms of modularisation and the extent to which students have the necessary background information to cope with that?
  (Mr Randall) We look at the structure of a programme in relation to the objectives that it is trying to achieve. I am personally very strongly of the view that you need to take a holistic view of what you are trying to do with a higher education programme and look at the overall outcomes that are achieved. One of the worries that I have—and I think it is shared by quite a lot of people in higher education—is that we may have got to a position where we are over-assessing the students, where the division of the course into relatively small elements, each of which has to be assessed to contribute towards a final degree award, may mean that the amount of assessment that is taking place to add, perhaps, one per cent or two per cent of the total credits towards the eventual degree, is disproportionate, and the amount of time that the academic staff are spending in that summative assessment and in all of the moderation that will go with ensuring that that is being carried out on a consistent basis, could be disproportionate, and we might be better off if there was more formative assessment that was giving feedback to students, encouraging them in their learning, and the summative assessment was perhaps in slightly larger chunks, so that the structure was promoting learning and reflection and was not so dominated by a series of formal assessments, perhaps cumulatively adding up to rather more than is strictly necessary to make the judgments about student performance.

  139. That would certainly chime in with much of what we have heard. I think one of the concerns that academics have expressed to us—and we heard it yesterday at Kingston—is that, particularly in the first year, particularly where you have the varied groups of students which Helen has talked about coming into large survey classes, maybe being taught by the top professors but maybe being taught by teaching assistants, with very little background available to the teachers as to what background knowledge the students are bringing to the table, there is very little time to address the sorts of strategies to keep those people in the course which would affect non-completion. Would you regard those as fair comments?
  (Mr Randall) Yes.I think that the answer to that lies very much in the things that happen before the students are in that class and the teacher is teaching them. That comes down to making sure the students have a very clear idea of what they are signing up to and trying to ensure that in the admission process the people do have the necessary knowledge, skill and understanding base that will enable them to progress successfully in that programme.