Student Retention

Report on the joint HESA-HERDSA national consultative workshop

held on 11 September 2007, Emperor’s Palace

OR Tambo International Airport, Johannesburg

December 2007

Contents

Part I:	Facilitator’s report and reflections

Introduction……………………………………………………………………	1

Purposes and expectations …………………………………………………	1

Key points from the presentations and discussions ………………………	4

Decisions on the way forward ……………………………………………...	6

Reflections on the way forward …………………………………………….	7

Part II: Summary of methodology and case studies

HERD-SA methodology ………………………………………………….….	10

Case studies presented at the workshop ……………………………….….	13

2.1	University of Johannesburg ……………………………………….…	13

2.2	Central University of Technology .……………………………….….	14

2.3	University of Fort Hare ……………………………………………….	15

2.4	Tshwane University of Technology ……………………………….…	17

2.5	University of Cape Town ……………………………………………..	18

Appendix 1: List of Participants…………………………………………………	20

Appendix 2: Reflections on the HERD-SA Project……………………………	21

Acronyms and Abbreviations ……………………………………………………	24

Part I:	Facilitator’s report and reflections

Professor Ian Scott

1. Introduction

The workshop was significant for two main reasons: it was the first Higher Education Retention Data for South Africa (HERD-SA) event to be held under the auspices of HESA, and signalled the parties’ faith in the project and willingness to co-operate on such work; and it was taking place close to the end of the HERD-SA grant period (which is 31 December 2007) and thus represented a unique opportunity to report and reflect on what the project had achieved and consider how the work could be taken forward in future.

The participants included Professor Roger Ludeman, the USA-based project director, Professor Amanda Lourens, the South African project leader, Hanlie Griesel from HESA, representatives from the participating institutions, and representatives of SAAIR, NASDEV, SAASAP. Apologies had been received from Dr Rosemary Hayes of the University of Oklahoma, the Department of Education (DoE) and the Council on Higher Education (CHE). A list of the participants is attached at Appendix 1.

Part I of the report aims to provide a brief account of key points raised in the workshop, with particular reference to matters related to the main challenges identified for the project and the way ahead. At HESA’s request, it also offers some reflections on these challenges by the facilitator. Reflections on the project by the director, Professor Ludeman, are included at Appendix 2.

Part II of the report, compiled and edited by Amanda Lourens, provides a summary of the formal presentations made by the South African project leader and institutional representatives.

Special thanks are due to Hanlie Griesel for organising the workshop and coordinating the production of the report.

2. Purposes and expectations

The purposes of the workshop were as follows:

to review the HERD-SA project and current results;

to discuss the value of collaborative institutional research in the area of student retention and the role of HESA as sector organisation;

to consider best practice in reporting student retention data; and

to discuss the continuation of the HERD-SA project in South Africa and the critical issues which must be addressed.

In the opening discussion, it was agreed that a related objective, in considering future possibilities, was to consider other projects with similar or complementary aims, with a view to co-operating with these where possible. Such projects included the CHE retention project and the SAQA database project.

At the outset, all participants were requested to outline their expectations. The great majority saw the main desired outcome of the workshop as establishing a basis for taking the work of the project forward, beyond the Kellogg grant period. The participants’ views of the aims of future work could be grouped as follows:

standardising definitions and the presentation of retention and graduation data across the higher education sector;

conducting and stimulating research on factors affecting retention and throughput;

using such research as a basis for effective interventions to improve student progression; and

incorporating retention data analysis, research and interventions into institutional strategic planning.

3. Key points from the presentations and discussions

Roger Ludeman

In his review of the origins and progress of the project, Professor Ludeman reflected on obstacles that had been encountered and congratulated the project team and the participating institutions on all they had achieved. He indicated that all the original project goals had been met, and was confident that the funders, the Kellogg Foundation, would be happy with the outcomes. He reminded the group that South Africa is not alone in having concerns about retention and graduation; in the USA, for example, one in four freshmen drop out, and there is a 53% overall non-completion rate. Familiarity with other countries’ approaches to data analysis and presentation, and affiliation to a project such as the Consortium for Student Retention Data Exchange in the US, could well help to advance the work in South Africa.

Professor Ludeman has produced a reflective report on the project that is highly encouraging in terms of what has already been achieved and the capacity that has been developed for further progress. Professor Ludeman’s report is attached at Appendix 2.

Amanda Lourens

In her review of achievements and results to date, Professor Lourens outlined the two phases of the project. The first phase had focused on standardising concepts and definitions. Co-operation with SAAIR had played a key role in facilitating and disseminating this work. The second phase focused on cohort studies. Aggregated findings for undergraduate three-year qualifications cohorts of the ten participating institutions – such as that about a quarter of first-time-entering students had exited before or by the beginning of the second year, 30% passed under 50% of their first-year subjects, and only 21% graduated in minimum time – confirmed the significance of the retention challenge and the importance of producing data to highlight and help to understand this. While qualitative research on retention was not within the scope of the current project, its importance for improving student performance was fully recognised.

Ian Scott

This presentation focused on selected findings from the CHE’s Improving Teaching and Learning for Success (ITLS) project. The first phase of this project – the central aim of which is to make a case for improving the status and recognition of teaching in higher education, as a means of improving graduate output – includes analysis of sector-wide cohort data produced by the DoE. Like the HERD-SA data, the findings of the ITLS quantitative analysis indicate serious attrition and wastage of human and material resources in higher education. In contrast with the US and other developed countries, the wastage is all the more significant in that it occurs in an environment of low and racially skewed participation rates. The data underscore the need for further analysis and qualitative research as a basis for implementing educational strategies for improving teaching and learning.

The case studies

Case studies were presented by project members from five of the participating institutions: Ashraf Conrad (UCT), Marthie Cronje (UJ), Dr Marie Fowler (TUT), Professor Rod Bally (UFH) and Dr Cay van der Merwe (CUT). The case studies were selected to give an idea of the range of approaches being used in the project.

The individual case studies are summarised in Part II of this report. Key general points that emerged included the following:

While the goals were largely shared, the different institutions’ teams had adopted significantly different approaches to data analysis, and showed a high level of expertise and creativity in shaping their work in accordance with what data were available and their understanding of their institutional contexts.

The range of approaches adopted by the participating HEIs suggests that there are different needs and challenges in different institutions. While standardisation of concepts and definitions is necessary and valuable, and while there is a great deal to be gained from the experience and creativity of other institutions’ approaches, it is evident that different forms and ways of presenting data may suit different institutional contexts and phases of development in teaching and learning. There are implications in this for capacity building in the institutions (see Section 4).

Good data, with strong educational implications, are now available in a number of institutions. A key question, then, is to what extent the analyses are being or will be used to address the important challenges they reflect. It was evident in the case studies that responsiveness to the data was uneven across the institutions. While this could be ascribed to various factors, including institutional mission and culture, the extent to which research teams and projects could influence institutional responsiveness was a key question for future development.

Yunus Ballim

Professor Ballim, DVC Academic at Wits, had been asked to comment on the challenges arising from the workshop and the project as a whole. His commentary included the following points:

There is a growing awareness of wastage – of human as well as material resources – in the higher education sector.

However, the problem is highly complex, and it is essential to recognise this complexity. For example, there are key tensions between short-term and long-term advantage. The cost of failure and wastage in the system is already on the agenda, but institutions nevertheless face short-term financial disincentives to investing in longer-term improvement strategies.

Notwithstanding financial implications, the case for improvement cannot be made on a purely utilitarian basis. Under-performance in the higher education sector has a range of effects on the country as a whole. At the same time, the functions and culture of universities are complex: the challenge cannot be met by dictate or quick fixes without threatening quality and standards. Systemic issues have to be addressed, including testing our assumptions about curriculum.

The complexity of the challenges is, however, no reason for apathy. The status quo is not an option, and projects like HERD-SA should have a key role in facilitating strategic thinking about improving performance.

4. Decisions on the way forward

The view that the work of the HERD-SA project needed to continue in an appropriate form became apparent in the forward-looking nature of much of the input and discussions at the workshop. Accordingly, at the request of the participants, Professors Ludeman and Lourens were asked to present their views on future possibilities, and Professor Lourens was invited to put a broad proposal to the workshop.

Professor Ludeman confirmed that HERD-SA would end on 31 December 2007 as scheduled, at the end of the Kellogg grant period. He reiterated that the project had achieved all its original goals, and he was confident that the Kellogg Foundation would accept the final project report, due to be submitted in January 2008. He expressed his personal support for the establishment of a new project and working group that could capture the energy generated by HERD-SA and take the work forward, as the developmental process had in fact only just begun. He recommended that, in building up a student research agenda, a new group might consider affiliating with the CSRDE in the US, evaluating existing research instruments for use in South Africa, and exploring new funding possibilities with US foundations. The workshop offered a strong vote of thanks to Professor Ludeman for his central contribution to HERD-SA.

In general discussion, the participants agreed that taking the work of HERD-SA forward was necessary and justified in three main respects: a) to continue the work of gaining acceptance of standardised concepts and definitions relating to student progression; b) to continue developing and disseminating effective ways of analysing and presenting student progression data, including research on factors affecting retention; and c) to explore ways of facilitating the use of student progression research for improving student learning and retention.

In response to a composite proposal by Professor Lourens, the workshop agreed unanimously on the following:

Key aspects of the work done through HERD-SA should be disseminated through appropriate forms of publication and through the presentation of institution-specific findings, which might be regarded as confidential, to the institutions concerned.

A new project, building on HERD-SA, should be initiated as soon as possible, preferably to commence early in 2008 to avoid discontinuity.

The aims of the new project should be worked out in detail as the core of a project proposal. On the basis of discussions at the workshop, the main aims would include:

continuing the development and dissemination of relevant definitions and standardisation of concepts;

continuing research on student retention and graduation data with the objective of refining existing approaches to the collection, analysis and presentation of such data and developing new approaches;

researching and developing ways of promoting the effective use of student retention and graduation data for improving student performance in the institutions; and

assisting with building institutional capacity to analyse and interpret student progression data.

The new project should draw primarily on the skills and experience of the HERD-SA group, should ideally have representation from relevant national higher education bodies including SAAIR, SAASAP and NASDEV, and should seek to co-operate with the DoE and CHE.

It should be proposed to HESA that the project be run under HESA’s auspices as a ‘Community of Practice’. The timeline for the development of a proposal in this regard would be end October/early November for inclusion in the agenda of the HESA Executive Committee meeting on 15 November 2007.

The workshop nominated a working group to take the matter of the proposed new project forward, and to draft a project and funding proposal. At the request of the participants, Professor Amanda Lourens agreed to convene the working group in the first instance, pending further decisions on how to proceed. The people nominated to the working group were:

Cay van der Merwe (CUT/HESA)

Hanlie Griesel (HESA)

Mariaan Klopper (NWU)

Prem Coopoo (Wits)

Ian Scott (UCT)

Glen Barnes (UKZN)

Brighton Mupangavanhu (NASDEV)

The working group was mandated to co-opt other members as it saw fit.

5. Reflections on the way forward

These reflections are offered from an educational development perspective that is primarily concerned with the improvement of student learning in higher education and a consequent improvement in the production of an appropriate mix of good graduates. Since research into student progression should be a key means of focusing attention on this challenge, the work done by HERD-SA, as well as the prospect of a new project to take the work forward, is highly valued.

The reflections are intended to offer suggestions that may be considered by the proposed new project’s working group as ways of approaching or strengthening the work. The suggestions are drawn from experience in the sector as well as from points arising from the HERD-SA project and the workshop.

Especially since South African higher education does not have a strong tradition of using management information, particularly data on student progression, for educational development purposes, there is a clear need for growing sophistication in the collection, analysis and presentation of such data. The strengths of the HERD-SA group include expertise and creativity in relevant methodologies, familiarity with MIS in South Africa, links with system and institutional planning, relationships with student affairs bodies, and insights into relevant educational issues.

To endorse points made at the workshop, it is strongly recommended that the proposed new project should build on the strengths of the HERD-SA group and thus be focused primarily on its core expertise in data analysis and research as its main contribution to the improvement agenda.

However, ensuring that such research is put to good use in the institutions is vital for improving the educational effectiveness of the higher education sector and its ability to meet national needs. The importance of this is underscored by, inter alia, the fact that shortages of high-level skills have been identified as a major obstacle to economic growth and development. According to the discussion at the workshop, the HERD-SA group sees the need to take up this challenge as a key element of its responsibility.

However, in accordance with its terms of reference, the focus of the HERD-SA project has been on data analysis and research rather than on educational development per se. In order to take on a more pro-active role in promoting the use of student progression research for informing educational initiatives, the new project team will need to build capacity to influence educational practice in the institutions, by some broadening of its membership and/or by linking up with a wider network that is dedicated to improving teaching and learning.

The following suggestions are offered as possible ways for the group to be actively involved in improving teaching and learning without diluting its core expertise:

In the first instance, institutional experience suggests that the way in which performance information is analysed and presented can in itself significantly influence its likelihood of being productively used. Impact and clarity of implications seem more important in this respect than volume. As one senior executive of a university with an extensive MIS recently put it, “The deans are swimming in data but very little of it is put to use”. An educational development perspective suggests that, to have impact, data presented to educational managers and academic staff should, inter alia:

refer as explicitly as possible to institutional (or national) strategic goals;

be at a level of aggregation/disaggregation that is appropriate to the audience; too high or too low a level often obscures key patterns that can point to practicable remedial action; and

be able to identify critical points of breakdown in the system flow as possible indicators of underlying problems and as pointers to fruitful areas for qualitative research.

The implication of this for the HERD-SA group is that the proposed new project could make a key contribution by identifying a range of forms of data analysis and presentation that would assist institutions to select approaches that suit their particular needs, and by workshopping the options across the sector. Moreover, a key aim could be to ensure that a clear sense of how data can be productively used, and by whom, should inform the architecture of the information system.

A central issue for the group could thus be, in Hanlie Griesel’s words, “What questions should we be asking to maximise system-wide benefit from student retention data?” A good start has been made in HERD-SA but there is much scope for increasing sophistication, which the group should be in a good position to take forward.

Secondly, institutional experience also indicates that retention-related challenges can differ markedly not only between institutions but also between faculties or programmes in the same institution. For example, academic exclusion resulting from failure may be the primary cause of attrition in, say, an Engineering programme, while dropout “in good academic standing” for financial reasons may be the primary cause in, say, a Humanities programme where inadequate financial aid is available. This has various implications.

In the institutions, it is important that data analysis can be undertaken at faculty or programme level, in order that specific factors affecting retention can be identified and addressed locally. Since institutional research (IR) units often do not have the resources to service this level adequately, and since it is in any case desirable for analysis to be engaged with where the responsibility for teaching and learning actually lies, it is important that capacity for undertaking and interpreting performance analysis be built at faculty or school level. This calls for the establishment of structures that link central IR expertise and faculty bodies in a common agenda of analysing and improving the educational process. In practice, the effectiveness of such structures or teams often also relies on drawing in educational development specialists, since the necessary educational expertise may not be in sufficient supply among the regular faculty.

The reality that improving teaching and learning has to happen on the ground, in the faculties and departments, clearly has implications for the HERD-SA group as well. Group members may be able to take a lead in motivating for effective structures in their own institutions, but it may well be the case that collective action, particularly under the auspices of HESA, would carry far more weight, particularly in relation to planning and implementing capacity building.

The need for close co-operation between data analysts and education specialists also has implications for the composition of the new project group. It is recommended that the group consider including a small number of education specialists or regular academic staff with a major interest in teaching and learning, not to dilute the core focus but rather in the interests of translating research findings into effective educational policy and strategies.

Thirdly, and perhaps most importantly, it seems to be the case that the availability or even the active presentation of good data to mainstream academic structures is no guarantee of consequent developmental action. There are numerous examples, one being the deans ‘swimming in data’ referred to earlier. There are also instances where well-intentioned executive efforts to demand compliance with success-rate or output targets have had unforeseen negative consequences. The lack of appropriate responsive action in such cases is a manifestation of the general difficulty of improving the effectiveness of teaching and learning in higher education. It is beyond the scope of this report to discuss the reasons for this phenomenon, which is experienced in many systems, but they include well-known contestation over purposes, priorities, roles and accountability in higher education.

The effectiveness of any teaching-and-learning improvement agenda depends ultimately on positive engagement by academic staff. The complexity of this matter calls for a comprehensive approach to addressing it. It is clearly beyond the capacity of any single group to deal with such systemic issues, so the success of the new post-HERD-SA group in contributing to the development of teaching and learning is likely to depend on its establishing a valued role in a wider network that understands the dimensions of the challenge and is committed to improvement.

Outside of the DoE, the two national bodies that are in a position to establish such a network, separately or jointly, are HESA and the CHE. The HERD-SA group’s decision to request ‘community of practice’ status under the auspices of HESA thus makes sense. There are some positive signs of increased activity around teaching and learning in the sector bodies, so it is hoped that the new group will be able to contribute its particular expertise to the development of a coherent change strategy that will result in measurable improvement of the outcomes of the sector in future.

Part II:	Summary of methodology and case studies

Professor Amanda Lourens

1. HERD-SA Methodology

The purpose of the HERD-SA project was to standardise the measurement of retention rates in terms of South African definitions, to develop the appropriate methodology for reviewing and presenting retention data and to develop a profile of student characteristics from the data collected/identified and determine what, if any, relationships the data may have to student retention. The following institutions participated in the HERD-SA study:

Tshwane University of Technology

University of the Witwatersrand

North-West University

Central University of Technology

University of the Free State

University of Cape Town

University of Johannesburg

University of Fort Hare

University of Limpopo

Cape Peninsula University of Technology

Each institutional team comprised representatives of staff in student affairs, strategic planning, and/or institutional research.

Only three-year undergraduate first-time entering student data for the period 2000 to 2003 were used for the purposes of the HERD-SA project.

Nine institutions used HEMIS data to extract the data for reporting purposes. The first part of the project comprised the finalisation of the definitions and standard reporting for each participating institution. The second part of the project focused on cohort studies whereby student progress was traced over a number of years. The profiles of second year dropout students (i.e enrolled for three-year qualifications that do not continue with their second year of study) were determined using variables that may be associated with second-year dropouts.

A total of 177 406 records were used for the statistical analysis in determining the profile of a second year dropout student as dependent variable (i.e. students who did not return in the second year of study and therefore dropped out during the first year of study). From the participating institutions, 27.2% were second year dropout students.

The main definitions used in the study are as follows:

Baseline Enrolment:

First-time entering students enrolled for a three-year diploma/degree in the year under consideration.

Second year dropout rate:

Students (of the baseline enrolment/cohort) who did not reregister (enrol) for the second year of study or officially cancelled their studies and did not graduate, as a percentage of the first time entering cohort.

.

Graduation rate:

The cumulative percentage of students from the baseline cohort who received a qualification within or before the specified year.

Table 1 below provides the standard of reporting in the first part of the project:

Table 1: Cohort statistics for 3 year undergraduate qualifications for First time entering students
Entering term	1st Yr	2nd Yr	3rd Yr	4th Yr	5th Yr	6th Yr	7th Yr
2000 Baseline enrolment	1609	1609	1609	1609	1609	1609	1609
Enrolment	1609	1381	1262	672	310	120	77
% Enrolment	100%	86%	78%	42%	19%	7%	5%
Drop outs	0	228	119	58	57	39	31
% Drop outs	14%	7%	4%	4%	2%	2%
Cumulative Drop outs	0	228	347	405	462	501	532
% Cumulative Drop outs	0%	14%	22%	25%	29%	31%	33%
Graduates	0	0	532	305	151	43
% Graduates	0%	0%	33%	19%	9%	3%	0%
Cumulative graduates	0	0	532	837	988	1031	1031
% Cumulative Graduates	0%	0%	33%	52%	61%	64%	64%

The second-, third- and fourth-year dropout rates for the 2002 cohort of students for the participating institutions (labelled A to J) is provided in Figure 1 below:

Figure 1: Dropout rates, second, third and fourth years of study, 2002

The third- and fourth-year graduation rate for the 2002 cohort of students per institution is given in Figure 2 below:

Figure 2: Graduation rates of 2002 cohort

A summary of the variables used as part of the statistical analyses is presented in Table 2 below.

Table 2: Variables used in statistical analyses

Variable	Coding
1. Year	2000, 2001, 2002 and 2003
2. Gender	F = Female M = Male
3. Aggregate	1 if Matric score <720 2 if 720 <= Matric score < 950 3 if 950 <= Matric score <=1199 4 if 1200 <= Matric score <=1439 5 if 1440 <= Matric score <=1679 6 if 1680<= Matric score
4. InstCode	Institutional codes
5. QualificationType	B. Tech, Degree 02/03 and National Diploma
6. Age	Student’s age when registering for the first time
7. SA_Citizen	Yes if SA citizen else No
8. Race	A = African C = Coloured I = Indian W = White
9. Classification of educational subject matter (CESM) category (i.e. major field of study)	The 22 CESM categories as defined by the Department of Education
10. Residence	Yes if accommodated at university/tech res otherwise NO
11. Home Language	All 11 SA languages and Other if unknown
12. NSFAS	Yes if a student received a bursary/loan otherwise No
13. GPassRate	1 if Pass Rate is ‘Less than 0.5’ 2 if Pass Rate is ‘Between 0.5 and 0.99’ 3 if Pass Rate is equal to ‘1 and More'
14. Gload	1 if Course Load is ‘Less than 1’ 2 if Course Load is ‘Equal to 1’ 3 if Course Load is ‘Greater than 1'

The descriptive statistics based on the variables listed above are given below for all the participating institutions.

Qualification type	40.7% general 1st Bachelor degree, 40.9% National Diploma, 18.4% B Tech degree
Course Load	52% course load less than 1 in first year, 18.4% >1
Race	63.5% African, 26.3% White, 3.1% Indian, 7.1% Coloured
Gender	52% female, 48% male
Citizen type	94% SA Citizens
Residence	25% live in university residences
Matric type	45% Senior Certificate Endorsement or Certificate of Exemption issued by the CUP, 29% other Senior Certificate
Previous year	57% secondary school students, 27% other activity
NSFAS bursary	16% received a NSFAS bursary
Home language	42% English, 16% Afrikaans, 11% Sesotho Larger % Tswana =North Sotho?
Pass Rate	30% passed less than 50% of subjects in the first year

The stepwise procedure of the logistic regression, in which variables are selected either for inclusion or exclusion from the model in a sequential fashion, was used for the analyses of the data. The procedure combines the backward elimination and the forward selection methods to add variables to the model or remove variables from the model as they meet or fail to meet specified significance levels respectively. For this study, a significance level of 0.05 was set to allow the variable to enter and/or stay in the model. All 177 406 records were used in the initial logistic regression model.

The following variables were selected by stepwise selection:

Pass Rate, Institution, CESM category, Age, Year, Course Load, Qualification Type, Financial Aid, Grade 12 Aggregate, Race, Citizenship, Residence, Home Language and Gender.

To avoid the influence of the imbalance between the dropout students and the non dropout students, data were divided into calibration/training and validation subsets. The calibration/training data contained 38 264 records of first-time entering students (19 076 randomly selected dropout and 19 188 non dropout students). The remaining records (126 324) were used as a validation dataset. The logistic regression model was built from the calibration/training dataset. (Note that some of the records were deleted due to missing values.)

The following explanatory variables were selected by the stepwise technique based on the training dataset:

Number	Variable	Chi Square Value	P Value
1	Pass Rate	6355.9548	<.0001
2	Institution	549.7133	<.0001
3	CESM	268.9521	<.0001
4	Year	239.7332	<.0001
5	Qualification Type	150.0634	<.0001
6	Course Load	137.1929	<.0001
7	Financial Aid	125.736	<.0001
8	Grate 12 Aggregate	120.3293	<.0001
9	Age	103.4022	<.0001
10	Citizenship	43.5705	<.0001
11	Home Language	35.5939	0.0002
12	Residence	9.6362	0.0019
13	Race *	5.1252	0.1629
14	Gender *	0.8885	0.3459

* Not significant at 5% level of significance.

The first twelve variables could have an influence on whether the student will drop out in the first year of study. Based on the analyses, it is important to note that Race and Gender were not significant as predictors for second-year dropout. However, the Pass Rate in the first year of study came out as the most significant predictor in the analyses from the participating institutions. It is therefore important to monitor the pass rate of first year students on a regular basis.

Similar analyses were conducted for each participating institution separately and the outcome will obviously differ from institution to institution (note that Institution was a significant variable).

2. Case studies presented at the workshop

A summary of the case studies of institutions who made presentations at the workshop is provided below.

2.1	University of Johannesburg

Marthie Cronje

Retention analysis has been carried out on an ad hoc basis at UJ for several years (since at least 1999, in the case of merger-partner RAU). A methodology to automatically extract cohort data on a regular basis, based on SAAIR definitions, was developed towards the end of 2006, and was ready for use in early 2007 (although it is still being refined on a regular basis).

The cohort data are used by the DVC: Academic, various student support divisions, for programme reviews and enrolment planning, and by individual researchers. There is no formalised reporting system yet, with data being provided to interested parties on request or as part of other reports, such as faculty progress reports. Closer links are being forged with the Academic Development Division, to assist in the identification of problem areas where interventions should be focused, as well as tracking the progress of interventions such as extended programmes.

Methodology

Cohort data are extracted for all qualification levels, and the dataset includes cohorts since 2000. SAAIR definitions were adapted to allow for all the different qualification types. Students are tracked from the first time they enter a specific qualification type (or group of related qualification types) until they complete that qualification type for the first time. The main events that are tracked are whether a student enrols and/or graduates in a given year. The primary indicator for first-time entry is not entrance category, but entrance category data are also included in the dataset, and can be applied as required. Currently, the data are extracted from operational data (not HEMIS data), using Access to extract the data and Excel to develop reports.

The example below shows eight of the more than 70 variables available in the dataset (e.g. number or percentage retained in year n, in the same faculty or in the institution as a whole) – the variety of variables allows the use of the single dataset to answer a wide variety of questions.

Example of report table

Qualtype	Values	2000	2001	2002	2003	2004	2005	2006	2007
Qualtype A	Baseline enrolment	3397	2913	3546	4413	3923	3766	3293	3329
% yr1 cancel	4%	4%	3%	3%	3%	3%	3%	3%
% retained yr2	67%	69%	69%	69%	70%	73%	73%
% stopout yr2	2%	2%	2%	2%	2%	1%
% retained yr3	56%	60%	59%	59%	61%	63%
% grad yr3 (cumulative)	15%	17%	14%	14%	17%
% retained yr4	33%	35%	38%	35%	35%
% grad yr4 (cumulative)	27%	29%	27%	27%
Qualtype B	Baseline enrolment	2494	3006	3215	3567	3966	3361	3438	4113
% yr1 cancel	3%	3%	2%	2%	3%	3%	3%	2%
% retained yr2	81%	79%	81%	81%	77%	78%	79%
% stopout yr2	1%	1%	2%	1%	2%	1%
% retained yr3	74%	74%	76%	74%	69%	71%
% grad yr3 (cumulative)	34%	33%	34%	30%	28%
% retained yr4	37%	38%	37%	37%	34%
% grad yr4 (cumulative)	50%	51%	51%	50%

2.2	Central University of Technology

Cay van der Merwe

A case study on CUT’s Engineering students from 2001 to 2007 was performed, looking at two forms of pass rate as well as first year attribution (drop-out). Applying techniques such as Logistic Regression, Neural Networks and Sensitivity Analysis, Association Analysis and Classification Trees to various characteristics and secondary and tertiary results, we came to certain conclusions. Detailed findings and methodology were presented at the EAIR and SAAIR (European and Southern African Associations for Institutional Research respectively). The students’ aggregated grade 12 scores, mathematics scores and score in the same language as the one which they would select as their preferred medium of instruction, were core values in promoting academic achievement and preventing attribution.

The CUT has for more than seven years been measuring quality, inter alia by annual graduate surveys and comprehensive annual retention analyses submitted to Senate. These are aggregated or anonymous results, useful for strategic management, and they have not outlived their purpose.

Our pleas for the use of Data Mining as well as standard methods are two-fold: firstly, students deserve the use of the latest technology (already incorporated into statistical software); and secondly, and mostly importantly, one has an automated, large scale, easily accessible set of tools by means of which individual “at risk” students (as profiled by DM research) can be identified and interventions can be timeously brought down to the single person. Simultaneously, broad academic development interventions will continue.

The next step, as with many endeavours at SA institutions, would be to roll out and apply these techniques and findings, and not let more reports just gather dust.

2.3	University of Fort Hare

Rod Bally

The concept of student retention at the University of Fort Hare has changed considerably over the last decade or so. At the start of that period there was no university policy or even an unwritten “ethos” around student retention. The focus tended to be on pass rates of individual courses and even here there was no common institutional approach to this. Indeed, a number of lecturers felt that a high failure rate was an indication of a “good” course. Even this view was not necessarily a simple one since it could include a sense that any student who could still pass such a course had to “have what it takes” and was a good student.

On the subject of retention, my personal observation (having moved from UCT to UFH) was that poverty of a student’s socio-economic background substantially increased the likelihood of students failing to complete their studies. The issues included serious injuries sustained in taxi accidents, problems at home requiring students to abandon their studies, families unable to sustain the student for an additional year, as well as the various problems that arose from the political tensions at that time.

These were just individual views, however. Issues of throughput, and later retention, only began to appear serious issues on the institutional agenda with the introduction of the new funding formula. In the late 1990s (around 1997), Z Jiya had produced a report that highlighted an alarmingly low minimum time completion rate for the BSc degree, but this was not expanded upon. Institution-wide analyses only began around 2002.

The first fully comprehensive analysis was undertaken in 2005, covering all undergraduate students for the period 1991-2004. This was a cohort analysis using the programme as the unit of analysis, and provided both retention and graduation rates. The system used borrowed from population biology approaches, in that cohorts were identified on the basis of their student numbers and the cohort sizes were then monitored over consecutive years. The advantage of this approach is that it is simple and easy to carry out. Its disadvantage is that it assumes that there are no transfers between programmes (or such transfers that do occur cancel each other out), that there are no stop-outs and that all new students enter with zero credits. It was felt, however, that these factors would not have a significant impact on the overall picture.

The results were summarised graphically; Figure 1 below is an example of such a diagram for the BSc programme with commentary “bubbles” (the figure is extracted from a PowerPoint presentation). Both retention and graduation rates can be seen.

These diagrams had an immediate impact on faculties, who naturally wished to take corrective action in certain cases. At the same time, the University began to introduce the Balanced Scorecard system to guide its strategic development and with this throughput and retention rates appeared as issues with targets on faculty scorecards.

Figure 1: Summarising graph for the Bachelor of Science qualification

The need to set goals raised a weakness of the earlier system, since this was based on a 15-year data set. Even the most effective interventions of faculties would be enormously dampened by the 15 years’ worth of data to which it would be added. Added to this was the fact that circumstances had changed enormously over the past 15 years, both in the country and in the institution.

A more responsive measure was therefore sought. However, such a measure needed to take into account the considerable amount of annual, and seemingly random, variation in retention and throughput rates. At the same time the measure needed to be sufficiently responsive to react reasonably quickly to new circumstances as well as to interventions introduced to affect retention and throughput. This was an important factor since Executive Deans, as heads of faculties, needed to be able to see changes within their 5-year contract cycles. Finally, the measure needed to be reasonably easy to calculate.

On the strength of the above criteria, a 3-year rolling average was selected as being suitable for measuring retention and throughput. These measures are now presented graphically together with the individual year’s result. In terms of retention, the following measures are given:

Percentage retention of year 1 students in year 2 – 3-year rolling average

Percentage retention of year 1 students in year 2 – most recent annual result

Percentage retention of year 1 students in year n – 3-year rolling average

Where n is the minimum time for completion, i.e. 3 years for 3-year programmes and 4 years for 4-year programmes

Percentage retention of year 1 students in year n – most recent annual result

Once again, these are reported on a programme basis, because it is at this level that faculties can best direct interventions where needed. Figure 2 shows an example of such a graph. Current rates are the points at the extreme right of the graph.

Figure 2: Graphic representation of retention rates for the BSc degree

This figure shows a number of the features of the system. One important feature is that the historical trajectory of the qualification has been built up, enabling one to identify long-term trends. A second feature is that the smoothing effect of the 3-year rolling averages can be clearly seen (white squares and circles). At the same time the very substantial annual variations are also evident, especially from 1994 to 1995 and 1997 to 1998. Finally one of the limitations of the method is also demonstrated in 1997 and 2001, where retention figures exceed 100%; this is a consequence of transfers from other qualifications within the institution.

2.4	Tshwane University of Technology

Marié Fowler

Introduction

A number of studies on retention figures, trends, and the reasons why students have dropped out have been conducted at the Tshwane University of Technology (TUT) and the erstwhile Technikon Pretoria and, more recently, in the HERD-SA project. Figures obtained in these studies were important to “position” the institution in terms of numbers and profiles of dropout of students. However, soon the question: “Well then, what now?” was being asked. It became apparent that to improve throughput at TUT would require a macro or holistic approach to the matter.

At TUT a Throughput Committee, under the leadership of the Chief Director (Higher Education Development and Support), was convened by the DVC Academic in 2006. This committee, in line with a macro perspective, is comprised of staff and faculty, able to view “throughput” from various perspectives. This committee is also the guiding force for driving this macro throughput strategy.

Macro systems strategy

The macro strategy is a systems strategy that may be applied, equally effectively, at institutional, faculty or departmental level. It is a given manner of working by multi-disciplinary teams which entails analyses, gap analysis, recommendation, decision-making and implementation phases.

Analysis phase

During this phase, the “path” of the first-time entering student is plotted through various phases or dimensions of his/her sojourn at the institution. Guiding phases, identified at TUT, include e.g. Programme Qualification Mix, Recruitment, Selection and Placement, Orientation and Adaptation, Educational Process, Institutional Context and Phenomena and Workplace – an iterative process from entry into higher education through to performance in the workplace. The various dimensions, as they reveal themselves, are analysed in progressive detail to a most fundamental level.

Gap analysis

During this phase, factors/issues/phenomena, in fact any matters possibly related to dropout, are identified. Where there are already strategies in place, these are to be assessed for appropriateness and effectiveness, and should stay intact if they prove to be efficient and effective.

Recommendation phase

This phase includes the suggestion of recommendations (which may take any form e.g. new policy/process or adaptation, training – ongoing or once-off, RPL, projects, research, interventions, assessments, entry modules, questionnaires). The important thing is that these recommendations, even though they may span a wide area of activities and departments/faculties, should tie in with and support each other.

Decision-making phase

Recommendations, especially those involving policy, should be put to management for their backing.

Implementation

Implementation follows next, but is not the end of the process. The idea is to embark on research to determine the effectiveness of measures that are instituted to combat dropout and studying for longer than minimum periods.

2.5	University of Cape Town

Jane Hendry and Ashraf Conrad

Methodology

This case study makes use of two types of analysis of student persistence. The first of these is a standard cohort survival analysis, which tracks first-time entering undergraduate enrolments by faculty and by programme for up to 10 years after their initial enrolment at UCT. The base data, which include a range of bio-demographic data, are currently extracted from the institutional operational database. The progress of each cohort is analysed and reported on after six calendar years in the case of Engineering students, and the following attributes of successive cohorts are summarised:

Completion rates – percentages of each entry cohort that have successfully completed an undergraduate programme (not necessarily the initial entry programme) at UCT;

Drop-out rates – percentages of each entry cohort who left UCT in good academic standing, without completing an undergraduate qualification;

Academic exclusion rates – the percentage of each cohort excluded from the university on academic grounds prior to the completion of an undergraduate qualification; and

Currently active students – the proportion of each cohort still busy with their undergraduate studies six years after the initial enrolment.

The cohort survival approach therefore summarises the outcomes of any particular intake of first-time entering undergraduate BSc (Eng) students. Where poor completion rates are discerned (due to either high levels of academic exclusion or drop-out in good academic standing), year-to-year retention pattern analyses provide an indication of the points along the path to degree that are problematic in terms of student persistence. Year-to-year retention coefficients are calculated based on three to five years of operational data, and are focused at the persistence from each academic year of study into the subsequent year. Year-to-year retention patterns provide, in a sense, a drill-down showing the following:

Attrition levels in relation to each year of study of the degree programme (thus highlighting the points at which most exclusions or drop-outs occur);

Comparisons of retention patterns of different student groups; and

How changes to the intake profile would affect graduate outputs (in terms of quantity and composition) if the retention patterns remained constant.

The combination of these two techniques provides a useful set of indicators of where attention should be focused – where additional analysis is necessary and what interventions may be required – in order to improve student persistence.

Appendix 1

STUDENT RETENTION

Joint HESA/HERD-SA Consultative Workshop

11 September 2007

Emperor’s Palace Hotel, Kempton Park

List of Participants

Institution	Names	Tel Numbers	Email Address
Central University of Technology	Dr Cay van der Merwe	051-507 3304	cay@cut.ac.za
North-West University	Prof Amanda Lourens	018 299 2558 082 458 0238	Amanda.lourens@nwu.ac.za
Rhodes University	Prof Chrissie Boughey	046 603 8171 082 802 8085	c.boughey@ru.ac.za
Tshwane University of Technology	Dr Marie Fowler	012 382 5430	fowlerm@tut.ac.za
University of Cape Town	Mr Ashraf Conrad	021 650 3375 082 979 0887	Ashraf.conrad@uct.ac.za
University of Fort Hare	Prof Rod Bally	046 622 4314	RBally@ufh.ac.za
University of Fort Hare	Ms Lulama Ngalo-Morrison	040 602 2430 082 202 2202	nmaha@ufh.ac.za
University of Johannesburg	Ms Marthie Cronje	011559 2917 082 870 8150	mcronje@uj.ac.za
University of Kwa-Zulu Natal	Mr Glen Barnes	031 260 3079 083 305 2964	barnes@ukzn.ac.za
University of Pretoria	Dr Gerhard du Plessis	012 420 3323 083 636 1378	gerhard.duplessis@up.ac.za
University of the Witwatersrand	Prof Yunus Balim	011 717 1122	Yunus.balim@wits.ac.za
Vaal University of Technology	Ms Masebata Tjabane	016 950 9489	masebala@vut.ac.za
IDSC	Mr Thys Lourens	018 293 3433 082 418 4134	thys@idsc.co.za
NASDEV	Brighton Mupangavanhu	021 959 3631	bmupangavanhu@uwc.ac.za
US Delegation	Dr Roger Ludeman	herdsa@hotmail.com
Walter Sisulu University	Mr Valile Dwayi	043 726 9430	vdway@wsu.ac.za
Nelson Mandela Metro University	Mr Charles Sheppard	041 504 2696	Charles.sheppard@nmmu.ac.za
Iniversity of the Witwatersrand	Ms Prem Coopoo	011339 6104	dean@sdo.wits.ac.za
Cape Peninsula University of Technology	Mr David Bleazard	021 959 6618	bleazzardd@cput.ac.za
University of Limpopo	Ms Rolda Mamabolo	015 268 2551	roldam@ul.ac.za
University of Limpopo	Ms Sonti Masipa	015 268 2102	masipas@ul.ac.za
North West University	Ms M Klopper	018 299 2902	Mariaan.Klopper@nwu.ac.za
HESA Executive Delegation	Prof Ian Scott (Workshop Facilitator)	082 926 7076	i.scott@mweb.co.za ian.scott@uct.ac.za
HESA Executive Office	Ms Hanlie Griesel Ms Ntswalo Ngobeni	082 454 6767 073 228 7321	Hanlie@hesa.org.za Ntswalo@hesa.org.za

Appendix 2

Reflections on the HERD-SA Project

Professor Roger Ludeman

E-mail correspondence of 25 September 2007

To the HERD-SA team – It certainly was a pleasure to be back in South Africa to be a part of the HESA/HERD-SA Consultative Workshop on 11 September in Johannesburg and the SAAIR Forum 12-14 September in Stellenbosch. Both events went well, especially the highlighting of our Kellogg project as it nears an end which will be 31 December. You all have worked so hard over the past 2 1/2 years to bring this grant project to a successful conclusion, providing the launch pad for a similar effort to continue in a sustainable way and in cooperation with HESA, SAAIR, SAASSAP, and NASDEV. I am sure you also will find a way to "co-opt" DoE and CHE into joining forces in the interest of improved student retention and throughput rates.

Some observations:

As we come to an end of the HERD-SA grant funded by the Kellogg Foundation in Southern Africa, I want to thank you all for your diligence, insights, and vision in seeing this project successfully achieve all of its original goals and objectives and, more importantly, successfully showed SA tertiary education that teams can do anything if committed to common purposes and properly supported. Congratulations!!

I want to mention in particular the continued involvement of Tshwane University of Technology, the University of Witwatersrand, and North-West University as pilot institutions. You certainly have been pioneers in this sort of team-based collaborative research in SA and your efforts have bolstered the confidence of all of us to stay the course. Your efforts are admirable and much appreciated. Many thanks.

Congratulations also go to the newer institutions that have joined us in the HERD-SA project. It has been quite impressive to see you all get your data organized, analyzed, and presented so that it can be included in the final HERD-SA report; and to all institutions: While all public displays of data will confidentially protect the identity of students and institutions, you now have at your disposal a way to look at your own data through the lenses of the HERD-SA common definitions. This allows for comparisons to be made internally and externally against the standards the HERD-SA group set for itself. While you have a way to go to such time when all this kind of data becomes public knowledge on a routine basis, I am amazed that you all have come as far as you have in preparing for that eventuality in SA someday soon. This has been a remarkable feat.

With interest from at least 2 other institutions in joining the HERD-SA project, that would mean that over half of SA tertiary institutions would be involved. I am sure others would have joined if we had more time; however, more importantly, all institutions will have the opportunity to be a part of a new and continuing effort that I am sure will be taking place.

The work of Amanda Lourens as your convener and then project director from the SA side has been superb. I think we were all fortunate to have a professional, colleague, and friend of such talent and stature facilitating this important effort. I am sure that the HERD-SA team concurs and thanks you, Amanda for your work and caring approach as you helped to bring us to this stage in our effort. Rosemary Hayes and I appreciate all that you do for HERD-SA.

Amanda, Kevin and I did discuss the need to follow up on Kevin's idea to develop a CD that each and every institution across SA could utilize to routinely input retention/throughput data to produce longitudinal profiles of your dropout students and your retained students. I agree that this would be a very helpful tool for all and we should be able to fund that effort through Kellogg funds under this HERD-SA project.

With the recently added support of SAAIR and HESA it is very likely that some form of ongoing effort in the area of student retention and graduation rates research and related new initiatives to attack these concerns will become a reality in the next few months. At the conclusion of the HESA/HERD-SA Consultative Workshop on 11 September, a group was formed to study such efforts and how they may best be organized so that this activity and new ones can continue on a sustainable basis following the conclusion of the HERD-SA project 31 December of this year. This is good news, especially since Amanda is serving as convener and some of our HERD-SA members are also involved along with some new people representing HESA and SAAIR. More news on this front will be evident as the next few weeks go by.

I will be working with Amanda to complete the final report of the HERD-SA project that I will present to the officials at the Kellogg Foundation some time in January. I expect a very positive response once they see how much you all have accomplished with the funds provided and within the timeframe allotted. Once again you are to be congratulated for all your stellar performances. Providing this basic research as a starting point for future efforts in SA is something you all should be proud of as it shows the way to future efforts you all can assist in designing and carrying out. Advocating locally and nationally for a better effort in retaining and graduating more students must be exciting for you. It certainly will assist South Africa in its continuing transformation toward world class status in all that you do.

To SAASSAP and NASDEV: My thanks to you for giving some of the main input that led to the grant proposal to Kellogg back in 2002. Your ongoing support has been so critical to the success of the project. ACPA and NASPA, also true partners in this venture, have enjoyed the relationships formed among these student affairs organizations and continue to seek new ways to support these efforts.

To SAAIR and HESA: Your involvement with HERD-SA and, more importantly, its future rendition, is very much appreciated and so essential in lending credence to what the project has set out to do. Thank you Ian and Hanlie for believing in HERD-SA and the potential for the future. I look forward to hearing how things evolve into a completely designed and carried out South African effort that promotes and initiates more investigations leading to new support for students so as to improve student retention and throughput rates. I hope you can find ways to bring CHE and DoE collaboratively into the fold.

Finally, while I hope to return to SA someday soon, this may not be "in the cards" as we say in the U.S. While I am still excited about being involved in new things and assisting others, at age 71 I do see things differently than when I was say 40 or 50 or 65 (my age when I first visited SA!!). The trips get a little harder each time. Who knows what the future will bring, especially since South Africa holds a warm spot in my heart. After all, it is my second country!!!

My wife Sandy and I want you know that you are always welcome in Eugene, Oregon where the Cascade Mountains and the beautiful Oregon Coast are each only an hour from our house and some of the best winelands in America (and some would say the world) are only a few miles from our doorstep. So, come visit us. It is least we could do for those of you who have been so kind to us while we lived in visited South Africa.

The very best to you all who are so fortunate to live in one of the most beautiful nations in the world among the most beautiful people anywhere,

Roger B. Ludeman, Ph.D.
Project Director
Higher Education Retention Data for South Africa (HERD-SA)
Kellogg Foundation in Southern Africa Grant
Email: herdsa@hotmail.com

Acronyms and Abbreviations

ACPA	American College Personnel Association
CESM	Classification of Education Subject Matter
CHE	Council on Higher Education.
CPUT	Cape Peninsula University of Technology
CSRDE	Consortium for Student Retention Data Exchange
CUT	Central University of Technology
DoE	Department of Education
DVC	Deputy Vice-Chancellor
EAIR	European Associations for Institutional Research
HEIs	Higher Education Institutions
HEMIS	Higher Education Management Information System
HERDSA	Higher Education Retention Data for South Africa
HESA	Higher Education South Africa
IDSC
IR	Institutional research
ITLS	Improving Teaching and Learning for Success
MIS	Management Information System
NASDEV	National Association for Student Development
NASPA	National Association of Student Personnel Administrators
NWU	North-West University
RAU	Rand Afrikaanse Universiteit
RPL	Recognition of Prior Learning
SAAIR	Southern African Associations for Institutional Research
SAASAP	South African Association of Senior Student Affairs Professionals
SAQA	South African Qualifications Authority
TUT	Tshwane University of Technology
UCT	University of Cape Town
UFH	University of Fort Hare
UJ	University of Johannesburg
UKZN	University of KwaZulu Natal
USA	United States of America
Wits	University of the Witwatersrand

The centrality of the need to improve graduate output is argued in Scott, Yeld and Hendry (2007), A case for improving teaching and learning in South African higher education, Pretoria: Council on Higher Education.

Pass Rate = Full Time Equivalent(FTE) Degree Credits Passed/FTE enrollments

Course Load = The sum of all credit values for all courses a student has enrolled for

Cf. CA van der Merwe & S van der Merwe, 2007.