QPDOC102

MANUAL FOR PROGRAMME REVIEW

JULY 2004

Compiled by Dr E Genis 
Chief Director: Directorate of Quality Promotion 

1.	INTRODUCTION

Higher Education Institutions need to assure themselves, their stakeholders, such as the Department of Education which provides funding, the Higher Education Quality Committee which is tasked with assuring the quality of higher education, parents, students and employers, that they can deliver the quality of education provision expected by these stakeholders. A first step toward such assurance is to have in place the necessary policies, strategies, processes, procedures, and guidelines, together with the necessary structures, resources and management system to deliver the education service to the level of quality provision the institution sets itself. A logical next step is to apply self-evaluation to determine whether the education provision meets the institutions own targets and requirements, as well as the expectations of the spectrum of stakeholders. 

The performance of the institution can be approached from two perspectives. Firstly, how the institution as a whole performs, and secondly, how well delivery in individual programmes performs. The Higher Education Quality Committee has in place a framework and programme of institutional audits aimed at evaluating how well the institution applies its mission, policies, strategies and processes to assure the quality of its education provision. At programme level, it has a framework and system in place to evaluate how well individual programmes are provided. Based on this evaluation, the HEQC will accredit individual programmes. These frameworks spell out the minimum expectations that we as a HEI should meet to be an accredited institute, and clearly places the responsibility on us for assuring the HEQC and all other stakeholders of the quality of our education provision. 

One of the mechanisms previously used to make judgements about our own performance, was self-evaluation. This critical appraisal or review of education delivery by those who provide the service remains an effective tool in the quality cycle of planning, implementing, reviewing, and improving. 

2.	THE PROGRAMME REVIEW FRAMEWORK

It may be useful to define, at the outset, some of the terms that are used in this document.

A programme is a purposeful and structured set of learning experiences that leads to one or more qualifications. It enables learners to achieve the pre-specified outcomes of a qualification.
A qualification is the formal recognition, through certification, of learning achievement, awarded by an accredited provider.
Programme review refers to internal quality assurance procedures that a provider undertakes to monitor and reflect on its education provision.
Programme evaluation is the external process of quality assurance, undertaken by an external agent such as the HEQC or a professional body ETQA or SETA.

2.1.	Our model

The aim of this manual is to guide you through the process of reviewing how well you are delivering the individual programmes in your department. Responding to the following set of questions forms the basis of such a review:

•	Why do we do what we do? (Do we have a comprehensive understanding of what we are, or supposed to be, doing?)
•	Why do we do it in a particular way (Do we do the right things in the right way?)?
•	How well are we doing it?
•	How do we know how well we are doing?
•	What can we do better? (Do we really achieve what we set out to do?)

To respond to these questions it is preferable to approach the review in a systematic way, so that all the relevant elements that affect the education provision, as well as the results of the delivery, are reviewed, and their performance judged. A model, which provides a framework for the process, is therefore useful when engaging in programme review. Our model is loosely based on the Excellence Model of the European Foundation and the South African Excellence Model, since these models include all the elements we consider relevant when judging our programme delivery. 

The advantage of such a visual representation of review is that it becomes clear that the input elements, or enablers, are the aspects over which we can exercise control through policy. This represents areas for pro-active decision-making. The results, on the other hand, provide the basis for planning improvement. It should be clear that it represents a continuous cycle.

The framework takes into account the HEQC framework for programme accreditation, but does not attempt to follow it closely or exclusively. The point of departure is that programme review is a natural activity within our concept of quality and our quality management system, the underlying quality management principles of continuous improvement and customer care, and our own strategic goals and targets. 

Programme review within our framework means that we can demonstrate:

•	How leadership and strategic planning guide us in planning the application of all the resources for the delivery of the education service
•	What is being done in the delivery of a programme
•	How well it is being done, and 
•	What plans are in place to bring about continuous improvement.

The review assists us in making judgements about the value or worth of a programme within the local and wider national and even international contexts. It also signals to us where we are in our developmental trajectory in a programme or in a department and assists us in formulating our self-improvement plans. Thus programme review is premised upon a reflective, diagnostic, and continuous improvement attitude. It requires us, as professionals, to make self-judgements regarding all our teaching, research and community service activities in a critical but constructive manner, and to signal our intentions to remedy any possible weaknesses and improve practices. 

Although our programme review is not in the first place undertaken to fulfil HEQC requirements, the rigour with which we conduct the review process will be a critical factor in determining whether the HEQC awards self-accreditation status to our institution. Therefore we need to demonstrate through this process that none of our programmes are of questionable quality, and that we have effective and efficient internal programme quality assurance mechanisms in place.

Self-Evaluation Model

1.	Leadership

•	Fitness for and of purpose 

•	Vision, mission, strategic goals

•	Management (planning)

•	Improvement mechanisms

2.	Policy & Strategy

•	Teaching & Learning strategy 

•	Policies & procedures

•	Teaching and Learning
•	Student Assessment
•	Post Graduate
•	Research
•	Community Service
•	NSSP

•	Compliance 

3.	Customer & Market Focus

•	Programme Design

•	Student recruitment, selection, access

•	Student academic development

•	Relevance

•	Administration 

ENABLERS

RESULTS

12.	Impact on society

•	Community service rendered
•	Use of research

9.	Customer Satisfaction

•	Student satisfaction 

•	Employer satisfaction 

•	Employment Data

10.	People Satisfaction

•	Staff satisfaction

7.	Student Performance

•	Pass, throughput & retention rates 

•	Related short course successes

•	Equity of outcomes / representivity

•	Integrity of Certification

8.	Research Output

•	Post-grad degrees

•	Papers, artefacts, patents

11.	Partnership performance

13.	Business Results

•	Programme Review

•	Attainment of Outcomes
•	Curriculum Alignment
•	Self-Assessment

•	Stakeholder Feedback

•	Reports (examination reports, quality reviews/audits) 

•	Improvement achieved 

•	Improvement recommended

Continuous improvement

6.	Teaching & Learning Process

•	Programme Coordination

•	Programme Delivery

•	Assessment in different modes  & methodology

•	Research

•	Postgraduate studies

•	Work based learning

•	Community service

•	Short courses

Input						Process			Outcomes

2.2.	Concept of Quality

Our concept of quality includes at least the notions of fitness of and for purpose, value for money and transformation (i.e. developing the capabilities of individual learners for personal enrichment as well as the requirements for social development and economic and employment growth). But how do we judge whether what we do and whether the results of our actions can be described as ‘quality’? Finding objective criteria is almost as difficult as defining the notion of quality, but however difficult, we need criteria to judge our actions against. 

We have over the past few years been using a number of indicators as criteria to reflect on the inputs, process and outputs of the teaching and learning process (such as number of students, their race and gender profiles, pass rates, throughput rates, graduation rates; profiles of teaching staff, research output, infrastructure provision, etc). Important as such statistical data may be, data is in itself insufficient and meaningless if not accompanied by reflective analysis, which identifies good practice underpinning good results, and areas for improvement to improve results. 

The HEQC provides another source of criteria in its Institutional Audit and Programme Accreditation Frameworks, which assists us in approaching programme review in a qualitative, reflective way, but utilising facts and data to inform judgements, conclusions and recommendations.

This guideline provides a sufficiently detailed framework for reflection within the framework of the model outlined above. Criteria of the HEQC have been covered, but are not addressed or phrased in exactly the same way as in the HEQC documents, as our own context and approach takes precedence in this internal process.

The guideline document further assists you in planning your review process. It addresses the components on learning programme delivery that should be included in a review report. For every element to be addressed, the guideline indicates what information and documentary evidence should typically be available to support your self-evaluation. 

Please note: The guidance notes are not aspects to be ticked off as you address them. They are only suggestions as to what could be addressed, and should in no way limit your self-reflection process.

3.	WHERE TO START AND WHAT TO DO?

PLAN

DO

CHECK

ACT

The quality cycle (Plan-Do-Check-Act) is as useful a basis for self-evaluation as it is for every other aspect of our education delivery. Although the self-evaluation procedure is described in our Quality Management System (Annexure A), and should provide adequate guidance, the unfolding of the process in terms of the quality cycle is provided to further assist you.

3.1.	Plan

A good place to start is to plan the review. This entails not only planning for the actual event on a specific date, but also putting the planning within the perspective of the entire improvement cycle. This includes the targets at different points in the cycle, what will be documented, how it will be documented, the responsibilities of every lecturer responsible for the programme, how every element of the review framework will be approached, who will collate the information for every element, including evidence of plans and results, the process of collating the necessary information and data and generating the review report; the evaluation event and the team that will be involved, and the generation of the evaluation report.

3.2.	Do

In this phase, all the information and data necessary are collated and organised, and a report, based on the elements set out in the model, prepared. This review will be made available to the evaluation team, prior to the evaluation event. In this phase, all the information and data relevant to each element are scrutinised, as well as any additional information or documentation that could serve as evidence of practice, planning or performance.

The development of a Quality Management System (QMS) lies at the heart of a successful self-evaluation process. Where Quality Manuals have been developed, containing the implemented policies, processes and procedures, with retrievable records of performance; self-evaluation merely requires systematic reflection on what we planned to do, how we intended doing it, how we executed our plans in practice, and how well we did this. The opposite is unfortunately also true. Reflection will be greatly hampered where the development and maintenance of the QMS has taken a backseat, because there may be insufficient documentation on the planning, or incomplete records of performance and results.

3.3.	Check

An evaluation committee is convened, comprising peers from the relevant discipline/career fields, persons knowledgeable regarding quality systems, and clients (students, employers, professional body representatives). This committee reflects on all the input, process and output information and supporting data, and judge these against the criteria of the review framework. Strengths and good practice are identified, but also weaknesses and areas for improvement. These judgements are reflected in a critical report, clearly identifying good practice and indicating areas for improvement.

3.4.	Act

Based on the evaluation report, the programme group will plan a new cycle, making provision to address the improvement targets, and setting target dates for the achievement of the improvements.

4.	THE REVIEW REPORT

There view report should cover all levels of a particular qualification that is offered in an academic department. The learning programmes for the different exit qualifications should be evaluated individually, or at least a differentiation made between under-and post-graduate programmes.

4.1.	The Audience

The report is prepared in the first instance for the programme group (i.e. those involved with the programme). It serves as a reference point for continuous programme renewal and improvement. It presents a foundation for future reports, enabling a programme group to up-date and edit the report annually, thus considerably reducing future workload.

Secondly the report is prepared for the faculty for the purpose of providing a profile of faculty performance for annual reporting.

Thirdly, the self-evaluation report informs external evaluation by the HEQC or a professional body ETQA. Together with its supporting documents that provide evidence, this portfolio allows the HEQC and/or another ETQA to make judgements about the standard of delivery, and whether this meets their pre-determined requirements.

4.2.	The Portfolio

The portfolio consists of your review report, plus all the supporting evidence that may be relevant to verify the claims and statements in your report. Although you are not required to attach hard copies of all relevant information and records, it would expedite the work of the panel if you would indicate clearly in your report, which records/documents apply, to facilitate quick retrieval/reference during the evaluation visit. 

In the event of technical problems prohibiting access for panels to the electronic documents and records, at least one hard copy of the Quality Manual per faculty, containing the relevant documents, is required. Minutes of the relevant QM structures and any other records (such as exam scripts, projects, portfolios, examiners'/moderators' reports, etc) that may serve as substantiating evidence of statements made in the self-evaluation report should also be available.

4.3.	The format of the report

The format below is recommended as organising structure for your self-evaluation report. It addresses the elements that should be reported on. You may add elements that you consider relevant. Also, no limit is set on how much information you should provide, or how long the report may be, but it is recommended that reporting is succinct without omitting important facts, data, judgements or conclusions.

Part A:	 Programme Detail

1.	Programme Name
2.	Faculty
3.	Name of the Department 
4.	Programme co-ordinator, programme committee 
5.	Other Faculties/departments involved in offering the programme 
6.	Qualification specifications (SAQA format)
•	Purpose
•	Outcomes
•	Assessment criteria
•	Summative assessment
•	Moderation
•	Levels
•	Credits
•	Entrance assumptions and requirements
•	Articulation possibilities

7.	Programme specifications
•	Curriculum (modules per level, as specified in your information flyer for the programme)
•	Entrance requirements

8.	Accreditation status

Part B: 	Review Framework

In the tables below, the different elements are addressed, together with evaluative questions and an indication of possible data and other evidence to corroborate your review statements. You may add evaluative questions as you see fit.

1.	LEADERSHIP:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
1.1.	 Fitness for Purpose	How does the programme serve to fulfil the institutional vision, mission and strategic goals (Fitness for Purpose)? 

How does the qualification add value to the institutional PQM?	Institutional mission and strategic goals 

Contextualised programme vision, mission and strategic goals.  

Record of approved PQM	
1.2.	Vision, mission and strategic goals (Fitness of Purpose)	How do the qualification and its learning programme (purpose, rationale, outcomes, and curriculum) responds to national, regional and local employment and socio-cultural needs (fitness of purpose)?

How does the programme address transformational issues including equity of access and equity of outcomes?	Needs analysis, feedback from industry/advisory boards.

Reference to national policies in programme planning.

Admissions policy and student development plan	
1.3.	Management	What defined lines of accountability, responsibility, communication and reporting exist in the programme?

How are planning, strategic choices, resource allocation and quality management linked (Alignment of programme planning and budgeting)? 

How is the strategic plan translated into clear objectives with specified times frames?

How are issues of feasibility and resource implication considered prior to its approval? 

Describe the programme quality management system?

How is programme quality maintained at every site of delivery of the programme?

	Programme organogram with accountability lines  
Evidence that management responsibilities and lines are clearly communicated, understood and operational

Review results and budgetary planning linked to quality objectives and improvement targets.
Evidence that the budget is appropriately aligned to programme planning to ensure an adequately resourced and managed learning environment.

Strategic and action plans.

Business plans and accurate projections.

Evidence of a documented QMS, its functioning, and improvement cycles

Evidence that there are policies, and a system for assuring quality, and that they are applied	
1.4.	Improvement Mechanisms	How is continuous improvement for this programme managed?
 
	Evidence of policy/rules; reporting; monitoring

Evidence of review cycles, their application, results, and improvements implemented.

2.	Policies and Strategy

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
2.1.	Teaching and Learning Strategy 
	How does the teaching and learning policy and plan relate to the mission, goals and the PQM of the University? 

How is academic planning in the faculty, department and programme linked to the institutional teaching and learning policy and plan

How does the teaching and learning strategy guide the different modes of delivery, including experiential learning, Telematic modes and research? 

How does the teaching and learning policy and plan guide the development and approval of new qualifications and programmes in the faculty?

How are issues of feasibility and resource implications of new programmes considered before approval?

How is provision made for new proposed programmes in the programme planning and resource allocation processes?

What implementation plan exists for the

•	Phases of new programmes 
•	Budgetary allocations for each implementation phase of new programmes
•	Human and other resource acquisition for the different phases of implementation of new programmes. 
•	Meeting of HEQC accreditation and   professional Board criteria  (including review, output and impact assessment) 	Evidence of a documented T& L plan 

Academic and Strategic planning documents

Teaching and Learning strategic documentation

Teaching and Policy

Business plan
Procedures for application to offer new programmes.
Approved procedures.

Evidence of linking programme planning and resource allocation

Programme strategic documents	

2.2.	Policies and procedures

•	Teaching and Leaning

•	Student Assessment

•	Post-Grad

•	Research 

•	Community Service

•	NSSP

•	Workplace Learning (Experiential Learning)

What defined policies and/or procedures exist and are operational for the design, development, approval and delivery of qualifications and learning programmes, and learning materials  

What defined policies and/or procedures exist and are operational for the management of and the assessment of students.

What defined policies and/or procedures exist and are operational for the management of post-graduate studies

What defined policies and/or procedures exist and are operational for the management of research

What defined policies and/or procedures exist and are operational for the management of community service

What defined policies and/or procedures exist and are operational for the management of NSSP

What defined policies and/or procedures exist and are operational for the management of workplace learning

Teaching and learning policies and procedures

Student assessment policies and procedures

Post-grad studies policies and procedures

Research policy and procedures

Community service policies and procedures

NSSP policies and procedures

Workplace learning policies and procedures	
2.3. Compliance	How well do the qualifications and programmes comply with the HEQC, NAP, SAQA and professional body requirements and specific qualification standards?

How does the programme articulate with other qualifications registered on the NQF?

How is RPL specifically catered for?	Evidence that qualifications meet registration requirements and level descriptor specifications at every exit level (outcomes, credits, assessment criteria), and has clear access and articulation descriptions

RPL policies and procedures	

3.	CUSTOMERS AND MARKET FOCUS:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
3.1.	  Programme Design
	How does the programme add value to the institutional PQM, vision and mission 

How does programme design, development, and approval take into account the needs, requirements and inputs of legitimate external stakeholders (industry, local community, students, professional bodies, academic peers)?

How coherent is the learning programme design and how feasible is the delivery of its design?

Are the modules/subjects coherently planned with regard to content, level, credits, purpose, learning outcomes, rules of combination, and relative weight?

How well does the design of the programme offer learning and career pathways to students including opportunities for access and articulation from other programmes within and across institutions?

How well does the programme’s design, learning outcomes, degree of curriculum choice, expected completion time, teaching methods and modes of delivery cater for the needs of its student intake?

How does the programme cater for an appropriate balance of theoretical, practical and experiential learning?

How are assessment criteria, integrated assessment and assessment strategies coherently linked to learning outcomes

How is workplace learning managed, assessed and moderated

How are learning materials designed and aligned to learning outcomes and integrated assessment

	Programme proposal and business plan

Market research, needs analysis, regional clearance, professional board support

Curriculum structure for every exit level, including EL component to show intellectual coherence.

Evidence that modules/subjects are integrated to realize the purpose of the qualification

Programme specification (HEQC and SAQA)

Student target group analysis, and evidence of enrolment planning

Programme specifications

Programme specifications, learning plan and assessment strategies.

Workplace learning plans

Learning materials	

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
3.2.	Student Recruitment, Selection and Access

•	Student Access

3.3.	Student Academic Development

3.4.	Relevance

3.5. Administration

	How are recruitment, selection and admission policy and procedures used to ensure that sufficient numbers of adequately prepared students enter the programme?

Does the institution’s and programme’s advertising and promotional material reflect accurate and sufficient information regarding admission policies, application procedures, completion requirements and academic standards?

How does the institution liase with potential pools of students in the local communities, and how effectively is information on the faculty offerings disseminated

How effectively is the target number of places in the programme filled? 

How does the programme apply the institutional admission policy and how is it aligned to the programme’s mission and strategic plan?

Are the programme’s admission criteria in line with national goal of widening access to HE (of  “representivity in enrolment” and “broadening of social base of students”)

Are Equity targets clearly stated, as well as the plans to attain them?

How are student support and development policies guiding student support services and how well are they provided to learners in this programme?

How are student development programmes integrated into, or linked to, the programme curriculum?  

Are academic development programmes and/or integrated curricula strategies available to cater for a range of students? 

How well is the availability and scope of these programmes communicated to the students?

How is institutional language policy implemented and what strategies are employed to ensure that language is not a barrier to learning and progress?
How is student progress monitored, how is this used to identify ‘at risk’ students, and how are interventions provided for such students?

 How effective is student referral to Academic support services in improving student success?

Are the admission requirements in line with the assumptions of learning stipulated for this programme?

How fair and effective are selection methods and mechanisms?

How does the faculty, department and programme in its recruitment and admission procedures, provide for flexible entry routes that take into account level of preparedness and prior learning?  

How are the needs of a vocational and professional programme taken into account in selection?

How does the programme group/dept and faculty respond to applicants who meet admission criteria but are not offered places?

How are applicants provided with guidance on flexible entry routes, assistance with second choices, and career guidance? (Specialised staff?)

How well are admissions and selection integrated and coordinated between the programme group and Administration

How effectively are applications processed?

How effectively are selection arrangements and outcomes communicated to students?

How accurate is the programme modular database, and how does this support student registration and certification?

How effective and efficient is student registration for this programme?  

How satisfied are students with the process?

How accurate are students’ academic and records?

How does the administrative service in the programme provide for information to students on venues, timetables, availability of staff, consultation hours, re-admission to programmes and referral to appropriate academic development programmes? 

How well is this service provided in terms of accuracy, clarity and consistency of information?  

How well do programme administrative and support staff deal with the needs of diverse student population? 

	Procedure linking programme to administration 

Promotional material,
Calendar/programme handbook, course outlines, information brochures.

Marketing plan (targeted marketing) 

Enrolment plan and data to show how size & shape and equity targets are met

Institutional and programme admissions policies and procedures.

Programme admission policy and criteria

Student equity plan, targets and action plans

Evidence of planning and budgeting for academic development programmes for students; 

Linkage between SD programmes and curricular requirements

SD programme outcomes

Advertising brochures and attendance records

Language policy

Data on student performance and referrals; evidence of counselling, guidance

Programme outcomes and student performance records – pre-test – post-test.

Programme planning documentation.

Students admissions policy and procedure

Student admissions policy and procedures
RPL policies and procedures

Professional/advisory board comments and inputs

Strategies/procedures for rerouting; inter-institutional links; analysis of reasons for non-placement

Programme advisors guides, counselling session report

Procedures linking programme and administration

Procedures linking programme and administration

Selection policy and procedure

Programme database and controls

Registration procedures

Student feedback

Controls and checks.

Programme procedures and records

Controls and checks

Student satisfaction survey results	

4.	PEOPLE MANAGEMENT:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
4.1.	Staff Recruitment And Profile

•	Staff Recruitment

•	Staff Equity

•	Staff Profile

  

•	Staff Development

•	Staff Appraisal

•	Research Capacity

•	Support Staff 	Do appropriate policies and procedures exist for the selection, appointment, induction and payment of part time academic staff and tutors? 

How effectively is the policy and procedures applied to ensure appropriately qualified staff for this programme?

Are there equity targets set for the staff in this programme, and how does the staff profile compare with these targets?

How appropriate is the staff profile and capacity for the delivery of the programme?  

What benchmark requirements are set in respect of relevant academic qualifications professional experience, teaching experience, assessment, and monitoring and evaluation of practical and experiential learning?  

How does part time and contract academic staff members meet the same teaching and professional criteria as full-time staff?

Is a coherent staff development policy in place in the programme?

How does staff development in this programme relate to the Skills Development Plan of the institution? 

How does the SD plan relate to the programme performance management system? 

How is the SD plan for teaching staff aligned to the Teaching and Learning strategy of the institution?

How does orientation and induction of new and temporary staff serve to prepare them for their role in this programme?  

How are new staff and temporary staff mentored or interned?  

How are staff provided with opportunities to improve their curriculum development and teaching practice capabilities to enable them to respond to diverse student needs especially, under-prepared and second language students? 

How is the academic staff equipped with skills to ensure that students are exposed to a diversity of ideas, styles and approaches? 

Is ongoing professional development and training of staff as assessors performed according to accepted higher education standards?

How is academic leadership developed in the HOD and Deans?

How do the results of staff appraisal inform staff development plans? 

What procedures are in place to manage poor staff performance, and how is support and training provided for under-performing staff?

How is research into teaching practice encouraged in the institution and this programme? 

How is participation in national and international conferences used to support benchmarking of teaching practice and the development of staff?

How is the quality management system in this department used to promote self-reflection on teaching practice, and what incentives/awards are provided in support?

Do academic staff members have research experience through their studies towards HE qualifications? 

How are the research outputs aligned with staff profile and curricula requirements?

How is research capacity developed in new and less experienced staff?

How is the support staff for this programme developed?

	Staff recruitment policies and procedures

Staff appointments and staff resumes

Equity targets, plans and strategies 

Documented Staff profile, indicating position, qualifications, experience, subjects/modules and their levels being taught

Policies and procedures for part time and contract staff induction practice and outputs.

Staff development policy

Linkage between programme staff development and institutional skills development

Programme review outcomes, programme quality performance indicators

Linkage between the institutional T&L strategy and the programme SD

Staff orientation policies, procedures and records

Mentorship plans, roles and reports

Staff development programmes and attendance records

Examples of teaching and learning innovative practice.

SD programmes topics cover developments in HE environment.

Training and development plans and attendance records

Record of staff appraisal policy, procedures, CPAs, appraisal.

Records of intervention strategies, outcomes and follow up.

Research papers, staff review and reports

Conference papers and linkages to teaching practice and SD

QMS, review data and inputs in strategic planning

Requirements for staff to teach at under- and post-graduate levels

Scholarly/research requirements of staff

Research development

Staff development policy and Skills Development Plan.

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
4.3 Staff capacity

	How satisfied is staff with the working environment (including workload, remuneration, appraisal)?
Is specific and sufficient time allocated for staff to fulfil their responsibilities related to the programme?

Is there sufficient opportunity for students to consult with the teaching staff? Are the times and modes of contact for consultation between academic staff and students made known?
  	Lecturer: student ratios

Record of teaching schedules, consultancy schedules

Satisfaction surveys	

5.	RESOURCE AND INFORMATION MANAGEMENT:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
5.1.	Library 

5.2.	IT Infrastructure

5.3.	 Classrooms

5.4.	Laboratories and Equipment

5.5.	Study Guides

5.6.	MIS

	How well and systematic are library resources integrated into the programme curriculum? 

Are library resources sufficient in size and scope to: 

•	Complement the curriculum, 
•	Provide incentives for students to learn according to their own need, capacity and pace 
•	Support appropriate professional and scholarly activities of staff and students in the proposed programme

How adequate is the access to the library and facilities?

How adequate is the IT infrastructure (hardware, software) for this programme and how well is it maintained? 

Is a financial plan available for the maintenance and continuous upgrading of the infrastructure?

How effective is implementation of suitable recent versions of industry standard software and databases in computer venues required by the programme?

How adequate is electronic access to information for students at different levels (After hours and during weekends)?

How well are classrooms equipped and how does this impact on the success of learning?

How effective is the planning of venue allocation and timetabling to accommodate the needs of students of this programme?

How well are laboratories equipped and how does this impact on the success of learning?

Does a code of conduct for laboratory practice and safety exist and is it adhered to? 

Are staff and students trained in the use of technology and advanced equipment required for the programme?

How informative are study guides w r t the purpose of the programme, its structuring, responsibilities of students, the learning and assessment schedule, the content of the individual modules, the expectations regarding assignments and projects, the expectation regarding experiential learning and its assessment?

How are study guides provided timely and seen to be guiding the teaching learning process? 

How is the MIS (management information system) and ICT used to record and disseminate programme information, learning progress and results?

Are reliable and accurate records of students in the programme, including admission, progression, grades/marks, fees and graduation, available?

How accessible is the MIS to lecturers and how does it enable them to follow student success on a continuous basis, to identify at-risk students? 

	Data on books, journals, library usage;

Prescribed texts, lists of recommended books and other resources

Circulation statistics

Curricula requirements, asset records, maintenance plans

Budget 

Industry/advisory board recommendations

Information access plan

Infrastructure and asset register

Programme specifications, student numbers, modes of delivery

Laboratory asset register, usage records

Code of conduct, safety notices, training and records

Training records and schedules

Study guides that corroborate claims about the quality, value and use of study guides

MIS records, data rich reports as inputs to planning

Student records as per programme performance

Evidence that lecturers utilise the MIS

	

PROCESS MANAGEMENT

6.	PROCESS MANAGEMENT (TEACHING AND LEARNING):

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
6.1.	Programme Coordination

6.2.	Programme Delivery 

6.3.	 Assessment

•	Moderation

6.4.	Research Management and Post-Graduate Supervision

6.5.	Work based Learning

6.6.	Community Service

6.7.	Short Courses

	Is a programme team established and does it operate within a framework of an agreed upon mandate and defined procedures and responsibilities?

How does the programme group manage the following?
•	Programme design.
•	Ensuring that all conditions for delivery of the programme are met.
•	Day to day coordination of programme delivery.
•	Review and evaluation of the programme.
•	All aspects of the programme quality management system.
•	Monitoring expenditure in terms of the programme budget.

Does opportunity exists for student input and participation in relevant activities of the programme team?) 

How do the programme purpose and outcomes direct the teaching-learning process?  

How are suitable learning opportunities provided to ensure the acquisition of the knowledge, skills and attitudes specified in the programme outcomes? 

How are principles and practices of instructional design put into practice, including appropriate balance and mix between teaching methods, fitness of teaching methods for the nature of the learning material, design and use of learning materials, and the design and use of instructional and learning technology?

How are learning opportunities provided to ensure that theory is put into practice? 

How is it ensured that learners are able to meet the learning requirements of the programme within the stipulated time?

Are skills and techniques mastered which are required by the specific occupation?

How does facilitation accommodate student diversity?
How well are institutional / faculty / professional rules governing assessment published and communicated to students and relevant stakeholders?

How well and effectively are breaches of assessment regulations dealt with?

Are there procedures for student appeals regarding explicitness and fairness of assessment? 

Are policies and rules in place to ensure the security of the assessment system, especially with regard to plagiarism and other misdemeanours?

Are assessment criteria explicitly stated and commensurate with the level of the qualification and aligned with the requirements of the NAP, SAQA, and where appropriate, Professional Bodies? 

Do the assessment instruments at the different levels of the programme reflect a progression in both depth and scope and complexity of cognitive skill?

How well are assessment tasks and criteria aligned with the programme learning outcomes? 

Are learning activities and assessment aligned with the learning outcomes at both the programme and the modular level? 

How are fairness, reliability and validity of assessment planned for and achieved and how satisfied are students in this regard?  

Are assessment procedures effective in measuring student attainment of the intended learning outcome?

How is assessment used to systematically and purposefully generate data for summative purposes (such as grading, ranking, selection, predicting) and also for formative purposes (such as providing timeous feedback to inform teaching and learning)? 

How accessible is assessment data to academic coordinators, administrators, teaching staff and students 

How is assessment data used diagnostically to inform assessment practice and to improve the curriculum?

What procedures are available to receive, record, process, and return assignments within a time frame that allows students to benefit from formative feedback prior to submission of further assessment tasks?

What policies and procedures exist for secure and reliable recording of assessment results? 

Do clear, consistent and published guidelines/regulations exist for:
•	Marking and grading of results.
•	Aggregations of marks and grades.
•	Progression and final awards.
•	 Compensation and/or condonement of RPL.

What system is in place for maximizing the accuracy, consistency and credibility of results regardless of who is assessing, including

•	Consistency of marking.
•	Concurrence between assessors and moderators on the nature and quality of the evidence which indicates achievement of learning outcomes

Are the approved policies, processes and procedures for assessment and moderation of the institution rigorously adhered to in the programme?

Is it conducted by appropriately qualified staff who are appointed in terms of clear criteria and procedures and who conduct their responsibilities in terms of clear guidelines

Are the guidelines for assessors and moderators clear and are the identified criteria in these guidelines met

How well is internal moderation conducted to provide a reliability check on the marking process and to provide developmental feedback to staff on the quality of their assessment practice? 

How well are external examiners’ reports completed regarding validity of the assessment instruments used in relation to the specific learning outcomes, the quality of student performance, standard of attainment in relation to the learning outcomes, relevant generic qualification standards and professional standards? 

How well are the recommendations for improvement discussed and implemented by academic staff?

Are policies, procedures and regulations in place for the following? 

•	Admission arrangements.
•	Selection criteria.
•	Rules of progression.
•	Selection and development of supervisors.
•	Supervision and reporting arrangements.
•	Assessment and examination requirements.
•	Mechanisms to deal with plagiarism and fraud.
•	Mechanisms to deal with student complaints and appeals.

Are these policies, procedures and regulations for post-graduate study clearly communicated to post-graduate students?

Are there appropriate policies/procedures for the appointment of supervisors, internal and external examiners, and for the examining process? 

How regularly and how well are post-graduate progress, completion and outputs monitored?

Is a senior academic allocated the responsibility to coordinate research programmes, monitor the progress to post graduate students and oversee assessment procedures?

Is a senior academic allocated the responsibility to coordinate structured Master’s programmes, monitor the progress and oversee assessment procedures?

Does monitoring and review of this system take place regularly and include student feedback on the quality of the research experience, supervision and support?

Is there appropriate induction to research and research skills?

Is provision made for additional research training, language and writing skills development and support for under-prepared students, both prior to and during the research process?

Is equity and access responsibly built into the selection criteria and protocols (RPL and alternative assessment protocols, including a combination of interviews, presentations, references and portfolios of previous work)?

Are selection criteria in place to ensure that students admitted to particular research programmes are adequately prepared to undertake the required research?

Are programme guidelines available which make explicit the role and responsibility of both the supervisor w r t performance of research? Do the guidelines address the:
•	Nature, format and expected turn-around time regarding work submitted to the supervisor.
•	Forms of assessment, and communication of feedback to the student, the periodicity of contact between student and supervisor, the schedule for the submission
•	Research ethics, code of conduct, and regulations on plagiarism and intellectual property rights.
•	Examination and qualification requirements

How effective and accessible is funding for post-graduate research?

Are Exit Level Outcomes and assessment criteria (including an understanding of the quality of research achievement required) clearly communicated to students on commencement of study? 

Is at least one external examiner to the institution per dissertation/thesis appointed?

Are examiners appointed on the basis of qualifications, experience, expert knowledge in the research area and independence?

Are clear institutional guidelines available to external examiners on the standard/ quality of research achievement required, on the nature of their task and on institutional examining regulations?

Without undermining the principle of assessment by academic judgement, are assessment decisions made transparently and students are afforded reasonable access to information (e.g. examiner’s report)? 

Are opportunities and appeal mechanisms available to students to defend their thesis? (e.g. through an oral defence).

Are higher degree committees or similar structures in place to consider examiner’s reports qualitatively and make considered decisions about examination outcomes? 

Are there clear guidelines on the quality of research to be attained, including the presentation of research in verbal and written forms?

Are there clear guidelines for post-graduate supervision and are these adhered to? 

Do supervisors have qualifications in the relevant field of study at the same level as (or higher than) the exit level of the postgraduate programme he/she is supervising? 

Has the supervisor relevant and appropriate research track record, as well as experience, expertise and peer recognition in the relevant field of study? 

Are the selection and appointment criteria for supervisors strictly applied?

Are training opportunities provided in the case of inexperienced or new supervisors provided and joint supervision is explored as an option?

Are staff provided with opportunities to engage in research and are they required to engage in research to keep their knowledge current?

How is work-based learning integrated into the curriculum

How is work based learning coordinated, delivered, monitored?  

How is work-based learning recorded, at both the institution and the employer, to keep track of the contents and progress of the student’s learning experience in the workplace? 

How effective is the communication between institution, student, mentors and employers?

How regularly is the monitoring done and what systematic methodology is used? 

How is the feedback utilized?

How well is the mentoring system implemented to ensure that students recognize their strengths and weaknesses in their work develop new and existing abilities and gain knowledge of work practices?

How are students and staff engaged in community service, in this programme?

How community service integrated in the learning programme? 

What is the level of engagement in short course provision by staff in this programme?

Does such provision contribute to knowledge acquisition in this field and does it open up further learning opportunities	Programme organogram with clear lines of accountability 

Programme QMS, role allocation and reporting lines.

Programme group Membership, student feedback

Programme specification, teaching and learning strategy

Teaching and learning methodologies and linage to programme outcomes

Instructional design strategies and examples 

Teaching and learning methodologies and linage to programme outcomes

Learning programme organisation and structure

Programme assessment outcomes

Teaching and learning strategies
Assessment rules and procedure / publication material

Disciplinary policies, procedures and records

Assessment policies, procedures and records

Qualification specifications

Use of appropriate cognitive taxonomy (Blooms) to ensure appropriate spread of question types

Learning programme specification, student guides and assessment instruments.

Qualification specification and learning programme 

Policies and procedures for assessment instruments and examiner reports

Assessment appeals procedures and records

Examiner reports

Statistical and diagnostic analyses of assessment events

Assessment records and planning inputs

Assessment results: Student retention rates per subject/module and per level of the programme

Assessment procedures 

Procedures for recording and verifying assessment records. ITS reports

Assessment policies and procedure

Assessment policy and procedure and reporting structure.

Assessment and moderation policies and procedures

Assessor and moderator appointment criteria and records.

Assessor and moderator guidelines and reports

Internal moderator reports.

External examiner and moderator reports

Programme planning documentation

Policies, procedures and regulations

Communication brochures

Policies, procedures and reports

Throughput and retention reports, programme review documentation

Role allocation and responsibilities and progress reports. 

Role allocation and responsibilities and progress reports

Student feedback surveys

Induction programme

Developmental programmes and attendance records

Equity profile and access policy

Admissions requirements and selection criteria

Guidelines, role and responsibilities allocation
Student feedback surveys

Research niche areas and funding policies.

Student guidelines

Examiner appointment policies and procedures

Examiner resumes

Assessment guidelines

Assessment policies and procedures

Assessment appeals policies, procedures and reports

Faculty board, research committee records

Assessment criteria

Post-graduate supervision policies and reports

Supervisor resumes

Supervisor resume

Appointment criteria and applicant records

Staff development programmes and attendance reports

Research plan

Programme specifications

Work based learning plan, role allocation and responsibilities

Work based learning records

Communication records

Monitoring plans and records

Planning input data

Mentoring plans, records and student survey results

Evidence of community service projects and their outcomes

Linkage to curriculum design

Data and information on the number and range of short courses, frequency of provision, and level of courses	

7.	STUDENT PERFORMANCE 

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
6.5.	Pass, throughput and retention rates 	How do students at every undergraduate level in this programme perform?

How has performance improved in this programme over the past three years? 

How do examiners’ reports (internal and external) provide evidence that qualifying and progressing students are attaining the specified learning outcomes?

How does student performance at undergraduate level compare to that in comparable higher education institutions nationally and/or internationally?

How is student performance monitored and tracked, and how does such information feed into improvement plans?

Do programme coordinators have access to accurate information on the retention and throughput rates for the programme?

Do programme coordinators have systems, procedures and processes in place to ensure that retention and throughput rates are satisfactory and communicated to relevant authorities?

Can the programme justify its race and gender profile in relation to the historical profile of its field and in relation to institutional equity targets?

Does the composition of the qualifying class resemble that of the entering class?

How satisfactory are the completion rates of post-graduate students?

	Data on subject performance, programme performance per level, through-put rates

Tracking data

Examiner’s reports
Qualitative evidence that outcomes are being achieved

Tracking of cohorts to demonstrate performance trends

Performance relative to benchmarks of the DoE

Throughput and retention reports 

Throughput and retention analyses and reports 

Equity profile and programme analysis

Equity of access and equity of outcomes

Data to reflect post-graduate output

8.	RESEARCH OUTPUT

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
8.1. Research	How satisfactory are staff performing regarding published research, participation in national and international conferences and the production of patents and artefacts?

How satisfactory are staff performing w r t the agreed research output in their CPAs?

How has research output in this programme improved over the past three years?
	Data to support statements on lecturer performance regarding research

Staff performance data

Trends data	

9.	CUSTOMER SATISFACTION:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
9.1.	Students

9.2.	Employer Satisfaction

9.3.	Employment Data

	How is student feedback obtained about the effectiveness of the teaching-learning process and the content of modules and the programme as a whole, and how is such feedback used to improve teaching practice and the curriculum?

How satisfied are students with the financial support provided by the institution?

How is employer feedback obtained about the effectiveness of the students, the content of modules and the programme as a whole, and how is such feedback used to improve teaching practice and the curriculum?

How are employment trends analysed and how do they inform programme planning 

How is feedback on the employment of graduates obtained? 

Is the quality of the education and training in a programme acknowledged in the workplace/community/other institutions?	Data and qualitative evidence or feedback

Student survey results

Employer survey results

Employment trends analyses

Survey instruments and results

Letters of commendation	

10.	PEOPLE SATISFACTION:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
10.1.	Staff Satisfaction
	How has staff feedback been obtained regarding the programme provision, teaching and research?

How satisfied are staff with their workload, performance expectations, appraisals and the allocation of resources for programme delivery?

How has such feedback been utilized?

How is lecturer self-evaluation and peer evaluation used to improve teaching practice?

What financial resources are provided for the delivery of the programme?

	Survey instruments

Survey analysis

Programme planning documentation

Evaluation report. Links to staff development

Alignment of planning and resource allocation

11.	PARTNERSHIP PERFORMANCE:

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
11.1. PARTNERSHIP PERFORMANCE
	What types of partnerships exist and how effective have they been in rendering successful students?	Evidence on partnerships and performance related to these; 

Evidence of monitoring of provision	

12.	IMPACT ON SOCIETY

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
12.1. IMPACT ON SOCIETY	How effective has teaching and learning, community service and partnerships been in terms of the value added to students’ industry and civil society?

How does research in this programme contribute to economic development, innovation, and competitiveness?	Evidence of services rendered	

13.	BUSINESS RESULTS

ELEMENT / FOCUS AREA	EVALUATIVE QUESTIONS	SUPPORTING EVIDENCE	
13.1.	Programme review

13.2.	Stakeholder feedback

13.3.	Reports

13.4.	Improvements Achieved 

13.5.	Improvement Recommended

	How are learning materials, teaching and learning strategies, modes of assessment and moderation reviewed

How are the review outcomes used in programme planning, staff development, curriculum development and improving student success

Is the cost effectiveness of the programme reviewed by considering costs, retention and throughput rates, and quality?

How are findings from graduate tracking and employer surveys utilised in programme review

Do students/graduates’ opinion indicate that the majority are satisfied with the programme and with its delivery and assessment practices in particular? 

Where student dissatisfaction has been expressed, how have the teaching staff concerned interpreted this, and has it resulted in efforts to address/improve the situation?

How are examiner, moderator, quality review and other reports utilised to effect improvements. 

Overall what improvements have been achieved in this programme over the last three years?

How are improvement plans supported by resource provision and staff development?

What improvements have you identified as a result of this review, and what are your plans for effecting these improvements?

	Review mechanisms, schedules and reports.
Evidence of triangulation of data to ensure validity and reliability 

Strategic planning documents 

Cost benefit analysis

Survey results and strategic planning documents

Survey results

Survey results, minutes and strategic planning documents

Reports and strategic planning documents

Trends and indicators report

Alignment of resource allocation and improvement targets

Improvement identified