This section frames the review process by setting down reasons for a review, encompassing looking back, appraising the present and moving forward. There are many reasons for reviewing your programme:  

‍

• Understanding the broader strategic context
• Meeting compliance obligations
• Assessing your programme’s strengths and challenges
• Identifying issues and developing approaches to address them
• Maximising opportunities for improvement.

‍

Understanding the broader strategic context

It is important to be aware of the sector’s strategic goals. The New Zealand International Education Strategy for 2022-2030, for example, prioritises the student experience. In particular it highlights in Goal 1 an excellent student experience that includes world-class wellbeing support. If your annual review shows that your investment in marketing has doubled, and yet your investment in the student experience has remained static, then it's likely your approach is out of sync with the wider sector. The strategy’s learner focus champions hospitality (manaaki) and interpersonal connectedness (whanaunga) and thus gives effect to Te Tiriti. 

‍

Meeting compliance obligations

If your programme’s objectives reflect Goal 1 in the International Education Strategy, this is likely to have positive flow-on benefits for your programme’s Code Self-review and Attestation. Outcome 1 A learner wellbeing and safety system (tertiary) lays out clear expectations in terms of a whole-of-provider approach to supporting the diverse needs of learners. Outcome 18 Safety and wellbeing (schools) requires policies and practices that adequately provide for the wellbeing  of international school learners. 

‍

Assessing your programmes strengths and weaknesses

Critical to an effective assessment of strengths and weaknesses is knowing the importance of the exercise. Taking weaknesses seriously allows you to understand how they might be impeding the success of the programme. Similarly, a preparedness to investigate your Unique Selling Proposition (USP) may allow you to seize an opportunity that may otherwise have been missed. 

‍

Identifying issues and developing approaches to address them

Identifying issues is not necessarily intuitive and harder than we think. John Fanselow, Distinguished Alumna at Teachers’ College Columbus University, argues that auto-pilot is a much more powerful driver than we realise. In his classic book Breaking Rules (1987) he advocates being prepared to ‘do the opposite’ in our professional practice and seeing what the outcome will be. By this he means carefully critiquing standard practice and being prepared to implement an innovation that may be counter intuitive.

‍

Maximising opportunities for improvement

Innovative programmes have the imperative to improve at the heart of its strategy. The challenges and opportunities in international education are multifaceted, and so it is important to preserve key foundational principles of business practice. In terms of marketing, Peter F Drucker’s wisdom remains central: 

Marketing is not only much broader than selling… it encompasses the entire business. It is the whole business seen from the point of view of its final result, that is from the customer's point of view
(Webster Jr, 2009, p.20) 

‍

Programmes that prioritise the customer’s point of view, i.e., the international learner, are well placed to frame their reviews in a way that lays the groundwork for substantive improvements. 

‍

Following Druker’s maxim, this topic will outline ways to review a programme in order to provide a more compelling offering for present and future students. The strategies and principles shared are also applicable to other review foci. 

NZQA has identified four core review components:

These components are integral to making sense of the effectiveness of your current practice. If one of these steps are given insufficient attention, the quality of the overall review is likely diminished. Every review must clarify its focus and is undertaken within the context of broader sector dynamics, such as the international education environment, stakeholder relationships, marketing and recruitment activities, wellbeing risk factors, sector policies and developments in professional standards.

‍

Using the NZQA four components as the framework,  key actions, goals, timeframes and outputs can be added to create a review template. Refer to the review template as you read the scenario below. Take time to consider your own response before viewing the recommended actions: 

‍

Component 1 Prepare strategies 

Questions:

• ‍‍How do we know what to focus on?
• What’s the best approach?
• Which stakeholders need to be involved? 

‍

First steps

You identify the key foci of the review to what extent are the majority of  your international learners engaging socially and academically? Teachers, learners, agents, student leaders, international programme managers, accommodation managers and learning advisors among others are invited to participate in the review. You’ve become aware of the issues early on, so you opt for a mid-term/semester review. After consulting stakeholders across the institution and clarifying the key questions, you’re confident you have the narrowed focus you need for your review. You’ve become aware through the cultural intelligence tool, Cultural Atlas, that questions need to be framed in a way to elicit honest responses from learners who are culturally disinclined to complain. 

‍

‍Component 2 Gather strategies

Questions:

• ‍‍How do we gather different kinds of evidence?
• How do we capture diverse voices?
• Are the key review questions addressed?

‍

Information gathering

Interviews, focus groups (qualitative) and anonymous surveys (quantitative) are chosen to gather authentic feedback. Three people are designated time to implement the review. You resolve to survey learner feedback at the mid-point of the next term/semester which allows you to critique any changes in perceptions. An educator with recent research experience assists with the questions and methodology. Input from the marketers, agents, student advisors and accommodation managers is gathered and synthesised. Their input particularly helps in the survey and interview questions for students. Accessible online research findings, e.g. ENZ’s Intellilab student experience surveys, also offers a window to current trends. Strategies for gathering and using international learner voices are carefully considered. Information that provides clues to your international learners’ socialisation and participation in learning is carefully critiqued. 

‍

‍Component 3  Make sense strategies 

‍

Questions: 

• How do we know that our information is accurate? 
• How do we form valid conclusions? 
• What does it mean in terms of changes required?

‍

Careful critique

Once the data is in, you cross-check the qualitative (interviews/focus groups) and quantitative (surveys) findings. You consult with teachers, accommodation providers and agents to discuss the findings. The evidence shows that some students have yet to form any meaningful connections with locals, so the question is how can they be supported in making connections?    Compare these findings with a survey circulated twelve months ago; it appears that you now have more international learners feeling isolated. You invite an educator with research experience from outside your institution to critique your results. Targeted pre-arrival initiatives and early post-orientation interventions are recommended.

‍

Component  4 Decide strategies

Questions: 

• What are our priority actions? 
• How will they be implemented and by whom? 
• In what format do we want our report to be – formal, publicly available, internal only, etc.?
• How will we know that our changes have made a difference?

‍

Improvements

The key findings are agreed upon by the review team and a report is prepared for internal circulation across the institution. Plans are simultaneously prepared for offering early interventions to incoming students. These include co-opting an online buddy programme which encourages current students to reach out to students yet to enrol. This will be complemented by a partnership with a student leadership programme that awards enrolled students points for volunteering on and off campus.  The review is being written with a view to adding findings from a brief follow-up survey at the mid-point of the following term/semester. The findings of this review process will help to refine your review methodology. It includes discussions with marketers and agents to manage students’ expectations. You are learning that your review processes need to be versatile and adaptive. You realise, for instance, that you want to avoid learners experiencing survey fatigue

‍

A review is essentially practical and intended to be conducted with, and presented to, peers, managers and/or stakeholders for a specific purpose that includes addressing gaps in practice. 

Its goal is to implement improvements.

‍

Programme managers and team leaders in the international education sector will be familiar with standard templates for reviews of programmes. There are a range of templates and guidance frameworks that are publicly accessible. What is common to review reviews/evaluations are the following features: 

• Introduction
• Background
• Purpose
• Scope
• Review/evaluation design
• Evaluation principles guiding the review
• Information gathering tools 
• Schedule
• Stakeholders
• Other considerations: Quality checks, Ethics, Limitations and Risks
• Communications: Plan and Dissemination 

‍

Even for a relatively narrow review of a single component of your programme, the above outline provides a helpful schema. NZAID managers overseeing scholarship programmes will be familiar with the templates that MFAT provides. See for example the Evaluation Plan Template.

‍

The scope of review is integral to its shape. It is either theme-based, i.e. critiquing one aspect of the programme, or is more comprehensive, i.e. an annual review. 

‍

It is common practice for education providers to make their policies and procedures for programme and course review publicly available. There are also government bodies and education consultancies that provide useful evaluation indicators. See the examples below.

‍

For tertiary education providers, lecturers who teach research methods are a source of expertise in close proximity. For schools, Te Kete Ipurangi (TKI) provides useful assessment tools and resources that are applicable not only for teachers but also international education managers and practitioners.

Happily, there are reputable organisations with charitable purposes that provide comprehensive resources for evaluation and review of practice. For example, Better Evaluation was established with a common good mandate to assist organisations undertake effective evaluations and change implementation. There are also international benchmarks for evaluation standards. The OECD document Quality Standard for Development Evaluation, for instance, provides a helpfully simple framework for overarching considerations,  clear purpose, careful planning and design, and successful implementation. 

‍

Agents as review participants

A programme review needs to be aware of its immediate institutional context and broader sector context. Clearly international learners are stakeholders whose interests are integral to an international programme’s sustainability. Institutionally, it is important to consider the interlinking between the marketers, the international programme, student services and the teaching profession. Another important stakeholder in the international education sector, usually more closely allied to the marketer, is the agent. Agents represent important intermediaries; cross-border connectors who assist students with their choice of programme and engagement with the host institution. 

‍

The agents represent an important source of information for a review. Reputable agents who have developed track records are more likely to be interested in student welfare and the delivery of products and services, i.e. an excellent student experience. There is interesting research that shows that agents remain actively involved in some international learners’ decision making during the course of their overseas studies. They may, therefore, be an important third-party reference point for understanding your learners’ sense of satisfaction, success and belonging. There are useful resources to assist these agents with their advice. ENZ’s Agent Lab and Understand Aotearoa New Zealand’s bicultural ethos, Understand key Privacy Act Provisions and Understand New Zealand’s education system provide agents with insights into New Zealand’s regulatory frameworks, education system and cultural values. 

‍

The nuts and bolts of gathering information

• Reliability and validity 
• Use of quantitative and qualitative research
• Ethical considerations

‍

Questions and technique: reliability and validity

It is important to attend to the fundamentals of reliability and validity when preparing questions for information gathering tasks. See some key considerations below:

‍

Use of quantitative and qualitative tools

It is common practice for programmes to gather both quantitative and qualitative sources of data as these two types are complementary. It’s helpful to consider their strengths and limitations (especially if the review is investigating the programme’s impact on the international learner experience) by looking at one example of each: surveys (qualitative) and focus groups (qualitative). Irrespective of the approach taken, the purpose and intended use of any collected data needs to be clearly communicated to participants.  

‍

Surveys

One of the most common quantitative methods for information gathering are surveys, which perhaps could be described as the go-to for international programmes. Perhaps this is because surveys can now be easily prepared, delivered and critiqued through online tools such as SurveyMonkey, Qualtrics and Google Forms. 

‍

The strengths of surveys are that they offer:

• Mechanical ease of preparation and delivery
• Breadth of perspective across your target group
• An opportunity to discern trends
• Valid statistical analysis
• Reassurance your programme is on the right track

‍

The limitations of surveys include the following:

• The information gathered is a mile wide yet an inch deep
• Overuse may lead to participant fatigue
• A false sense of achievement, e.g., the self-selecting participants who respond may be 

happier than those who don’t

• The positive findings can be used to paper over programme weakness
• They are generally not designed to identify at-risk learners

‍

Focus groups

Focus groups, in addition to one-to-one interviews, offer rich qualitative data that complement the ‘inch-deep’ data provided by surveys. 

‍

The strength of focus groups is that they offer:

• Diverse participant voices
• Rich insights, anecdotes and a human connection with the participants
• Opportunities for participants to share their experience in-depth
• Clarification or more nuanced interpretation of survey response.

‍

The limitations of focus groups include the following:

• Participants you most need to hear from may be reluctant to participate  
• Out of respect, participants from high power distance countries may be culturally inclined to express opinions they think you’d like to hear
• Some individuals might dominate the discussion if the group is not well moderated
• The focus groups may not be properly representative
• Some participants may be inclined to share more frank opinions in a one-to-one interview.

‍

Human ethics and cultural considerations

If you’re undertaking formal research involving human subjects, then human ethics guidelines are non-negotiable. Ethics guidelines include awareness of and respect for participants’ cultural practices. Culture-specific knowledge is important for an effective exercise of ethics.

‍

It would be helpful to know, for instance, if you asked Malaysian learners to participate in an interview that you were aware they might say yes even if they didn’t want to, i.e. they wanted to ‘save face’ or didn’t want to offend you. You could then stress that they are not disrespecting you if they decline to participate. Or it might be helpful for you to know that German learners appreciate clear, direct and precise explanations for the purpose of your interview, so they can be reassured that you are communicating honestly with them and they understand why you are asking them to participate.

‍

More review tips

• Ensure the purpose of your review is crystal clear
• Be alert to the feedback fallacy; the idea that you can form an objective view without the critique of an external, independent party..   
• Be prepared to collaborate with diverse stakeholders even those who might appear to only have a peripheral interest in your programme.
• When seeking information, respect and look for culturally diverse viewpoints.
• In-flight discoveries: be prepared to take in surprising insights and fresh ways to understand the issues raised.
• Ensure your inquiry team use standardised interview and note-taking techniques
• Aim for a triangulated approach, i.e., three intersecting sources of information. International learner voice is, of course, a key information source.
• Be mindful of unconscious bias when critiquing the data, i.e. do you intuitively favour one source of information over another?
• Don't take negative feedback personally. Be prepared to reflect on feedback that stings 
• Don’t shelve it: use evidence to implement changes even if small. Sometimes 2% adjustments produce multiple benefits.

‍

Embrace the review

The mention of a review can perhaps create a sense of trepidation.  Yet why not adopt a positive attitude? An experienced international director shared her appreciation of what a review offers: 

‍

A review offers an opportunity for encouragement. It shouldn’t be about a big stick. There are no expectations of perfection. It's a way of reflecting on and improving practice. 

‍

Pragmatic considerations

There is not necessarily one right way to undertake a review in an international education context, and its design will depend on its purpose. And there are a number of variables that need to be considered including budgets, time constraints and regulatory demands. Consider strategies for working with time and budget constraints. Many programmes have found themselves dealing with significant pressures as a result of the COVID disruptions and the slow recovery. 

A case study approach is particularly useful for programmes under pressure. Case study types include illustrative, critical instance or cumulative. If standard question types and recording templates are used, you will be able to build a steady body of case study evidence in the course of your day-to-day operations, which will offer rich, in depth data across different facets of your programme ready for review.

‍

Reports are shaped by purpose, scope and frequency. In general, standard report features include statements of intent, methodology, results, conclusions and recommendations. The use of a standard format enhances the readability and impact on the intended audience. 

‍

Report Basics

Clear headings, a coherent flow of ideas and economy of expression are important. These three qualities assist with audience engagement and need to be maintained for either a short or long report. Your readers are likely to be busy and will want to grasp the key points in the timeframe they’ve allocated to digest your report. For a presentation, your introduction will need to succinctly highlight the key questions, issues and key findings.   

‍

The report needs to be tied together by a coherent thread from the beginning to end: what was the issue identified and what are the recommended actions? This is likely to be one of the first questions asked by senior management. They will be looking for the call to action.

‍

There is an increasing number of resources and standards available to assist with the use of plain English. Economy of expression avoids wordiness, repetition and redundancy. 

‍

Figures and tables are useful as visual aids, but they can be overused: some data is more effectively presented in a brief summary paragraph. Appendices are helpful for among other things showing the reader the particulars of your research tools, e.g., interview questions and survey forms. They can judge for themselves the validity of the tools you used. 

‍

Annual reports are more comprehensive and investigate a range of programme components. The basic principles of report writing remain the same: clear statement of intent, standard report features, economy of expression and awareness of the audience. With the greater breadth of review, however, it is important to dedicate sufficient resources to give sufficient attention to each component part. If the review concentrates on some components at the expense of others, the overall quality of the report is diminished and the findings potentially skewed. See the two publicly available report templates below:

Annual programme quality enhancement report

Annual programme director’s review/report

‍

This concluding section introduces a case that illustrates the value of review curiosity and readiness to improve, which can lead to benefits for the wider institution.

‍

A call to action

Reviews are integral to continuous improvement, which is a Education Code of Practice expectation. So, the review is much more than an assessment of your programme; it is a springboard for thinking critically and acting wisely to ensure high standards of performance into the future.

‍

Deliver a review in a way that is not ‘yawn-able’. Present the gaps, implications and recommendations concisely. Address the ‘so what?’ question for decision makers.

‍

There are wider sector-specific factors to consider when conducting reviews. An innovator is cognisant not only of a programme’s strengths and weaknesses, but also the wider market in which it operates. In other words, there are structural dynamics in the international education sector that programme reviewers should take into account.

‍

Firstly, there are a range of stakeholders involved in the delivery of education services and products. The question is how interlinked or interlocked are these stakeholders? The Chinese proverb ‘It takes two hands to clap’ nicely illustrates the importance of a joined-up approach. The scenario in Section 2 described the cooperation of a range of stakeholders in a programme review. Yet it is not always clear how often marketers, international programme directors, student services managers, educators and researchers are in the same room together. They all have parts to play in delivering learner-centred services and products. What is the impact on the consumer (student) experience if they’re not joined-up?

‍

Secondly, there is currently no research institute dedicated to the international education sector in Aotearoa. This is a research limitation of which programme reviewers need to be aware. A regular flow of research enhances our knowledge of the international student experience, including the graduate experience. Knowledge of the graduate trajectories assists us in following Drucker’s maxim: ‘the whole business is seen from the customer’s point of view’. 

‍

The current absence of a dedicated research community reinforces the importance of programmes undertaking robust reviews and committing to steady improvements. ENZ does commission a range of studies that investigate demand-side and supply-side issues, and this provides a useful sector backdrop for reviewers.

Well-developed review policies and practices alert us to the intercultural dynamics to which programme managers need to wrestle. A developing awareness of cultural difference is immensely helpful for meeting the 2021 Education Code of Practice requirements, which include an expectation that education providers will deliver tailored support services to diverse learner groups.

‍

In this topic you explored the ‘why’ and ‘how’ questions in relation to a programme review.  

Scenarios, strategies, tips and a case study provided tools for considering the fundamentals of reviews and report writing. Reviews that investigate the quality of the learner experience not only reflect a core concept in marketing and management but also express te Tiriti values.  

‍

The fluid dynamics in the wider context international education sector was considered and pragmatic solutions were offered for programmes dealing with time and budgetary constraints.  

‍

Reviews represent the benefits of looking back, appraising the present and moving forward. Sound review principles are applicable to different facets of programme operations. In short, reviews offer opportunities to seek continual improvements which have positive flow-on effects for the international learner experience.  

‍

Better Evaluation. (n.d.). Choose methods and processes. https://www.betterevaluation.org/en

‍

Cao, L., & Tran, L.T. (2015). Chinese international students in Australia: An insight into their help and information seeking manners. The International Education Journal: Comparative Perspectives, 14 (1), 42-56.

‍

Darr, C. (2005). A hitchhiker’s guide to validity. Set, 2. 55-56. https://www-nzcer-org-nz.eu1.proxy.openathens.net/nzcerpress/set/articles/hitchhikers-guide-validity

‍

Darr, C. (2005). A hitchhiker’s guide to reliability. Set, 3. 59-60. 

https://www-nzcer-org-nz.eu1.proxy.openathens.net/node/42944

‍

Education Review Office. (2016). Effective internal evaluation for improvement. https://ero.govt.nz/our-research/effective-internal-evaluation-for-improvement

‍

Education Review Office. (2022). How ERO reviews schools/kura with international students. https://ero.govt.nz/how-ero-reviews/how-ero-reviews-schools-kura-with-international-students

‍

Education Review Office. (2016). School evaluation indicators. https://ero.govt.nz/how-ero-reviews/schoolskura-english-medium/school-evaluation-indicators

‍

EIT. (2022). Course review and quality assurance policy. https://www.eit.edu.au/policies/Course_Review_and_Quality_Assurance_Policy.HE.pdf

‍

Evaluation Associates. (n.d.). International programme evaluation. https://www.evaluate.co.nz/evaluation/international-programme-evaluation

‍

Fanselow, J. F. (1987) Breaking rules: Generating and exploring alternatives in language teaching. New York & London: Longman

‍

Grima, J. (2014). Going ‘beyond the manual’: The practicum host experience with event management students. Event Management, 18 (4), 479-485.

‍

Hofstede insights. (2022). National culture.  https://hi.hofstede-insights.com/national-culture

‍

London School of Hygiene and Tropical Medicine. (n.d.). Programme and module management, monitoring and review: Annual programme directors review template. https://www.lshtm.ac.uk/aboutus/organisation/academic-quality-and-standards/programme-and-module-management-monitoring-and

‍

Ministry of Education. (2022). TKI: Assessment tools and resources. https://assessment.tki.org.nz/Assessment-tools-resources

‍

New Zealand Foreign Affairs and Trade. (2022). 'Tools and guides for aid activities: Evaluation plan template'. https://www.mfat.govt.nz/en/aid-and-development/working-with-the-aid-programme/tools-and-guides-for-aid-activities/

‍

NZQA. (2021). Education (Pastoral Care of Tertiary and International Learners) Code of Practice 2021. https://www.nzqa.govt.nz/providers-partners/tertiary-and-international-learners-code/

‍

NZQA. (2022). Guidelines for self-review: The Education (Pastoral Care of Tertiary and International Learners) Code of Practice 2021. https:/ NZQA govt.nz

‍

OECD. (2010). Quality standards for development evaluation. https://www.mfat.govt.nz/assets/Aid-Prog-docs/Tools-and-guides/DAC-Quality-Standards-for-Evaluation.pdf

‍

University of Auckland. (2022). Programme reviews. https://www.auckland.ac.nz/en/staff/learning-and-teaching/quality-assurance-and-continuity/learning-teaching-quality-assurance/programme-reviews.html

‍

University of Dundee. (2021). Quality framework: Annual programme review template and guidance. https://www.dundee.ac.uk/qf/documents/details/annual-programme-review-report-template-and-guidance.php

‍

University of Otago. (2022). Annual programme reports. https://www.otago.ac.nz/administration/academiccommittees/proformas.html#annual-programme-reporting

‍

University of Waikato. (n.d.). Ethical conduct in human research and related activities regulations. https://www.waikato.ac.nz/research-enterprise/ethics/human-ethics

‍

Universities New Zealand (2015). External Review Report: Academic Quality Agency for New Zealand Universities. https://www.aqa.ac.nz/sites/all/files/AQA%20External%20Review%20Report%202015.pdf

‍

Webster Jr, F.,E. (2009). Marketing IS management: The wisdom of peter drucker. Journal of the Academy of Marketing Science, 37(1), 20-27. doi:https://doi.org/10.1007/s11747-008-0102-4

Write. (n.d). Free tools. https://write.co.nz/resources/free-tools