A.3 COVID-19 and student achievement in Aotearoa New Zealand Te KOWHEORI-19 me te whakatutukitanga ākonga i Aotearoa

A.3 COVID-19 and student achievement in Aotearoa New Zealand | Te KOWHEORI-19 me te whakatutukitanga ākonga i Aotearoa

Section 8.4.3 summarises the results of research that Progressive Achievement Test (PAT) data held by the New Zealand Council for Educational Research (NZCER). The research did not use Stats NZ's Integrated Data Infrastructure.

A.3.1 About the PAT tests | Mō ngā whakamātautau PAT

PAT assessments have a long history in New Zealand, with the first PAT test developed by NZCER in the late 1960s. The tests have been regularly updated to stay current with the curriculum and testing standards. They have been one of the most widely used assessment tools in New Zealand schools.

A student's raw score (that is, how many questions they got right on a particular test form) is converted into a scale score, using a psychometric Rasch model. This score indicates both how difficult the questions were that the student could reliably answer and where that sits relative to curriculum expectations. For the purposes of our analysis, a change in the scale score of roughly 5–7 units can be thought of as corresponding to one year's average progress in mathematics. A difference of roughly 7–9 units is approximately one year of learning in reading comprehension.

Scale score points were converted into approximate weeks of learning, by assuming that six scale points represents 38 weeks of learning in mathematics, and that eight scale points represents a similar period of learning in reading comprehension.

A.3.2 Data | Ngā raraunga

To allow for the inclusion of school-level demographic information, the NZCER student PAT database was merged with the Ministry of Education's school directory.

As PAT assessments have historically been disproportionately administered in term 1, analysis was restricted to term 1 exams only. This gave a single, clean measure of achievement for each student in the dataset for each year they were present.

To control for school-specific factors that were not in the database, analysis was restricted to those schools that used the PAT mathematics assessment for at least ten students each year between 2020 and 2023. For PAT reading comprehension assessments, the analysis window was extended to five years, covering 2020–2024 (providing information primarily about 2019 through 2023).

The dataset was restricted to English-medium, state and state-integrated schools to ensure alignment between the assessments used and the New Zealand Curriculum. PAT tests are available only in English, and many private schools do not follow the New Zealand Curriculum in full.

A.3.3 Analysis populations | Ngā taupori tātari

Ethnicity in the PAT database is encoded into the following categories: NZ European; detailed European subgroups (such as Australian or Dutch); Māori; Pacific subgroups; Southeast Asian subgroups; Chinese; Indian; other Asian subgroups; Middle Eastern; Latin American; African; Other Ethnicity; and Not Stated.

Modelling was done using three complementary population frames. The first and primary frame was a stable-schools analysis population that included all schools that consistently used PAT exams throughout the analysis period. The second looked at each of the three Equity Index (EQI) bands, and the third frame looked at ethnicity and gender focused subsets. Ethnicity groups were pooled based in part on cultural affinity but also on similar achievement coefficients in the models themselves. For this reason, North Asian (Chinese, Korean and Japanese) and South and Southeastern Asian populations were distinguished.

Table 7: PAT test data: Analysis populations, mathematics

Table 8: PAT test data: Analysis populations, reading comprehension

A.3.4 Statistical methods | Ngā tikanga tātari tatauranga

All analyses used linear mixed-effects regression fitted in R (version 4.5 or higher) with the lme4 and lmerTest package, estimated by restricted maximum likelihood. Inference for fixed effects used Satterthwaite degrees of freedom as implemented in lmerTest, and the analysis reports model-based 95% confidence intervals and two-sided p-values.

The causal contrast was defined through a differences-in-differences design comparing Auckland (as defined by the Regional Council boundary) with the rest of New Zealand across multiple years. This removes changes that both groups would have experienced anyway (such as national curriculum changes), so that what remains is the extra change associated with Auckland's additional disruptions. A key assumption is that, without Auckland-specific disruptions, Auckland and the rest of New Zealand would have followed similar underlying movements.

Time was represented with an 'exposure year' factor that linked each term 1 test to the school year immediately preceding it, took the earliest pre-COVID-19 exposure year as the baseline, and included a binary region indicator for Auckland versus the rest of New Zealand. To improve comparability and reduce confounding, models further adjusted for student year level, gender, school socioeconomic context via the Ministry of Education's Equity Index (which was treated as continuous) and broad ethnic-group indicators where sample sizes permitted stable estimation.