Appendix 2: Explanation of 2014 Kiwis Count Calculation Changes

A change in the Kiwis Count IT system in 2014 gave the Kiwis Count team an opportunity to review the way Kiwis Count figures are calculated. As a result, two changes have been made to better account for the rolling average nature of the quarterly results:

1 The averaging of results across six months (i.e. the rolling average calculation itself).

2 The calculation for statistical significance in changes between quarterly results from two overlapping time periods.

These changes are explained here.

All annual and quarterly results since 2012 have been recalculated in line with these changes and this report and future reports will use the recalculated figures. In summary, the new results show some minor changes in numeric value for some services and there are a few more services which are now identified as having significant changes than with the previous significance calculation.

The Rolling Average Calculation

In order to ensure a sufficient sample size, Kiwis Count quarterly releases report on the past two quarters of data. For example, a September quarterly release, will report on data collected from April to September, and compare these results to those collected between January and June. Using two quarters of data boosts the sample size to over 1,000 per quarterly release.

In the past, calculations for all quarterly results were based on averaging the aggregate results from two quarters of data, i.e.

For example, if the SQS score for a service in the June quarter was 70 and for the September quarter was 72, then the SQS score for the September rolling-average time period would be 71.

Going forward, calculations for all results will be based on pooling two quarters of data into one six monthly sample, i.e.

The two methods will give the same result if the same number of people answered a particular question in both Quarter (n-1) and Quarter (n)

However, if there are more responses in Quarter (n-1) than Quarter (n), averaging the quarters under the old method meant that the responses in Quarter (n-1) were worth less with respect to the final result than the responses in Quarter (n). Therefore, the new method has the advantage that each survey response will be worth the same with respect to the final result of the period. The new method also has the advantage of being less IT intensive to calculate.

The Statistical Significance Calculation

In the past, calculations for statistical significance in changes in service quality scores were made:

• for quarterly results, using the rolling average samples, and
• for both annual and quarterly result, at the 90% significance level (to be consistent with the methodology used to measure significant change between 2007 and 2009 in the 2009 Kiwis Count report).

Quarterly Rolling Averages

This meant the statistical significance test for differences in quarterly results was being applied on non-independent samples. This is illustrated below, where if significance is calculated on R2 against R1, sample Q2 is repeated:

Including Q2 in both R1 and R2 had two opposing effects on the significance testing. First, it increased the sample size making the change appear more significant. Second, it averaged out the difference between the quarterly results, making the change appear less significant. The second effect is larger, meaning that the previous method was calculating significance around the 97%-98% significance level.

Going forward, for results from 2012 onward:

• The calculations for statistical significance of changes in quarterly results will remove the common quarter sample from both rolling samples. In this example, calculating the significance of R2 against R1, will mean Q2 is removed from both rolling samples and effectively the significance of Q3 is calculated against Q1.

This is the most commonly used method statisticians use to deal with non-independent samples.

• Calculate significance at the 95% level.

Because the previous method (non-independent samples and testing at 90%) produced results at the 97%-98% significance level, the new method (calculating quarterly significance on independent samples at the 95% level) means more services are identified as having quarterly significant changes than previously published.