2018 Census External Data Quality Panel: Minutes of Meeting
on 12 April 2019
Date and time
12 April 2019, 9am to 3:30pm
Location
Stats NZ
BNZ Centre, 120 Hereford Street
Christchurch
Present -
Richard Bedford – co-chair
panel members
Alison Reid – co chair
Barry Milne
Donna Cormack
Ian Cope – via video conference
Len Cook – via video audio conference
Tahu Kukutai – via audio conference
Thomas Lumley
Present – Stats
Adele Quinn, Manager Census Analytics
NZ
Carol Slappendel, Deputy Government Statistician
Christine Bycroft, Principal Statistician
Gareth Meech, Senior Manager Census Data Quality/ panel secretariat
Kathy Connol y, General Manager Census
Liz MacPherson, Government Statistician
Steph Prosser, Senior Analyst Census
Vince Galvin, Chief Methodologist
Meeting minutes
Review previous minutes and action points
Richard Bedford chaired the meeting. This is likely to be the last ful panel meeting that he wil chair. The ful
panel meeting in May wil be chaired by newly appointed co-Chair Alison Reid.
The panel confirmed that the minutes contained a fair summary of discussions at the March meeting.
The Chair agreed to work through the action points at the end of the meeting to ensure most time was
devoted to methods. At that time, the Chair and secretariat worked through the action point list from the
March meeting, removing those that can be closed.
Updated Terms of Reference
The Chair noted that the panel had an opportunity to review the updated version and that both Chairs were
happy to endorse version 2.0 of the External Data Quality Panel Terms of Reference. The substantive
changes focus on the extension of the panel sitting period and the recognition of a co-Chair arrangement.
This version wil be published on the website on, or soon after, 29 April 2019.
Report back from 1 April in-committee meeting
At their panel only meeting on 1 April, members focused on the internal papers they were preparing on
specific issues, and discussed whether they had enough information from Stats NZ to give more formal
advice relating to methods. The fol owing papers have been sent to Stats NZ. These discuss issues or
concerns where the panel needs additional clarification:
- “Iwi data in 2018 census” received by Stats NZ on 22 March 2019
- “Census 2018 - issues to consider in assessing data quality and inter-censal comparability” received
5 April 2019
- “Use of administrative data in Census 2018” received 4 April 2019
- “What is the IDI” received 8 April 2019
Page 1 of 10
Stats NZ thanked the panel for producing the documents and reiterated that they found them useful. Kathy
noted that the presentation that had been prepared for this meeting would try to address some of the
significant issues raised. Stats NZ wil produce a written response back to the panel for each paper in due
course.
The panel was not able to further its assessment of the overal methodology that Stats NZ was employing to
improve coverage of the census, due to the absence of documentation. Once this documentation is
available, the panel wil need time to ful y reflect on the methodology that had been employed.
The Chair also noted that there had been a lot of negative media articles about the 2018 census in the
previous few weeks, and that members are not able to publicly comment on issues being raised. This was a
source of frustration for some members of the panel.
The panel are supportive of Stats NZ being as transparent as possible in its announcements on 29 April.
Methods and data quality session
Christine presented slides that provided a refresher on the completed methods work to date. As the slides
were only loaded to the shared workspace on the morning of the meeting, panel members had not had a
chance to see them prior to the meeting.
At several stages of the presentation, panel members noted discomfort in Stats NZ using the term “admin
enumeration” as it promotes a sense of enumeration, when none existed before census col ection. Stats NZ
is actual y careful y selecting additional people from lists. The panel noted the risks around inventing
language ‘after the fact’. Stats NZ acknowledged that their language around this is developing. Some
international sources use the same term, but the method utilised here is not exactly the same as overseas.
Some of the methodological approaches taken by Stats NZ have not been done in other countries.
It was also suggested by a panel member that Stats NZ needs to employ statistical terminology and
language, rather than terms that imply value judgements.
The panel suggested changing the language ‘Census undercount’ into something like ‘indicative coverage
gap’. Undercount suggests it has been measured by the PES.
The panel also suggested that the term ‘signs of life’ should be amended to read something like ‘signs of
activity’.
The panel noted that Stats NZ needs to be very clear that they are using real data from admin sources and
that it is not imputing data for individuals. It also needs to be very clear that reference is being made to
individuals and not households.
When discussing the IDI spine, the panel noted that it is important for StatsNZ to be clear if the spine has
names attached.
AP7-1 Need to be clear what the IDI spine is, and produce a document col ating date ranges for each spine
data source used in the 2018 Census.
The panel questioned how Stats NZ removed duplicates in 2013 Census. The response was that there was
no process for this in place in 2013. Methods are much more rigorous in that area in 2018.
During the discussion about admin households, it was noted that the methods quality assurance paper
provided to the panel prior to the meeting contains reference to making an arbitrary judgement about
where the line was drawn when to including or excluding people in the model. Stats NZ responded that
there is a method behind this and it can be better explained.
Page 2 of 10
During the presentation on ethnic distribution by source group, the panel asked for similar analysis of NZ
European and Asian level 1 ethnicity alongside the Māori and Pacific peoples analysis provided.
AP7-2 More analysis of ethnic distribution for NZ European and Asian ethnic groups and the percentages of
these that contribute to the Māori descent population.
Advice was given by the panel to avoid saying that the 2018 Census data is ‘better than’ 2013 without
evidence – Stats NZ needs to be cautious about such comparisons in the upcoming 29 April media release.
With reference to the slide about minimum accuracy targets, the panel raised the question around what
Stats NZ’s customers wil be more interested in - levels of change or actual counts. Users expect consistency
– but a balance needs to be struck. It was noted that the denominator for many significant rates wil have
changed.
Reference was made to the variable warrant of fitness document for some key variables that the panel
wants to see at the May meeting.
Stats NZ presented summaries of sensitivity analysis work completed to date.
The panel suggested that Stats NZ either run a technical seminar for DHBs or ensure they have been invited
to the current seminar series.
The panel was interested in reading the Dot Loves Data report about electoral count and boundary
sensitivity.
AP7-3 StatsNZ to load DLD report to shared workspace.
StatsNZ presented proposed changes to iwi count methodology to create more accurate historic census
counts on the 2018 classification. The panel noted that if the method is refuted, it could undermine the
2013 counts and that new iwi wil not have been fairly represented given the list effect in the questionnaire.
There was not clear support expressed by the panel for the changes.
There was some discussion about the use of data from prisons. Tahu and Donna expressed serious concerns
about the ethics of obtaining identifiable prisoner data from Corrections without prisoner consent and
strongly advised Stats NZ against doing so.
The panel asked whether Māori had been involved in the decision making. They also expressed the view
that Stats NZ needs to be unequivocal in its April 29 media release that the 2018 Census wil not produce
high quality iwi data.
Programme critical path and April key messages
The panel would like more definite timeframes about the drafting of method papers, so that they can take
this into account when finalising their meeting schedule and work plan through to September.
AP7-4 Stats to update method paper timeline spreadsheet.
Regarding the key messages for the 29 April announcement, the panel provided the fol owing advice:
- Use terms such as ‘indicative coverage estimate’ rather than ‘undercount’.
- Be clear that Stats NZ are not talking about the official PES measure, but an ‘indicative coverage
measure”
- Tel the public that some of the more detailed results might not be of sufficient quality to be
termed “official statistics”
Page 3 of 10
- Say that a lot of the methods work is brand-new and that while it had not been anticipated that
some of the changes in approach would have to be used for the 2018 census, there are some
advantages in implementing these approaches.
- There wil not be a census in 2021
- ‘Fit for purpose’ can be problematic – perhaps talk about ‘known quality’
- Note that StatsNZ has done more work to assess the quality of census data than ever before
- StatsNZ is in a much better place now to start development of the next census
- Can we be up-front about names and addresses and how we do the linkage?
- Make sure StatsNZ documents why we think we can use admin data – what are the facts about
this?
- Need to be clear about what anonymisation is in this context.
- If talking about Māori and iwi data it would be best to have a te reo version of the media release so
it can be used quickly by Māori language media.
- Whether StatsNZ should publish census night counts at the same time as usual residence.
- Suggested StatsNZ should be confident about what the quality of the data is and is not.
- Shouldn’t underplay problems with variables such as families and households.
- Acknowledge that because this methodological work was not planned, there was not an
opportunity to have an open and honest conversation with the public and Māori stakeholders.
Need to be more upfront about what this means for Māori data governance going forward
Legal basis and ethics session
Carol shared StatsNZ interactions with the Office of the Privacy Commissioner. They have reviewed al three
PIAs and provided feedback. The OPC is comfortable with Stats NZ use of administrative data in the census
from a privacy perspective.
The Privacy Act al ows for data to be re-purposed for statistical and research purposes as long as the
outputs are confidentialised, which StatsNZ always ensures.
StatsNZ asked the panel about their degree of confidence in the method. After discussion, the Chair said
they would discuss at the in-committee meeting on 15 April. There may not be a general statement from
the panel in April.
The next meetings dates are:
- in-committee only 15 April, Auckland
- in-committee only 8 May, Auckland
- ful meeting 23 May in Auckland
- ful meeting 24 June in Wel ington (to fit in with PANZ conference on 20 and 21 June).
- July TBC in May
Action log
Ref
Date
Description
Owner
Date
Progress
Status
raised
required
Meeting 2 actions
AP 2-4
23/10/18 More information
Panel
By Dec
9/11 Information
Closed
requested about what
meeting presented that include
investigations we are
2013 response rates
doing into 2013 counts
and counts. Gareth to
and response rates –
check with Richard
are we assuming that
whether this should be
2013 is correct?
closed by 23 Nov.
6/12 Gareth to discuss
with Richard
Page 4 of 10
18/2 Stats may need a
specific 2013 methods
paper to document
any inconsistencies in
approach and wil be
captured in
methodology papers
and metadata
documentation.
Gareth to upload
proposed method
papers (completed 18
Feb). Panel to review
paper and offer
comments by 6 March.
11/4 Summary
spreadsheet of
proposed papers to be
published uploaded on
10 April. Close
AP 2-6
23/10/18 Need more detail
Gareth
By 9
Sessions at 30 Oct
Closed
about the totality and
Nov
meeting and priority at
components of the
ful 9 Nov meeting.
imputation methods
9/11 Information
initial y planned and
presented at meeting.
now being
Gareth to check with
implemented.
Richard whether this
should be closed by 23
Nov.
6/12 Gareth to discuss
with Richard
18/2 Spreadsheet
detailing variables and
imputation processes
to be sent to panel
prior to 6 March
meeting.
6/3 Methods paper
and variable source
spreadsheet sent to
panel. Confirm with
Richard whether this
item can be closed
before 12 April in
committee meeting.
11/4 Panel stil
requires ful er
methods paper which
is planned to be
written. Close
AP 2-16 23/10/18 Re electoral
Kathy and By 9
Discussion planned for Closed
implications, more
Gareth
Nov
9 Nov meeting.
information is required
9/11 No additional
about the
electoral data was
representativeness of
presented (as not
Page 5 of 10
the data and impacts
complete yet).
of imputations. A
Updated draft
range of scenarios,
electoral calculations
including
to be presented to
constitutional impacts
panel in Dec meeting.
for Māori, would be
6/12 Electoral data has
useful
not yet been run –
targeting Feb meeting
update.
18/2 Updated
electoral data to be
presented to 6 March
meeting
6/3 Data is not ready
for presentation to the
panel. To be provided
to the panel before or
at the 12 April
meeting. Note that
Dot loves Data has
been contracted to
look at the sensitivity
of the detailed
electoral calculations.
11/4 Slides included in
April meeting
presentation. Close.
Meeting 3 actions
AP 3-10 9/11/18
Clarification is needed
Panel
By 7 Dec 6/12 Leave open, not
Closed
about what data is
members
addressed yet
used in the census
14/2 PIA includes
administration
detail, but hard to
enumeration (referred
fol ow. Suggest adding
to as IDI data in the
a flowchart to aid
presentation), and a
communication. Panel
list of the current
to review the PIA and
variables being
give feedback.
imputed, where it
6/3 One panel
comes from, whether
member had provided
the data suppliers are
feedback. Carol stated
aware of its use and
that the PIAs on the
whether it uses
website require
identifiable or de-
further revision work
identified person data.
has started on this.
Has worked with legal
counsel on creating a
generic statement
about the legal basis of
the use of admin data.
A statement wil go
into an appendix
before the (likely end)
April announcement.
Leave open.
Page 6 of 10
11/4 Legal basis
document added to
shared space. Close.
AP 3-11 9/11/18
Present criteria about
Gareth
By 7 Dec 6/12 Some
Closed
the thresholds being
and
information in Dec
used to make decisions Christine
meeting present, more
about methods.
info to come after
meeting. Leave open.
14/2 Panel wil be
using Canadian quality
framework. Request
more information
about quality plans at
March meeting.
6/3 An update about
the quality measures
was presented at 6
March meeting.
Variable quality rating
scale paper to be sent
to panel before 12
April meeting.
11/4 Variable rating
scale paper loaded to
workspace 10 April.
Close and add to May
meeting agenda.
Meeting 4 actions
AP 4-6
7/12/18
Document ‘What is the Carol
By 14
14/2 Although Liz
Closed
legal basis that it is OK
Feb
discussed the issues
to do what we are
meeting with the panel at
doing with admin data
meeting on 5 Feb,
and what are the
panel stil wants a
ethical implications we
statement from Stats
have considered?’
about the legal basis of
‘What commitment
admin sources.
has Stats NZ given to
Without, it is a risk for
respondents and data
the panel to endorse.
suppliers about what
High ethical standard
we are doing with
set by the Stats NZ at
admin data? What
the 2017 data summit.
have they said? What
6/3 This matter
are the key data
continues to be a
sources?’
source of concern to
panel and Stats is
addressing some of
these concerns in the
variable list (see AP 3-
10)
11/4 On agenda for
April meeting. Some
of the panel does
not agree that the
Stats Act is not
Page 7 of 10
prescriptive. Stats
wil provide a brief
written response on
the paper. Close.
AP 4-8
7/12/18
Consult over when
Richard
By 14
14/2 Stats to cover in
Closed
Stats NZ wil return to
and
Feb
meeting on 14 Feb.
the panel with a
Gareth
meeting Check with Richard if
decision about public
enough information to
release of iwi data
close.
6/3 iwi data leaders
meeting key messages
sent to panel. Decision
made that Stats wil
not talk publicly about
data quality before the
April announcement.
However, need to
have conversation
with iwi before then.
Leave open until iwi
data messaging is
confirmed. Panel is
writing up document
to be sent to Stats
about panel view
about quality of iwi
data. The panel noted
that the 2018 Census
webpage has a link to
information about iwi
that implies it wil be
available. Stats to
review and update.
11/4 On agenda for
April meeting. Close.
AP 4-9
7/12/18
Update panel once
Gareth
By 14
14/2 Planned item on
Open
more detail has been
Feb
6 March agenda.
completed on the
meeting 6/3 Variable quality
quality framework
rating scale paper to
including an ordering
be sent to panel
of decision making
before 12 April.
criteria.
11/4 Paper loaded, set
time in May agenda.
Leave open.
Meeting 5 actions
AP 5-1
14/2/19
Need to clarify admin
Adele,
By 6
6/3 Thomas has
Closed
enumeration, admin
Christine
March
written a paper for the
imputation and other
panel about definitions
terminology. Thomas
of admin vs survey
created a short paper
data. Kathy to send to
of definitions to be
SNZ staff for review.
reviewed by Stats NZ.
11/4 Written response
to be developed by
Page 8 of 10
Stats for each paper
written by panel. Date
to be confirmed. Close
AP 5-6
14/2/19
Revise terms of
Richard
By 6
6/3 Sent to Richard for Closed
reference mainly
March
initial review. Richard
around the mechanics
to review by 1 April.
of the panel purpose,
11/4 Richard and
then for the panel to
Alison reviewed and
review and sign-off
spoke with Carol and
from Liz and Richard
Liz. To be confirmed at
April meeting,
effective from 29 April.
Close.
AP5-8
14/2/19
Include ‘what is a
Christine
By 10
6/3 No update
Closed
response’ paper in the
April
11/4 Paper uploaded
list of method papers
to shared space. Close.
that would assist the
panel to write the
reports
Meeting 6 actions
AP6-1
6/3/19
Provide the panel
Adele
By 12
11/4 Unable to
Open
information on
April
provide counts of
smoking question data
lower priority variables
as an example of a
for April meeting.
variable without other
Suggest smoking data
admin enumeration
and associated quality
sources.
rating topic for May
meeting.
AP6-2
6/3/19
Present list of papers
Gareth
By 12
11/4 Updated list
Open
and approx.
April
loaded to shared space
completion (or draft)
however panel
timing so panel can
requires more specific
base own reports off
dates quickly to know
Stats documents.
when to book
upcoming meetings.
Leave open.
AP6-3
6/3/19
Add numbers on
Chris H
By 12
11/4 Additions were
Closed
‘address not found’ to
April
made, the new paper
the table at the end of
was uploaded. Close.
the dwel ing frame
paper
AP6-4
6/3/19
Upload variable quality Gareth
By 12
11/4 Uploaded to
Closed
rating scale paper to
April
space. Close.
workspace before 12
April.
AP6-5
6/3/19
Upload ethnic mobility Gareth
By 12
11/4 Original uploaded Closed
paper to workspace
April
on 29 March. Updated
version 2 uploaded on
10 April. Close.
Meeting 7 actions
AP7-1
12/4/19
Produce document
TBC
By 22
Open
about what the IDI
May
spine is and is not.
Include date ranges for
Page 9 of 10
each spine data source
used in the 2018
Census.
AP7-2
12/4/19
Analysis of ethnic
Adele
By 22
Open
distribution for NZ
May
European and Asian
ethnic groups and the
percentage of each
source that
contributes to Māori
descent.
AP7-3
12/4/19
Load Dot Loves Data
Gareth
By 29
12/4 Completed
Close
electoral paper to
April
shared workspace.
AP7-4
12/4/19
Update method paper
Gareth
By 29
Open
timeline spreadsheet.
April
Page 10 of 10
