2018 Census External Data Quality Panel: Minutes of Meeting
on 6 March 2019
Date and time
6 March 2019, 9am to 3:30pm
Location
Auckland University of Technology
16th floor, 56 Wakefield Street
Auckland 1010
Present -
Richard Bedford - Chair
panel members Tahu Kukutai
Donna Cormack
Alison Reid
Thomas Lumley
Len Cook
Barry Milne
Ian Cope
Present – Stats
Kathy Connol y, General Manager Census
NZ
Vince Galvin, Chief Methodologist
Gareth Meech, Senior Manager Census and Secretariat
Steph Prosser, Senior Analyst Census
Adele Quinn, Manager Census Analytics
Christine Bycroft, Principal Statistician
Chris Hodgins, Manager Census Geography (afternoon session via phone only)
Meeting minutes
Review previous minutes and action points
The panel confirmed that the minutes contained a fair summary of discussions at the February
meeting.
The Chair and secretariat worked through the action points from the February meeting, removing
those that were now closed.
Alison Reid has been confirmed as co-chair who wil work with Richard to ensure the work of the
panel continues in his absence from May 2019.
The panel noted that hosting some meetings in Wel ington would be a positive step.
The Action Points have been updated in the table at the end of this set of minutes, but there were
some specific conversations regarding the Privacy Impact Assessments (PIAs) to be noted:
AP3-10: Carol outlined the work she has been doing to update the PIAs, making them consistent
and more accessible. A statement about the Stats and Privacy Acts, making it clear that we are
operating under the exemption that applies to statistical and research use, wil be added to an
appendix before the April announcement.
The panel suggested the statements about PIAs should be linked on the website and made more
readily available. It was also noted that there are some broken links to previous documents and it
should be clear which documents have been superseded.
Page 1 of 11
Report back from 27 Feb in-committee meeting and suggested changes to Terms of Reference
At the 27 Feb meeting, the panel spent time discussing how to provide the best value. Ideas
included creating a set of materials that reflect on issues discussed as a more formal way to
document views. These would not be formal reports and may be just framed by a set of questions.
The panel is currently preparing a series of short papers for Stats NZ on iwi data, privacy and
confidentiality and one on what the census is used for and the chal enges faced.
The discussion moved to the terms of reference. A proposed updated version was sent to the
Chair just prior to the meeting and requires the Chair to review the TOR before being presented to
the panel. At the in-committee meeting, the panel agreed the panel should continue to meet until
the around the time of the first release of census data, but wanted it acknowledged that the
extension wil not be easy for panel members as they have busy timetables. There was also
concern expressed that the longer that the panel continues, the more difficult it becomes for
individual members to keep silent about what is being discussed.
Carol commented that it is vital for the panel to continue to meet until first release of census data
and that the advice being given by the panel has been extremely important and influenced many
decisions. Stats risks making decisions in isolation and becoming too insular without the regular
panel meetings.
There was also a discussion at the 27 Feb meeting about what constitutes official statistics.
Census programme timeline update
Kathy presented a slide on the timeline we are targeting noting that not much has changed since
the February meeting. There were questions from the panel about whether the timeline remains
feasible. The Stats NZ response indicated increased confidence in the methods being
implemented but it was noted that the timeline stil has risk attached.
Methods session
Christine reviewed in some detail the methods paper that had been uploaded to the secure site
shortly before the meeting. Christine emphasised that we are people missing from households
were only being replaced when there was a high level of confidence that the replacement
matched the missing individual.
There was quite a wide-ranging discussion about the communication chal enges Stats faces,
including comment on:
- What are the key questions the users are going to have in April?
- What are the key chal enges?
- If the external reviewers publish their report in July, how wil this affect the messaging at
the time and at the time of first release?
- Depending on the timing of the public discussion about what went wrong and why, this
might weaken the impact of the published census data and some positive results arising
from the accelerated use of admin data sources.
- Stats should produce more information in April about the quality of what is being
produced by the 2018 Census.
- With Stats being silent in the media, the media themselves only have the attackers to talk
to, as there is no stated defence.
- When discussing the changed definitions of responses, how wil people know that they
can trust Stats?
Page 2 of 11
- The NZ public did not have a discussion about the extensive use of admin data in the 2018
Census. Stats noted that greater use of admin data was flagged, but not admin
enumeration. How confident is Stats that the public wil understand and be comfortable
with the use of admin enumeration? There are risks that the public wil not be happy with
admin enumeration of family structures, parents and children.
- Stats needs to up front with the public about what linking has been used inside and
outside the IDI ‘spine’
- What is happening with the use of admin enumeration for people in NPDs?
- More elaboration is needed from Stats about:
o The distribution of admin records around the col ection date
o Summary of who is on the margin
o With item imputation – how much is the method stereotyping people eg. smoking
o That there wil be no unit donor imputation (which is good) but wil stil use it for
item imputation
- The people in admin enumeration are different eg. more Māori, Pacific and migrants
- Stats NZ need to clarify some of the detail around administrative data such as:
- what we do from both a usual residence and census night perspective
- how up to date is administrative address information
- - when do different administrative data sources get updated
- why we put some people in a meshblock rather than a household
- how does the administrative prioritised usual residence address get created
- how we wil handle administrative addresses in 2023 as a planned activity as there are
lags in the address information coming through
- how many of the people who were included in the the census file from administrative
sources were from Auckland
During a discussion about how the new admin enumeration methods may improve the key
demographic variables, the panel requested an update on the quality of other variables where
there are no other data sources – smoking being an example.
AP6-1: Ask the chair about whether the panel would benefit from a data quality overview of a
variable with a limited admin enumeration component..
One of the panel members ran through a description of the regularly updated counts slide so that
al panel members had a shared understanding. It was noted that the admin data sources wil not
neatly fit the “census day” point of col ection. For example, some admin data wil be from sources
containing data generated at a range of dates around census day. Some agencies don’t update
their IDI data frequently enough. Stats staff wil need to explain which admin sources were used
and any caveats about their quality and timeliness that data users wil need to know.
A question was also raised about how confident Stats staff are that people have been placed in
the right households and meshblocks (including for electoral purposes). Christine responded that
where we do not have confidence in placing people in households we have placed them in a
meshblock. Stats staff noted that they have been working with the electoral commission on
Central Auckland‘s population and electoral enrolments. There are a number of possible
explanations, including high numbers of hard to reach students and Auckland being cited as the
local destination on immigration cards when this not necessarily the case.
The panel asked about how the methods have treated al population groups – and that Stats
should produce a paper that covers al segments. Stats responded saying there are a number of
Page 3 of 11
internal technical advisory group papers that cover several data areas. One panel member noted
there is limited understanding on the quality of household and family data. The panel suggested
Stats needs to summarise and explain al the changes in a technical report and then work on the
key messages.
The panel discussed whether users want to see information relating to change between census or
is there more interest in actual levels and frequencies in the data. In other words, how have the
characteristics of a set of people changed compared to level relating to those characteristics?
The panel discussed whether it is clear about what has and has not been captured in the admin
data as a result of the methodology decisions made. Of interest here is understanding the
characteristics of meshblock admin enumerations compared with the characteristics of
enumerated meshblock populations without the addition of admin data.
Prison and defence force population data
Stats described their approach to using Department of Correction and Ministry of Defence admin
records to supplement census forms. Stats is planning to use prison and defence admin data for
key demographic information but not about iwi due to the quality of the iwi data. Some members
of the panel expressed serious concerns about the use of prison authority data in place of other
methods as there are additional issues around control and access to people’s data. Coercion can
be an issue and some members were very uncomfortable with the ethical implications and
possible chal enges. Stats responded by informing the panel that a PIA wil be published that
explains our use and that we have been very keen to use as many good quality admin data
sources to improve data in this census.
The panel requested a methods document that covers a range of topics including the method,
PIAs, iwi data decision and ‘what is a response?’ – although some may need to be split.
AP6-2 Present list of papers and approx. completion (or draft) timing so panel can base own
reports off Stats documents (rather than meeting minutes).
Programme benefits and performance indicators
The panel were keen to understand performance measures, especial y in terms of benefits. Carol
presented brief slides on the topic noting that response rates wil look different as this is a proxy-
transition to an admin based census model. Coverage calculations wil change to reflect the new
model. Carol was in the process of reworking the benefits, given the failures, to determine where
the new focus should be to ensure benefits are stil delivered.
The panel noted that it would be a mistake for Stats to say the response rate is higher, and that it
should ‘take the hit’ early to keep it away from the census data releases.
Stats discussed how it has referred to response rates since census day.
Stats noted that there were a lot of KPIs and we should have focused more on the key count and
undercount KPIs.
Dwel ing frame quality
Chris Hodgins dial ed in from Christchurch to answer questions about the paper provided to the
panel on the dwel ing frame. There was a question about the occupation of holiday homes – were
there registers available to help? Stats explained that a cautious approach was taken in the field,
Page 4 of 11
documenting validated numbers and field changes to give confidence to users. It was noted that a
few paragraphs need to be added to the overview paper about the dwel ing frame quality.
AP6-3 Add numbers on ‘address not found’ to the table at the end of the dwel ing frame paper.
Quality plan update
Gareth gave a short presentation on the progress of quality assurance including some more
information about the variable quality rating scale. Gareth would send a TAG reviewed paper on
the subject to the panel prior to the 12 April meeting. The sensitivity analysis slide attracted
questions about what else is possible to be completed. Local government cases were discussed
and whether a smal er council such as Gisborne could be approached alongside larger councils
such as Auckland and Christchurch.
AP6-4 Upload variable quality rating scale paper to workspace before 12 April.
The conversations then started to refer to the variables and admin enumeration sources
spreadsheet that was provided to the panel prior to the meeting. The panel found the document
useful, but Stats noted it is draft and there are some errors including a reference to MOJ sourced
data which needs to be checked and corrected if necessary.
The panel asked questions about whether Stats is doing enough to look at how close the census
response data is to the admin data used to increase coverage. Christine noted work on this,
including statistical assessment of relationships between the two data sets, is being planned.
Kathy gave an update about the meeting with a group from the iwi chairs’ forum. The panel noted
that equity of access is important when talking with some but not al stakeholders. Not al iwi are
represented by the iwi chairs’ forum. Al iwi are highly likely to be planning to use 2018 Census
information for decision-making on and they need to know as soon as possible what data may be
available.
The additional slides with updated Māori ethnicity and descent data generated several questions
which were addressed by Adele. The panel asked for an update on electoral Māori descent data
which is not yet available. A draft paper on ethnic mobility written by Robert Didham wil be
uploaded to the workspace before the next meeting.
AP6-5 Upload ethnic mobility paper to workspace.
The next meetings dates are:
- in-committee only 1 April, Auckland
- 12 April, in Christchurch
- 22 or 23 May in Auckland
- June date to be confirmed, but Wel ington venue to fit in with PANZ conference.
- July TBC
Action log
Ref
Date
Description
Owner
Date
Progress
Status
raised
required
Meeting 2 actions
Page 5 of 11
AP
23/10/18 More information
Panel
By Dec
9/11 Information presented Open
2-4
requested about what
meeting that include 2013 response
investigations we are doing
rates and counts. Gareth to
into 2013 counts and
check with Richard whether
response rates – are we
this should be closed by 23
assuming that 2013 is
Nov.
correct?
6/12 Gareth to discuss with
Richard
18/2 Stats may need a
specific 2013 methods
paper to document any
inconsistencies in approach
and wil be captured in
methodology papers and
metadata documentation.
Gareth to upload proposed
method papers (completed
18 Feb). Panel to review
paper and offer comments
by 6 March.
11/4 Summary spreadsheet
of proposed papers to be
published uploaded on 10
April. Feedback needed
from panel by 30 April.
AP
23/10/18 Need more detail about
Gareth
By 9
Sessions at 30 Oct meeting
Open
2-6
the totality and
Nov
and priority at ful 9 Nov
components of the
meeting.
imputation methods
9/11 Information presented
initial y planned and now
at meeting. Gareth to check
being implemented.
with Richard whether this
should be closed by 23 Nov.
6/12 Gareth to discuss with
Richard
18/2 Spreadsheet detailing
variables and imputation
processes to be sent to
panel prior to 6 March
meeting.
6/3 Methods paper and
variable source
spreadsheet sent to panel.
Confirm with Richard
whether this item can be
closed before 12 April in
committee meeting.
11/4 Gareth to check with
chairs.
AP
23/10/18 Request for data on
Christine
By Dec
Topic for December
Close
2-10
birthplace.
meeting meeting
9/11 To be presented at
Dec meeting.
7/12 Data added to
workspace by 7 Dec
18/2 Richard to discuss
further with Christine
6/3 Richard pop insights set
of data showing net gains
and losses by birthplace
over inter-censual period
Mar 2013 to Dec 2017. Not
Page 6 of 11
on workplan, but planning
further work on using
admin data more
effectively. Country of birth
on list. This request wil be
dealt with outside the
panel.
AP
23/10/18 Re electoral implications,
Kathy and
By 9
Discussion planned for 9
Open
2-16
more information is
Gareth
Nov
Nov meeting.
required about the
9/11 No additional electoral
representativeness of the
data was presented (as not
data and impacts of
complete yet). Updated
imputations. A range of
draft electoral calculations
scenarios, including
to be presented to panel in
constitutional impacts for
Dec meeting.
Māori, would be useful
6/12 Electoral data has not
yet been run – targeting
Feb meeting update.
18/2 Updated electoral
data to be presented to 6
March meeting
6/3 Data is not ready for
presentation to the panel.
To be provided to the panel
before or at the 12 April
meeting. Note that Dot
loves Data has been
contracted to look at the
sensitivity of the detailed
electoral calculations.
11/4 Slides to be included
for April meeting
presentation. Suggest
Close.
Meeting 3 actions
AP
9/11/18
More information about
Gareth
By 7 Dec 6/12 Did not have time for
Close
3-5
what admin data is used in
this item in the Dec agenda.
the method and for what
Wil present information in
purpose. Privacy Impact
Feb.
Assessment update. Add to
14/2 Requested list for al
agenda for December.
variables about what
commitments we have
made for use of admin data
in the past and now. Rol ed
into AP2-6.
6/3 As AP2-6 and 3-10
cover this action and are
stil open, close.
AP
9/11/18
Clarification is needed
Panel
By 7 Dec 6/12 Leave open, not
Open
3-10
about what data is used in
members
addressed yet
the census administration
14/2 PIA includes detail,
enumeration (referred to
but hard to fol ow. Suggest
as IDI data in the
adding a flowchart to aid
presentation), and a list of
communication. Panel to
the current variables being
review the PIA and give
imputed, where it comes
feedback.
from, whether the data
6/3 One panel member had
suppliers are aware of its
provided feedback. Carol
use and whether it uses
stated that the PIAs on the
identifiable or de-
website require further
identified person data.
revision work has started
Page 7 of 11
on this. Has worked with
legal counsel on creating a
generic statement about
the legal basis of the use of
admin data. A statement
wil go into an appendix
before the (likely end) April
announcement. Leave
open.
11/4 Legal basis document
added to shared space.
Suggest close.
AP
9/11/18
Present criteria about the
Gareth and
By 7 Dec 6/12 Some information in
Open
3-11
thresholds being used to
Christine
Dec meeting present, more
make decisions about
info to come after meeting.
methods.
Leave open.
14/2 Panel wil be using
Canadian quality
framework. Request more
information about quality
plans at March meeting.
6/3 An update about the
quality measures was
presented at 6 March
meeting. Variable quality
rating scale paper to be
sent to panel before 12
April meeting.
11/4 Variable rating scale
paper loaded to workspace
10 April. Wil need time to
discuss suggest on agenda
for May meeting. Leave
open.
AP
30/10/18 The Dwel ing Frame
Adele
14 Feb
14/2 To be uploaded to
Close
3-14
evaluation report to be
2019
shared space by 6 March.
shared with the q/a panel
6/3 Uploaded to workspace
AP
30/10/18 Provide information on
Adele/
14 Feb
14/2 Some information was Close
3-15
births and removals from
Chris H
2019
presented to the panel in
the Dwel ing frame during
November. More to be
the live operation
loaded to workspace before
March meeting
6/3 Uploaded to workspace
Meeting 4 actions
AP
7/12/18
Document how multiple
Christine
By 14
14/2 Assign to methods
Close
4-5
de-identified data points in
Feb
team. Confirm priority with
admin sources have been
meeting Richard.
protected from
6/3 Panel sent paper prior
identification once they
to meeting and meeting
are assembled.
session discussed at length.
Close
AP
7/12/18
Document ‘What is the
Carol
By 14
14/2 Although Liz discussed Open
4-6
legal basis that it is OK to
Feb
the issues with the panel at
do what we are doing with
meeting meeting on 5 Feb, panel
admin data and what are
stil wants a statement
the ethical implications we
from Stats about the legal
have considered?’ ‘What
basis of admin sources.
commitment has Stats NZ
Without, it is a risk for the
given to respondents and
panel to endorse. High
data suppliers about what
ethical standard set by the
we are doing with admin
Page 8 of 11
data? What have they
Stats NZ at the 2017 data
said? What are the key
summit.
data sources?’
6/3 This matter continues
to be a source of concern to
panel and Stats is
addressing some of these
concerns in the variable list
(see AP 3-10)
11/4 On agenda for April
meeting. Suggest close
after meeting.
AP
7/12/18
Consult over when Stats
Richard and By 14
14/2 Stats to cover in
Open
4-8
NZ wil return to the panel
Gareth
Feb
meeting on 14 Feb. Check
with a decision about
meeting with Richard if enough
public release of iwi data
information to close.
6/3 iwi data leaders
meeting key messages sent
to panel. Decision made
that Stats wil not talk
publicly about data quality
before the April
announcement. However,
need to have conversation
with iwi before then.
Leave open until iwi data
messaging is confirmed.
Panel is writing up
document to be sent to
Stats about panel view
about quality of iwi data.
The panel noted that the
2018 Census webpage has a
link to information about
iwi that implies it wil be
available. Stats to review
and update.
11/4 On agenda for April
meeting. Suggest close
after meeting.
AP
7/12/18
Update panel once more
Gareth
By 14
14/2 Planned item on 6
Open
4-9
detail has been completed
Feb
March agenda.
on the quality framework
meeting 6/3 Variable quality rating
including an ordering of
scale paper to be sent to
decision making criteria.
panel before 12 April.
11/4 Paper uploaded, set
time in May agenda. Leave
open.
AP
7/12/18
Create a table to
Gareth
By 14
14/2 Did not have time to
Close
4-13
summarise variable quality
Feb
present, planned for 6
measures.
meeting March meeting.
2/6 Added to workspace
AP
7/12/18
Check panel members
Gareth
By 21
14/2 Meeting is booked to
Close
4-17
availability for April
Dec
12 April in Auckland, but
meeting.
wil check with panel
members via email on 22/2.
6/3 Proposed venue for
April meeting is
Christchurch. Logistics to
fol ow. Close.
Page 9 of 11
Meeting 5 actions
AP
14/2/19
Need to clarify admin
Adele,
By 6
6/3 Thomas has written a
Open
5-1
enumeration, admin
Christine
March
paper for the panel about
imputation and other
definitions of admin vs
terminology. Thomas
survey data. Kathy to send
created a short paper of
to SNZ staff for review.
definitions to be reviewed
11/4 Written response to
by Stats NZ.
be developed by Stats for
each paper written by
panel. Date to be
confirmed. Leave open
AP
14/2/19
Load technical papers
Gareth
By 6
6/3 several papers added,
Close
5-3
presented to internal
March
more to come. Any specific
Technical Advisory Group
requests welcome
to the shared space
including approach to
prison and defence admin
enumeration.
AP
14/2/19
Additional ways for
Gareth
By 6
Regular emails each week
Close
5-4
information to get to the
March
panel during the rapid
changes
AP
14/2/19
Find and load 2018 Census
Gareth
By 6
6/3 Added to workspace
Close
5-5
business case document
March
and valuing the census to
the shared space
AP
14/2/19
Revise terms of reference
Richard
By 6
6/3 Sent to Richard for
Open
5-6
mainly around the
March
initial review. Richard to
mechanics of the panel
review by 1 April.
purpose, then for the
11/4 Richard and Alison
panel to review and sign-
reviewed and spoke with
off from Liz and Richard
Carol and Liz. To be
confirmed at April meeting,
effective from 29 April.
Suggest close.
AP
14/2/19
Panel members to add
Tahu and
By 6
6/3 Link added to shared
Close
5-7
links to the previous
Donna
March
documents
research and papers on
Māori descent and
ethnicity they already have
and provide any further
feedback to Stats NZ.
AP5-
14/2/19
Include ‘what is a
Christine
By 10
6/3 No update
Open
8
response’ paper in the list
April
11/4 Paper uploaded to
of method papers that
shared space. Suggest
would assist the panel to
close.
write the reports
Meeting 6 actions
AP6-
6/3/19
Provide the panel
Adele
By 12
11/4 Unable to provide
Open
1
information on smoking
April
counts of lower priority
question data as an
variables for April meeting.
example of a variable
Suggest smoking data and
without other admin
associated quality rating
enumeration sources.
topic for May meeting.
AP6-
6/3/19
Present list of papers and
Gareth
By 12
11/4Updated list loaded to
Open
2
approx. completion (or
April
shared space. Suggest
draft) timing so panel can
close.
base own reports off Stats
documents.
Page 10 of 11
AP6-
6/3/19
Add numbers on ‘address
Chris H
By 12
11/4 To be completed by
Open
3
not found’ to the table at
April
end of April. Suggest keep
the end of the dwel ing
open.
frame paper
AP6-
6/3/19
Upload variable quality
Gareth
By 12
11/4 Uploaded to space.
Open
4
rating scale paper to
April
Suggest close.
workspace before 12 April.
AP6-
6/3/19
Upload ethnic mobility
Gareth
By 12
11/4 Original uploaded on
Open
5
paper to workspace
April
29 March. Updated version
2 uploaded on 10 April.
Suggest close.
Page 11 of 11
