2018 Census External Data Quality Panel: Minutes of Meeting
on 14 February 2019
Date and time
14 February 2019, 9am to 3:30pm
Location
Stats NZ, Wel ington Office
8 Gilmer Terrace
Wel ington
Present -
Richard Bedford - Chair
panel members Tahu Kukutai
Donna Cormack
Alison Reid
Thomas Lumley
Len Cook
Barry Milne
Ian Cope
Present – Stats
Kathy Connol y, General Manager Census
NZ
Vince Galvin, Chief Methodologist
Gareth Meech, Senior Manager Census and Secretariat
Steph Prosser, Senior Analyst Census
Adele Quinn, Manager Census Analytics
Christine Bycroft, Principal Statistician
Meeting minutes
9-10am – Review previous minutes and action points
The Chair and secretariat worked through the action points from the December meeting. Specific
updates are noted in the second half of this document.
The Chair also noted that panel members are not able to give good advice if they are not given
time to review the materials prior. The slide pack for this meeting was only finalised by Stats NZ in
the morning of the meeting. This cannot happen for future meetings.
Feedback from combined external reviewers and data panel meeting 13 February
With Richard being less available after April, Liz has agreed to appoint a co-chair to the panel.
Alison Reid has agreed to do this and wil work closely with Richard. This needs to be formalised.
Ian wil be in NZ until after the March meeting.
Panel wil start preparing a report which focusses on methods used to improve census coverage
that wil be given to Liz at the end of April, or beginning of May. The panel wil require more
written information on methods and has started preparing documents addressing relevant issues
including legal and privacy.
The panel suggested that Stats NZ needs to make a distinction between what wil be official
census counts and what data wil be provided in other ways. Equity of access as a result of uneven
responses is a concern as some groups wil have good data while for others the data quality wil
be much more variable. Data quality wil be more variable than in the past.
A question was raised about the key users of census and whether they wil receive quality data.
Page 1 of 11
The panel noted there wil be method papers needed to help them produce the report(s). Carol
discussed the possibility of producing provisional counts before our official first release. The panel
responded noting that the data is not just late but that it won’t have everything present and that
we should say what kind of quality measures we wil be producing to support use. There wil likely
be a number of technical caveats.
The panel also suggested that if we release some provisional but not other data there could be
equity concerns. We need to flag if we can’t release data as soon as possible. Stats NZ noted that
this may be more around which products contain data rather than not being prepared to provide
data in other ways.
The panel’s final report is being planned for delivery in September 2019.
AP5-1: Need to clarify admin enumeration, admin imputation and other terminology. Stats NZ to
review short paper by Thomas on definitions produced by a panel member.
AP5-2: Provide list of proposed external method documents to support communication including
‘What is a response?’
The panel also made the point that the Government Statistician has a duty of care to make sure
the data is trustworthy – that this is not market research findings. Stats NZ noted that we need to
take a ‘top-down’ view of the situation.
Data and methods update
Stats NZ presented slides updating basic counts and Māori ethnic data and then moved onto the
recent decision made about methods. A comment was made about the Māori ethnic data
presented that Stats NZ wil need to be able to show evidence to justify adding different
categories of data together (eg. 2018 survey responses + 2013 Census data + admin data) and
how this improves the quality and usefulness of the dataset.
The panel noted that with the methods being presented that we may end up with higher and
lower quality data sets.
During the presentation the panel asked to see more detailed papers that support the decisions
being made.
AP5-3: Gareth to make internal papers that have been sent to the internal Technical Advisory
Group available to the panel including the occupancy misclassification paper and PES
methodology change papers.
During the discussion about the preferred method and decision to progress with ‘Individual
focused approach’ that panel noted:
- The method doesn’t impute households particularly for NZ Dep
- The original plan B ran the risk of ‘messing up’ household data. The enhanced ‘individual
focused method may address some of these issues
- Stats NZ wil need to be able to explain why this option was chosen
- That the impact of these changes may lead to the 2018 dataset being more suitable for
producing ‘level or prevalence’ data rather than measuring intercensal changes (with
impacts of time series comparability)
- Most users would want us to prioritise change of absolute level
Page 2 of 11
The panel noted that the rapid change in approach means that the panel is catching up if we only
meet monthly. Is there a way to keep the panel better informed more frequently?
AP5-4: Stats NZ to propose additional contact channels with the panel to aid engagement during
rapid changes to methods.
Non-private dwel ings
During the presentation section showing the approach to admin enumeration of prisons and
defence establishments, the panel raised a concern that prison data is not necessarily provided
voluntarily and agrees data col ected on prisoners’ iwi may be poor. The panel questioned the
overal quality of the prison data.
AP5-3: Internal paper explaining approach to prison and defence admin enumeration to be
circulated to the panel.
Māori descent imputation discussion
Stats explained that the intended approach to Māori descent imputation for 2018 was to not
impute for non-response. But given the higher non-response and that ethnicity is being imputed
this needs to be checked. The panel also noted:
- That from longitudinal research produced, an individual’s Māori descent appears to be
more stable than ethnicity
- Advice to preserve the relativity of Māori descent and ethnicity
- This wil need good documentation about what was done and why.
AP5-4: Tahu and Donna to link previous research and papers they already have and provide any
further feedback to Stats NZ.
Census engagement
Kathy presented on the overal ‘fit for purpose’ approach, engagement progress and next steps.
The panel noted:
- Combining 2018 Census and Next Census engagement may create issues as the focus
needs to be on 2018.
- Talking to the Iwi Data Leaders Forum on 22 Feb is a good opportunity to engage but we
wil need to engage with many others who are not there very quickly
- The proposed timetable to engage with a wide range of Māori and iwi stakeholders wil be
hard to resource and complete in time – Stats NZ wil be scrambling with the timetable
- Iwi and Māori are investing time and money into getting ready to use 2018 Census data so
the earlier Stats let them know what might be published (or not published) the better
- Current Census webpages have information about census performance that is out of date
and may now be misleading”.
- Building internal Māori capability is important.
- Stats NZ should consider a risk assessment for Māori and iwi use.
Gareth presented an update on the fit for purpose engagement findings and resources. The panel
had some questions about the data use spreadsheet that had recently been uploaded to
sharepoint. There was interest in the sensitivity analysis work started. The possible addition of
approaching Ngai Tahu to complete some sensitivity work was not supported given equity of
access and political considerations.
Programme performance indicator progress
Page 3 of 11
The panel ran out of time to hear this presentation. A request was made to see the original 2018
Census business case and the cost-benefit analysis commissioned in 2013.
AP5-5: Load the 2018 Census business case and the cost-benefit analysis commissioned in 2013 to
the shared space.
This item wil be rol ed over to a later meeting.
Meeting summary discussion
- Panelists are struggling with no or little reading time to prepare for meetings. Stats NZ is
only getting a ‘reaction’ rather than considered advice.
- Clarification of the panels role is required as the panel is now starting to write the papers
need to be clear.
- The Terms of Reference should be updated to reflect extension of panel engagement, co-
chair arrangement and any concerns the chair has around the role of the panel
- Panel aiming to have initial report to Liz by end of April/beginning of May.
- The critical path explained at the combined meeting of the panel and external reviewers
on 13 Feb wil be uploaded to the shared space.
AP5-6: Stats initial y to revise the terms of reference for pragmatic things like date extensions,
role of other governance etc but then to panel for a deeper review by the panel. Liz and Richard
wil need to sign-off additional revisions.
AP5-7: Stats NZ to include a ‘what is a response?’ paper in the list of method papers to assist the
panel report writing
The next meetings dates are:
- in-committee only 27 February, Auckland
- 6 March, Auckland
- 12 April, in Auckland
- May and later dates to be confirmed.
Action log
Ref
Date
Description
Owner
Date
Progress
Status
raised
required
Meeting 2 actions
AP
23/10/18 Revise minutes from the
Richard and By 9
9/11 To be completed by
Close
2-0
first meeting and present
Gareth
Nov
23 Nov.
to for confirmation at the
18/2 v1.1 uploaded to
meeting on 9 November.
sharepoint
AP
23/10/18 Create and distribute a
Gareth
By 9
9/11 To be completed and
Close
2-2
conflict of interest register
Nov
added to Sharepoint before
Dec meeting.
6/12 Added to
Sharepoint/Shared
Documents – 7/12 Leave
action open for panel
members to fil in prior to
14 Feb 2019 meeting
18/2 Close but note that
the most panel members
Page 4 of 11
have not added anything to
the conflict document
AP
23/10/18 More information
Panel
By Dec
9/11 Information presented Open
2-4
requested about what
meeting that include 2013 response
investigations we are doing
rates and counts. Gareth to
into 2013 counts and
check with Richard whether
response rates – are we
this should be closed by 23
assuming that 2013 is
Nov.
correct?
6/12 Gareth to discuss with
Richard
18/2 Stats may need a
specific 2013 methods
paper to document any
inconsistencies in approach
and wil be captured in
methodology papers and
metadata documentation.
Gareth to upload proposed
method papers (completed
18 Feb). Panel to review
paper and offer comments
by 6 March.
AP
23/10/18 Need more detail about
Gareth
By 9
Sessions at 30 Oct meeting
Open
2-6
the totality and
Nov
and priority at ful 9 Nov
components of the
meeting.
imputation methods
9/11 Information presented
initial y planned and now
at meeting. Gareth to check
being implemented.
with Richard whether this
should be closed by 23 Nov.
6/12 Gareth to discuss with
Richard
18/2 Spreadsheet detailing
variables and imputation
processes to be sent to
panel prior to 6 March
meeting.
AP
23/10/18 Request for data on
Christine
By Dec
Topic for December
Close
2-10
birthplace.
meeting meeting
9/11 To be presented at
Dec meeting.
7/12 Data added to
workspace by 7 Dec
18/2 Richard to discuss
further with Christine
6/3 Wanting pop insights
set of data showing net
gains and losses by
birthplace over intercensual
period Mar 2013 to Dec
2017. Not on workplan, but
planning further work on
using admin data more
effectibvely. Country of
birth on list.
AP
23/10/18 More information needed
Christine
By 9
9/11 Presentation on
Close
2-13
about IDI household
Nov
mitigation including
matching including
imputations completed at
whether there wil be any
Nov meeting. Gareth to talk
more specific targeting of
with Richard about whether
areas with the threshold
this should stay open.
cut-offs (have we
Page 5 of 11
considered using different
6/12 Gareth to discuss with
cut-off rules for different
Richard
parts of the population?).
18/2 Close, – covered by
AP2-6
AP
23/10/18 Re electoral implications,
Kathy and
By 9
Discussion planned for 9
Open
2-16
more information is
Gareth
Nov
Nov meeting.
required about the
9/11 No additional electoral
representativeness of the
data was presented (as not
data and impacts of
complete yet). Updated
imputations. A range of
draft electoral calculations
scenarios, including
to be presented to panel in
constitutional impacts for
Dec meeting.
Māori, would be useful
6/12 Electoral data has not
yet been run – targeting
Feb meeting update.
18/2 Updated electoral
data to be presented to 6
March meeting dot loves
data note
Meeting 3 actions
AP
9/11/18
Correct minutes from 23
Gareth
By 23
6/12 added, new version to Close
3-4
Oct meeting adding a note
Nov
be uploaded after Dec
to the sentence ‘that we
meeting.
could test the model
18/2 Completed, , updated
accuracy’ in addition to
v0.9 to be added to
col ecting more data from
sharepoint, close
respondents.
AP
9/11/18
More information about
Gareth
By 7 Dec 6/12 Did not have time for
Open
3-5
what admin data is used in
this item in the Dec agenda.
the method and for what
Wil present information in
purpose. Privacy Impact
Feb.
Assessment update. Add to
14/2 Requested list for al
agenda for December.
variables about what
commitments we have
made for use of admin data
in the past and now. Rol ed
into AP2-6.
AP
9/11/18
Check the table stating the
Christine
By 7 Dec To be completed
Close
3-6
percentage of the IDI_ERP
14/2 Christine checked as it
using different admin
did not look right. Data is
sources as data does not
correct but need it
look right.
documented in time for
first report. Close.
AP
9/11/18
More information needed
Christine
23 Nov
6/12 included in Dec
Close
3-7
about the sources and
presentation
component of admin data
14/2 To be rol ed up into
used to help fil gaps in
AP2-6. Presented in
Māori descent data.
December and also close as
duplicate
AP
9/11/18
More information needed
Gareth
By 21
6/12 Info not yet available,
Close
3-9
about the sub-types of
Dec
can target Feb or Mar
Non-Private Dwel ings
meeting
(NPDs). Check when this
14/2 Presented at Feb
detail wil be available for
meeting. Close
the panel and add to
sharepoint folder when
ready.
AP
9/11/18
Clarification is needed
Panel
By 7 Dec 6/12 Leave open, not
Open
3-10
about what data is used in
members
addressed yet
the census administration
14/2 PIA includes detail,
enumeration (referred to
but hard to fol ow. Suggest
Page 6 of 11
as IDI data in the
adding a flowchart to aid
presentation), and a list of
communication. Panel to
the current variables being
review the PIA and give
imputed, where it comes
feedback.
from, whether the data
6/3 Any comments to get
suppliers are aware of its
back to stats ASAP
use and whether it uses
identifiable or de-
identified person data.
AP
9/11/18
Present criteria about the
Gareth and
By 7 Dec 6/12 Some information in
Open
3-11
thresholds being used to
Christine
Dec meeting present, more
make decisions about
info to come after meeting.
methods.
Leave open.
14/2 Panel wil be using
Canadian quality
framework. Request more
information about quality
plans at March meeting.
6/3 Some information at 6
March meeting. Confirm
whethe
AP3-
Panel members to think
Kathy
By 7 Dec 6/12 Check with panel
Close
12
further about going back
about whether there was
into the field and whether
any more feedback.
they had any further
14/12 Panel unable to
advice for us (as they
comment on this yet with
didn’t think they had time
only a brief discussion in
to appropriately digest and
November. The panel
respond at the meeting).
needs more information
from Stats NZ on what sorts
of options exist for
col ecting further data from
the field. Kathy to report
back to February or March
2019 meeting.
14/2 Stats is not going back
into the field, this needs to
be documented internal y.
Close
AP
30/10/18 The Dwel ing Frame
Adele
14 Feb
14/2 To be uploaded to
Close
3-14
evaluation report to be
2019
shared space by 6 March.
shared with the q/a panel
6/3 Uploaded to workspace
AP
30/10/18 Provide information on
Adele/
14 Feb
14/2 Some information was Close
3-15
births and removals from
Chris H
2019
presented to the panel in
the Dwel ing frame during
November. More to be
the live operation
loaded to workspace before
March meeting
6/3 Uploaded to workspace
AP
30/10/18 Produce an estimate of
Adele
14 Feb
14/2 This was investigated
Close
3-16
population at smal
2019
and did not improve the
geographies by applying
situation. Only changed
the average household size
count by very marginal
from 2013 census to the
amount. Close
number of dwel ings on
the dwel ing frame.
AP
30/10/18 Produce a matrix showing
Adele
14 Feb
14/2 Provided. Close
Close
3-17
the counts of people listed
2019
on the dwel ing
questionnaire against the
number of individual
questionnaires
Page 7 of 11
AP
30/10/18 Confirm the source of local Gareth
14 Feb
14/2 Was investigated but
Close
3-18
estimates of those born
2019
didn’t provide specific help.
overseas (e.g. in Auckland)
Close
AP
30/10/18 Provide the panel with
Gareth
14 Feb
13/12 Gareth: Suggest this
Close
3-19
information on staff
2019
is out of scope but in scope
shortages at local levels of
of the external review and
geography – ideal y in a
to close action
way that identifies the
14/2 Out of scope to
impact on workloads
external data quality panel.
Close
AP
30/10/18 Report back on how might
Gareth
14 Feb
13/12 Gareth: Suggest this
Close
3-20
the problems with the
2019
is out of scope but in scope
scanning field staff
of the external review and
recruitment vendors
to close action
impact on response rates?
14/2 Out of scope to
external data quality panel.
Close
AP
30/10/18 Share paper with the panel Gareth
14 Feb
14/2 Duplicated action
Close
3-21
on the imputation
2019
point. Close this but keep
approach and the CPT
AP2-6 open. Close
recommendations on
which admin variables to
use for imputation. Ideal y
also to include a flow chart
of the process.
AP
30/10/18 Elaboration of the options
Christine
14 Feb
14/2 Update given at 14
Close
3-22
around imputation – with
2019
Feb meeting. Close
the pros and cons of each
option and the impacts on
users spelt out (maybe in a
table with a
Red/Amber/Green status?)
Meeting 4 actions
AP
7/12/18
Ensure all panel members
Gareth
By 21
13/12 Panel members now
Close
4-1
have shared workspace
Dec
given write access.
write-access and items are
Documents to be renamed
label ed appropriately.
by 21 Dec
14/2 Completed, close
AP
7/12/18
Secretariat to ensure al
Gareth
By 21
13/12 Noted for future
Close
4-2
documents have useful
Dec
attachments. Al current
headers and footers so
docs to be updated by
once open it is clear what
Xmas
panel members are
14/2 Completed, close
reviewing.
AP
7/12/18
Create and load document
Adele
By 14
18/12 Two documents
Close
4-3
with Internet Col ection
Feb
(AP4-3a and AP4-3b)
System screen shots for
meeting uploaded to workspace
panel members.
with screenshots. Wil leave
open until Feb meeting for
panel let know whether
there is enough
information.
14/2 No more information
available for panel. Close
AP
7/12/18
Document how multiple
Christine
By 14
14/2 Assign to methods
Open
4-5
de-identified data points in
Feb
team. Confirm priority with
admin sources have been
meeting Richard.
protected from
6/3 Panel to look at
identification once they
methods paper
are assembled.
Page 8 of 11
AP
7/12/18
Document ‘What is the
Carol
By 14
14/2 Although Liz discussed Open
4-6
legal basis that it is OK to
Feb
the issues with the panel at
do what we are doing with
meeting meeting on 5 Feb, panel
admin data and what are
stil wants a statement
the ethical implications we
from Stats about the legal
have considered?’ ‘What
basis of admin sources.
commitment has Stats NZ
Without, it is a risk for the
given to respondents and
panel to endorse. High
data suppliers about what
ethical standard set by the
we are doing with admin
Stats NZ at the 2017 data
data? What have they
summit. Done
said? What are the key
data sources?’
AP
7/12/18
Share draft list of technical Gareth
By 14
14/2 Draft paper describing Close
4-7
reports with the panel to
Feb
planned papers added to
see if this list wil also help
meeting sharepoint on 18 Feb.
the panel produce the
Duplicate from previous
independent report. The
meeting. Close
panel has also created a
list of documents that
would be helpful, so these
can be compared.
AP
7/12/18
Consult over when Stats
Richard and By 14
14/2 Stats to cover in
Open
4-8
NZ wil return to the panel
Gareth
Feb
meeting on 14 Feb. Check
with a decision about
meeting with Richard if enough
public release of iwi data
information to close.
6/3 iwi data leaders key
messages sent to panel.
Decision made Stats wil
not talk publical y about
data quality until April
announcement. Need to
have conversation with iwi
before
Leave open nuntil decision
about April iwi
announcement is
confirmed. Panel is writing
up document to be sent to
stats about panel view
about quality of iwi data.
Keep open
AP
7/12/18
Update panel once more
Gareth
By 14
14/2 Planned item on 6
Open
4-9
detail has been completed
Feb
March agenda.
on the quality framework
meeting 6/3 To be reviewed after 6
including an ordering of
Mar presentation
decision making criteria.
AP
7/12/18
Upload the ful customer
Gareth
By 21
14/2 Completed on 12/2.
Close
4-10
engagement spreadsheets
Dec
Close
to the workspace.
AP
7/12/18
Discuss with Len after the
Kathy and
By 21
14/2 Close duplicate of (stil Close
4-11
Dec meeting whether the
Len
Dec
open) AP 4-6
document ‘Information
census legislative
obligations – DRAFT 5 Dec
2018’ loaded into the
workspace goes into
sufficient detail about
Stats NZ legal obligations.
Page 9 of 11
AP
7/12/18
Create a table to
Adele
By 14
14/2 Duplicate of AP2-6.
Close
4-12
summarise variable data
Feb
Close
sources, concepts and
meeting
hierarchy.
AP
7/12/18
Create a table to
Gareth
By 14
14/2 Did not have time to
Close
4-13
summarise variable quality
Feb
present, planned for 6
measures.
meeting March meeting.
2/6 Added to workspace
AP
7/12/18
Facilitate set-up of an in-
Gareth and
By 21
13/12 Room in Wel ington
Close
4-14
committee meeting
Sophie
Dec
Stats office and travel
(without Stats people in
booked for Auckland
attendance) in late Jan or
people coming. Food and
Feb 2019.
room access remaining
actions.
14/2 5 Feb set up. Close
AP
7/12/18
Check panel members
Gareth
By 21
14/2 Meeting is booked to
Close
4-17
availability for April
Dec
12 April in Auckland, but
meeting.
wil check with panel
members via email on 22/2.
Meeting 5 actions
AP
14/2/19
Need to clarify admin
Adele,
By 6
6/3 Writing a paper for
Open
5-1
enumeration, admin
Christine
March
panel aboiut admin vs
imputation and other
survey data. Kathy to send
terminology. Thomas
to SNZ for review.
created a short paper of
definitions to be reviewed
by Stats NZ.
AP
14/2/19
Provide list of proposed
Gareth
By 6
14/2 Duplicated with AP2-4 Close
5-2
external method
March
documents to support
communication including
‘What is a response?’
AP
14/2/19
Load technical papers
Gareth
By 6
6/3 several papers added,
Close
5-3
presented to internal
March
more to come. Any specific
Technical Advisory Group
requests welcome
to the shared space
including approach to
prison and defence admin
enumeration.
AP
14/2/19
Additional ways for
Gareth
By 6
Regular emails each week
Close
5-4
information to get to the
March
panel during the rapid
changes
AP
14/2/19
Find and load 2018 Census
Gareth
By 6
6/3 Added to workspace
Close
5-5
business case document
March
and valuing the census to
the shared space
AP
14/2/19
Revise terms of reference
Gareth and
By 6
3/3 Sent to Richard for
Open
5-6
(mainly around the
panel
March
initial review
mechanics of the panel
purpose, then for the
panel to review and sign-
off from Liz and Richard
AP
14/2/19
Panel members to add
Tahu and
By 6
6/3 Link added to shared
Close
5-7
links to the previous
Donna
March
documents
research and papers on
Māori descent and
ethnicity they already have
and provide any further
feedback to Stats NZ.
Page 10 of 11
AP5-
14/2/19
Include s ‘what is a
Christine
By 10
Open
8
response’ paper in the list
April
of method papers that
would assist the panel to
write the reports
Page 11 of 11