2018 Census external data quality panel 9 November minutes.pdf

2018 Census External Data Quality Panel: Minutes of Meeting
on 9 November 2018
Date and time
9 November 2018, 9am to 3:30pm
Location
Stats NZ, Auckland Office
48 Greys Avenue,
Auckland Central
Present -
Richard Bedford - Chair
panel members  Tahu Kukutai (via audio conference from about 11am)
Donna Cormack
Alison Reid
Thomas Lumley
Len Cook
Barry Milne
Ian Cope (in person)
Present – Stats
Auckland
NZ
Kathy Connol y, General Manager Census
Vince Galvin, Chief Methodologist
Gareth Meech, Senior Manager Census Customer Focus and Secretariat
Steph Prosser, Senior Analyst Census
Adele Quinn, Manager Census Analytics
Christine Bycroft, Principal Statistician
Christchurch via v/c for selected sessions
Chris Hodgins, Manager Census Geography project
Alex Bayley, Senior Manager Census Respondent Focus

Meeting minutes
9-10am – In-committee meeting and feedback
The Chair reported back that the level of detail in the draft minutes from the 2nd meeting is better
than the first meeting and is a good model for the subsequent minutes. One change was
suggested by a panel member in regards to the sentence around going back into the field and
once draft changes were made, the Chair accepted the minutes as approved. The action points list
was revisited before the body of the meeting started and this has been updated at the end of
these minutes. The Chair agreed to work with the secretariat to confirm which actions could be
closed in the week of 12 November.

The Chair reported that the panel had discussed the legal obligations of Stats NZ to the col ection
and publication of accurate census data. The panel wants assurance from Stats NZ that
procedures/methodologies to improve the quality of census data are legal in the context of
legislation including Stats Act, Electoral Act and Privacy Act. The panel needs to know they are
giving advice that does not breach the requirements of any act or any commitments that have
already been made. The panel strongly recommended that suitable advice is sought by Stats NZ
that informs the panel.

The panel also discussed a draft proposal for the content of their report that had been circulated
to panel members prior to the meeting. The Chair wil  send a copy of a revised version of this
proposal to the secretariat and Kathy and this wil  become the working draft that wil  help
determine the priority of information and data the panel need to complete the report.
Page 1 of 10

The Chair noted that the panel wil  soon need to have additional meetings to start forming and
writing their independent report. The Chair wil  suggest additional dates and times prior to the
December meeting.

The secretariat advised that a solution for al owing panel members access to presentations,
papers and data prior to meetings has been approved and wil  hopeful y be in place by 16
November. As previously advised, al  panel members wil  need RealMe usernames to access the
solution.

AP3-1: Each panel member to ensure they have a functioning RealMe username by 23 November.

A set of guidelines to access the system and set expectations on how workspace members should
work to reduce risk of the data being shared wil  be developed.

AP3-2: The secretariat to develop and distribute guidelines when the file sharing solution is in
place.

The Chair requested that panel members and Stats NZ people hold both 13 and 14 February and 5
and 6 March for the dates of the February and March meetings respectively. The Chair is working
with the 2018 Census external reviewers to set up a date to share findings in February and some
panel members need more time to confirm the March dates.

AP3-3: The secretariat and Chair to agree and distribute future meeting dates by 23 November.

Progress on actions from the meeting on 23 October was summarised verbal y and details are
documented in the new action log at the end of these minutes.

AP3-4: Correct minutes from 23 Oct meeting adding a note to the sentence ‘that we could test the
model accuracy’ in addition to col ecting more data from respondents.

10:15am to 3pm – Imputation approach and updated data, Christine and Adele

Stats NZ presented many but not al  of the slides prepared for the meeting because of time
constraints. The slides and discussion covered:
- a summary of the problems we are facing
- constraints
- presentation of national and regional level data on Māori ethnicity, Māori descent and iwi counts
- dwel ing frame quality metrics
- mitigations for non-response adjustment – specifical y use of administrative data sources and
imputation methods
- programme timelines and
- options to going back into the field.

We had also planned to discuss the quality assurance framework, recent customer engagement
and update the panel on the progress and issues faced by the Post Enumeration Survey (but we
ran out of time).

Page 2 of 10

In the discussion that accompanied the Stats NZ presentation, Panel members made a number of
requests for further information to be provided at subsequent meetings. These requests are
itemised below as specific action points.

AP3-5: More information about what which administrative data sources are to be used and
whether their use in this context is what respondents and data suppliers would expect - or have
been advised. The term IDI is used but this is de-identified data and we are linking using identified
data. Is this described in the Privacy Impact Assessment and what do our data providers (both
source and intermediaries understand about this particular use). Gareth to add this item to
agenda for December meeting.

AP3-6: Check the table stating the percentage of the IDI_ERP using different admin sources. Data
does not look right. Christine to check and get back to panel in week of 23 November.

Note that there are (at least) two different stories about Māori ethnic group response rates in
Northland compared to Auckland and that Stats wil  need good analysis to tel  the data story.

AP3-7: More information needed about the sources and component of admin data used to help
fil  gaps in Māori descent data. Christine to check and get back to panel in week of 23 November.

Note that concerns were raised about the lack of other admin sources of iwi data and in particular
new iwi added to the classification since the 2013 Census. There was a brief discussion about the
impact on users of this data in particular for Treaty settlement processes. Comprehensive
engagement wil  be needed with iwi.

AP3-8: For future data presentations, request that any data created to Territorial Authority level,
that we include Auckland Local Boards. Gareth to note for future meetings.

Chris Hodgins joined the conference and presented slides on the dwel ing frame quality. There
was a lot of information displayed, so panel members wil  need time to review it once it can be
uploaded into sharepoint.

AP3-9: More information needed about the sub-types of Non-Private Dwel ings (NPDs). Gareth to
check when this detail wil  be available for the panel and add to sharepoint folder when ready. By
end of Dec.

A session on data mitigations was presented by Christine. There was recap of the strategies
planned and how Stats is progressing towards implementation.

Panel led a set of questions about the reference to the IDI in our methods and made the point
that the IDI is a de-identified research dataset – whereas we are using a set of data from the IDI
that initial y uses identifying information to link individuals (before being de-identified for census
dataset use).

AP3-10: Clarification is needed about what data is used in the census administration enumeration
(referred to as IDI data in the presentation) and a list of the current variables being imputed,
where the data comes from, whether the data suppliers are aware of its use and whether it uses
identifiable or de-identified person data. Christine and Kathy to provide information to panel in
time for the Dec meeting.

Page 3 of 10

Kathy then led a discussion about the timelines the programme is working towards. Stats wil  be
updating customers by mid-December about the first release dates. The panel made several
suggestions including whether and when an ERP could be produced that was ‘informed by the
census’.

Engagement with iwi and Māori was discussed This census could/should be the catalyst for a
forward looking discussion on how to produce robust iwi and Māori data across the Official
Statistics System (OSS). Some of the panel members suggested that Stats should engage strategic
public relations support for this period. Other members emphasized the need for Stats NZ to be as
open, transparent and proactive as possible in their communications.

AP3-11: The present criteria about the thresholds being used to make decisions about methods
was requested. Gareth to work with Christine to update quality assurance framework
presentation that was not able to presented due to a lack of time today.

Alex Bayley joined the meeting via video conference to discuss options that have been drafted
about going back into the field. The panel asked questions and talked about possible pros and
cons to each idea. Options to improve Māori and iwi data was discussed including the possible use
of iwi held registers.

AP3-12: Panel members to think further about going back into the field and whether they had any
further advice for us (as they didn’t think they had time to appropriately digest and respond at the
meeting). Richard to summarise before December meeting.

There was not time for the presentations on the quality assurance framework, the recent key
customer engagement and PES update – due to the need to prioritise the data and methods
discussions.

3pm to 3:30 Summary and December meeting priorities
The Chair opened a discussion on what areas would be priorities for the December meeting.
These included:
-  Presentation of the messages to be used in the update to the public in early December
-  Communication messages that the panel could use
-  Quality assurance framework and customer feedback
-  PES update
-  Updated data shown to date eg (people and dwel ing counts, ethnicity, Māori descent
and)
-  A realistic programme plan and timeline
-  Present data on the birthplace data (and include 2013 and 2018 comparisons and IDI-ERP
birthplace)
-  Present data on gender (sex) and the intersex paper option that was available
AP3-13: Chair to agree priority of topics for Dec meeting by 23 November.

The next meetings dates are:
- 7 December, Auckland
- 13 and/or 14 February, Wel ington
- 5 and/or 6 March, Auckland
- April date to be confirmed by 23 Nov.
Page 4 of 10

- May and later dates to be confirmed at 7 Dec meeting.

Action log
Ref
Date
Description
Owner
Date
Progress
Date
raised
required
closed
AP 1-
29/8/18
TOR: Amend section of report
Gareth
Before
23/10 TOR was finalised,
Closed
1
where it states that the GS wil
and
Oct
distributed, reviewed and
14/9/18
sign off on the panel report.
Richard B  meeting  published 14 Sept. Can be
  Include reference that advice
closed.
on admin data is wanted -
make this more explicit.
  - Add line to TOR around
statutory requirements.
  - Add line to TOR around
impacts of census data on iwi
and Māori.
AP 1-
29/8/18   Send independent ABS report
Gareth
Before
23/10 ABS report not yet sent
Close
2
to members.
Oct
given the information is
meeting  sensitive. Gareth to work with
Kathy and Richard on how to
give access or present at future
meeting. Update at Nov meeting
9/11. Paper wil  be added to
sharepoint folder once up and
running. To close before Dec
meeting.
6/12 Added to
workspace/Shared Documents
AP 1-
29/8/18
Set up some time to talk in
Gareth
Before
23/10 Links to electoral
Closed
3
next meeting to talk about
Oct
calculation added to Oct
23/10/18
electoral legislation and how
meeting  meeting agenda. Electoral data
it's calculated - and prepare
update delivered in Oct
something to send out.
meeting.
AP 1-
29/8/18
Create and send out template
Gareth
Before
23/10 No existing template, but
Closed
4
for conflicts of interest.
Oct
discussion at Oct meeting has
23/10/18
meeting  led to AP 2-2 – to create a
register. Wil  close this one.
AP 1-
29/8/18
Give information on how
Gareth
Before
23/10 Adele has prepared some
Close
5
respondents were to meet
or at Oct  information, to be discussed at
their legal obligations.
meeting  the informal meeting on 30 Oct.
9/11 Short paper printed and
added to pack at meeting.
Additional paper SNZ presented
to conference to be added to
sharepoint.
6/12 Added to
workspace/Shared Documents
AP 1-
29/8/18
Complete declaration of
Gareth
At Oct
23/10 Ian and Tahu signed
Closed
6
secrecy docs for Tahu and Ian
meeting  declarations at the Oct meeting.  23/10/18

Close
AP 1-
29/8/18
Confirm dates of next
Richard
At Oct
23/10 Nov, Dec, Feb and Mar
Open
7
meetings.
meeting  dates confirmed. April date to
be confirmed. Leave open
9/11 Feb and March dates wil
likely include additional
meetings to work on panel
report and meet with external
review team.
Page 5 of 10

AP 1-
29/8/18
Col ate and send out
Gareth
Before
23/10 Summary not provided at
Open Close
8
information about languages
Oct
Oct meeting. Wil  target Nov
(what was considered &
meeting  meeting.
priorities).
9/11 Some information was
provided with the October
agenda. To be added to
Sharepoint folder before Dec
meeting. Additional information
wanted on how the languages
were selected.
6/12 Added to shared folder
AP 1-
29/8/18   Find out the response rate of
Gareth
At Oct
23/10 Ran out of time for PES
Open
9
the PES
meeting  update. To be prioritised in next
/Close
2 meetings.
9/11 Ran out of time at 9 Nov
meeting, but wil  be added to
later meeting date.
6/12 PES added to agenda for
Dec meeting
AP 1-
29/8/18
Find out the methodology of
Richard S  Before
23/10 Summary of information
Closed
10
the independent agency that
Oct
attached to October meeting
23/10/18
measured the 'awareness' KPI.
meeting  agenda.
AP 1-
29/8/18
Prepare information for critical  Gareth
Before
23/10 Not specifical y discussed
Closed
11
triggers.
Nov
at meeting. Gareth to discuss
9/11
meeting  whether this fits with the panel
or better with the external
review and advise panel.
9/11 Richard wil  work with the
external reviewers on what the
panel wil  or not need to
produce their findings. Close
AP 1-
29/8/18
Provide information of bias.
Gareth
At Oct
23/10 presentation of method
Closed
12
meeting  and the pro and cons of
9/11
approaches only touched on at
Oct meeting. This wil  be subject
to Nov meeting content. To
close after that meeting
9/11 Some information about
bias was discussed as part of the
mitigations sessions. Close
AP 1-
29/8/18
Provide rough indication of
Gareth
At Oct
23/10 Māori descent and
Closed
13
Māori response rates for next
meeting  ethnicity counts presented at
23/10/18
meeting.
October meeting. Wil  be
subject to future discussions,
but wil  close this action point.
Meeting 2 actions
AP 2-
23/10/18  Revise minutes from the first
Richard
By 9
9/11 To be completed by 23
Open
0
meeting and present to for
and
Nov
Nov.
confirmation at the meeting on  Gareth
9 November.
AP 2-
23/10/18  Work out way to give access to  Kathy
By 9
9/11 DIA Sharepoint solution
Open Close
1
sensitive information outside
and
Nov
approved, but not yet
of the meeting room
Gareth
implemented. Target date 16
November.
6/12 Sharepoint solution
created and access given to
panel members
AP 2-
23/10/18  Create and distribute a conflict
Gareth
By 9
9/11 To be completed and
Open
2
of interest register
Nov
added to Sharepoint before Dec
meeting.
Page 6 of 10

6/12 Added to
Sharepoint/Shared Documents –
7/12 Leave action open for
panel members to fil  in prior to
Feb 2019 meeting
AP 2-
23/10/18  More information requested
Gareth
At 9 Nov  Topic for 9 Nov meeting.
Closed
3
about the components and
meeting  9/11 Presented on at Nov
9/11/18
quality of the dwel ing frame
meeting. Close.
including mail-out and list
leave approaches, what was
added and deleted during
operations.
AP 2-
23/10/18  More information requested
Gareth
By Dec
9/11 Information presented that  Open
4
about what investigations we
meeting  include 2013 response rates and
are doing into 2013 counts and
counts. Gareth to check with
response rates – are we
Richard whether this should be
assuming that 2013 is correct?
closed by 23 Nov.
6/12 Gareth to talk to Richard
AP 2-
23/10/18  Send apartment strategies (and  Gareth
By 9
9/11 To be completed and
Open Close
5
other targeted strategies) as
and
Nov
added to Sharepoint before Dec
background to next meeting.
Steph
meeting.
6/12 2018 Census Targeted area
slides added to workspace
AP 2-
23/10/18  Need more detail about the
Gareth
By 9
Sessions at 30 Oct meeting and
Open
6
totality and components of the  and
Nov
priority at ful  9 Nov meeting.
imputation methods initial y
Richard
9/11 Information presented at
planned and now being
meeting. Gareth to check with
implemented.
Richard whether this should be
closed by 23 Nov.
6/12 Gareth to talk to Richard
AP 2-
23/10/18  Next time data is presented,
Gareth
For Nov
25/10 – Wil  note for future
Closed
7
include like for like
and
meeting  meetings
25/10
comparisons between ERPs
Adele
and Census usual y resident
counts – we have used 2013
census usual residence counts
and 2018 ERP as benchmarks in
same tables.
AP 2-
23/10/18  Next time maps are produced,
Gareth
By 9
25/10 – Wil  note for future
Closed
8
try to use the same colours and
Nov
meetings
25/10
more detailed response rate
breakdowns. Also ensure
furture TA breakdowns include
Auckland Local Boards in
future.
AP 2-
23/10/18  Ensure labels on al  graphs are
Gareth
By 9
25/10 – Wil  note for future
Closed
9
clear and any footnotes noted
Nov
meetings
25/10
with graphs. To note for next
meetings.
AP 2-
23/10/18  Request for data on birthplace.  Gareth
By Dec
Topic for December meeting
Open
10
meeting  9/11 To be presented at Dec
meeting.
7/12 Data added to workspace
by 7 Dec
AP 2-
23/10/18  Discussion needed about the
Gareth
By 9
Discussion planned for 30 Oct
Closed
11
decisions to impute a larger
Nov
meeting and then at 9 Nov
9/11/18
range of variables and impact
meeting
on not stated. Include ethnicity
9/11 (Not final) Māori descent,
and Māori descent.
ethnicity and iwi data presented
at meeting. Close
Page 7 of 10

AP 2-
23/10/18  Send summary table and links
Gareth
For 9
9/11 Email sent in week before
Closed
12
to Census Transformation
Nov pre
Nov meeting. Close
9/11/18
papers (in SNZ archive site) as
reading
background to next meeting.
AP 2-
23/10/18  More information needed
Christine
By 9
9/11 Presentation on mitigation
Open
13
about IDI household matching
Nov
including imputations
including whether there wil  be
completed at Nov meeting.
any more specific targeting of
Gareth to talk with Richard
areas with the threshold cut-
about whether this should stay
offs (have we considered using
open.
different cut-off rules for
6/12 Gareth to talk to Richard
different parts of the
after dec meeting
population?).
AP 2-
23/10/18  Stats NZ needs to consider the
Kathy
By 9
9/11 Discussion at Nov meeting
Closed
14
public perception of the
Nov
has led to additional need for
11/9/18
extended use of admin data
information and new action
and use of name and address
point 3-10. Close.
to match to admin sources of
personal information. Kathy to
action and respond back
before 9 Nov.
AP 2-
23/10/18  Require more information
Christine
By 9
Pre reading to be sent and
Closed
15
about the qualities of the
Nov
Christine to discuss at 9 Nov
9/11/18
ethnicity data in the IDI-ERP.
meeting.
9/11 Email with links sent prior
to Nov meeting and additional
info presented on 9/11. Close.
AP 2-
23/10/18  Re electoral implications, more  Kathy
By 9
Discussion planned for 9 Nov
Open
16
information is required about
and
Nov
meeting.
the representativeness of the
Gareth
9/11 No additional electoral
data and impacts of
data was presented (as not
imputations. A range of
complete yet). Updated draft
scenarios, including
electoral calculations to be
constitutional impacts for
presented to panel in Dec
Māori, would be useful
meeting.
6/12 Electoral data has not yet
been run – targeting Feb
meeting update.
AP 2-
23/10/18  Request for more information
Gareth
By 9
Discussion planned for 9 Nov
Closed
17
about Plan C in advance of next
Nov
meeting.
9/11/18
meeting. Gareth to assign SNZ
9/11 Presented to Nov meeting.
owner and agree priority with
Close.
Richard.
AP 2-
23/10/18  Need more information about
Kathy
By Dec
Topic for December meeting.
Open Close
18
the Post Enumeration Survey
and
meeting  9/11 PES to be included in Dec
methods and outcomes under
Brooke
meeting agenda.
the different imputation
6/12 added to agenda
scenarios.
AP 2-
23/10/18  Only some panel members are
Gareth
By 26
25/10 Travel being booked by
Closed
19
available to go to Chch on the
Oct
Brenda
26 Oct
30th. Confirm Ian, Len, Thomas
and Barry wil  be going. Gareth
to arrange travel with the four.
Meeting 3 actions
AP 3-
9/11/18
Each panel member to ensure
Panel
By 23
6/12 Panel members to update
Open Close
1
they have a functioning
members  Nov
RealMe username.
AP 3-
9/11/18
Develop and distribute
Gareth
By 23
6/12 Brief guidelines provided
Open Close
2
guidelines when file sharing
Nov
via email
solution is in place
Page 8 of 10

AP 3-
9/11/18
Agree and distribute future
Richard
By 23
6/12 Check with Richard
Open
3
meeting dates by 23 November  and
Nov
Gareth
AP 3-
9/11/18
Correct minutes from 23 Oct
Gareth
By 23
6/12 added, new version to be
Open
4
meeting adding a note to the
Nov
uploaded after Dec meeting.
sentence ‘that we could test
the model accuracy’ in addition
to col ecting more data from
respondents.
AP 3-
9/11/18
More information about what
Gareth
By 7 Dec  6/12 Did not have time for Dec
Open
5
admin data is used in the
agenda. Propose to present in
method and for what purpose.
Feb.
Privacy Impact Assessment
update. Add to agenda for
December.

AP 3-
9/11/18
Check the table stating the
Christine
By 7 Dec
Open
6
percentage of the IDI_ERP
using different admin sources
as data does not look right.
AP 3-
9/11/18
More information needed
Christine
23 Nov
6/12 included in Dec
Open
7
about the sources and
presentation
component of admin data used
to help fil  gaps in Māori
descent data.
AP 3-
9/11/18
For future data presentations,
Gareth
By 7 Dec  6/12 Noted
Open close
8
request that any data created
to Territorial Authority level,
that we include Auckland Local
Boards.
AP 3-
9/11/18
More information needed
Gareth
By 21
6/12 Info not yet available, can
Open
9
about the sub-types of Non-
Dec
target Feb or Mar meeting
Private Dwel ings (NPDs).
Check when this detail wil  be
available for the panel and add
to sharepoint folder when
ready.
AP 3-
9/11/18
Clarification is needed about
Christine
By 7 Dec  6/12 More info in Dec
Open
10
what data is used in the census  and
presentation – Gareth to check
administration enumeration
Kathy
with Richard after Dec meeting
(referred to as IDI data in the
to see whether can close
presentation), and a list of the
current variables being
imputed, where it comes from,
whether the data suppliers are
aware of its use and whether it
uses identifiable or de-
identified person data.
AP 3-
9/11/18
Present criteria about the
Gareth
By 7 Dec  6/12 Some information in Dec
Open
11
thresholds being used to make
and
meeting present, more info to
decisions about methods.
Christine
come after meeting. Leave
open.
AP3-

Panel members to think further  Richard
By 7 Dec  6/12 Check with panel about
Open
12
about going back into the field
whether there was any more
and whether they had any
feedback.
further advice for us (as they
didn’t think they had time to
appropriately digest and
respond at the meeting).
Page 9 of 10

AP 3-
9/11/18
Agree priority of topics for Dec
Richard
By 23
Close
Open/Close
13
meeting.
Nov

Page 10 of 10