2018 Census external data quality panel 23 October minutes.pdf

2018 Census External Data Quality Panel: Minutes of Meeting
on 23 October 2018
Date and time
23 October 2018, 9am to 3:15pm
Location
Stats NZ, Auckland Office
48 Greys Avenue,
Auckland Central
Present -
Richard Bedford - Chair
panel members  Tahu Kukutai
Donna Cormack
Alison Reid
Thomas Lumley
Len Cook
Barry Milne
Ian Cope (in person)
Present – Stats
Auckland
NZ
Kathy Connol y, General Manager Census
Vince Galvin, Chief Methodologist
Gareth Meech, Senior Manager Census Customer Focus and Secretariat
Steph Prosser, Senior Analyst Census
Adele Quinn, Manager Census Analytics
Christine Bycroft, Principal Statistician
Meeting minutes
9-10am – In-committee meeting and feedback
The Chair reported back on a discussion about the draft minutes from the first meeting. The Panel
felt they were too detailed and, although anonymised, comments could be attributed to
individuals. In revising the minutes the Panel requested that al  action points were numbered and
presented in the minutes where the discussion they relate to occurred. Dates for completion and
owners of the actions needed to be specified. A summary of actions, with appropriate cross-
referencing, needed to be provided at the end of the minutes. The Chair agreed to work with the
secretariat to revise these minutes which would be presented to the Panel for confirmation at the
meeting on 9 November. Action Point (AP) 2-0).

The Panel had agreed that minutes of the meetings should capture in a brief summary the
essence of the discussion, the issues raised and actions requested/approved. The Chair noted that
if Panel members wanted a specific item of discussion or point to be recorded in the minutes then
this needed to be noted at the time during the meeting.

On behalf of the Panel, the Chair requested that al  written or tabular material provided by Stats
NZ to the Panel includes the date the material was provided and is version-control ed.

The Chair also advised the meeting that there had been a conversation about written information
and powerpoint presentations that were supplied in meetings but which had to be returned at the
end of the meeting. The Panel is concerned that without papers to read before the meeting, and
not being able to take notes home with them, limited their ability to be effective. AP 2-1: Gareth
and Kathy to work out way to give access to sensitive information outside of the meeting room.

The Chair reported he been introduced to the two external reviewers who had recently been
appointed to undertake a comprehensive assessment of the operation of the 2018 Census. The
Panel Chair and reviewers wil  keep the other party updated on progress with their respective
tasks. It is likely that there wil  be a joint meeting of the Panel and the reviewers in February 2019.
The Chair noted that the reviewers had requested that if individual Panel members wish to
engage with them they should do this through the Panel Chair.

With regard to Panel members getting more operational information about what went wel  and
didn’t go wel  during execution of the census, an optional meeting in Christchurch on 30 October
with some of the operational experts had been arranged. Panel members Len Cook, Ian Cope,
Thomas Lumley and Barry Milne wil  attend this meeting. A further meeting with operations staff
wil  be arranged early in the new year.

Progress on actions from the meeting on 23 October was summarised verbal y and details are
documented in the new action log at the end of these minutes.

Changes to conflict of interest
No new conflicts were raised at the meeting, but it was requested that a formal register is created
for the Panel to access at each meeting.   AP 2-2: Gareth to create and distribute a conflict of
interest register before the meeting on 9 November.

10:30am to 3pm– Data presentation and electoral update by Kathy, Vince, Christine and Adele

Stats NZ presented many but not al  of the slides prepared for the meeting because of time
constraints. The slides covered overal  response rates (discussed briefly at the meeting on 23
October), more detail about the missing groups with particular reference to ethnicity and
geographic area, counts (with maps) by region, TA and SA2, summary counts by key demographic
variables, and a description of administrative data sources.

Imputation strategies were introduced briefly along with some implications of undercounting for
delivery of data required by legislation for the determination of electoral boundaries and the
numbers of seats for Māori registered on the Māori Electoral Rol .

In the discussion that accompanied the Stats NZ presentation, Panel members made a number of
requests for further information to be provided at subsequent meetings. These requests are
itemised below as specific action points.

AP2-3: More information about the components and quality of the dwel ing frame including mail-
out and list leave approaches, what was added and deleted during operations. Gareth to arrange
for this information to be provided for the meeting on 9 November.

AP2-4: More information about investigations into the reliability of 2013 counts and response
rates that are being used as reference points for the 2018 census data. Gareth to assign SNZ
owner and identify, with Richard, the meeting where this matter can be discussed.

AP2-5: Send details of strategies for addressing target groups, including apartments, as
background material for the next meeting. Gareth to col ate and send to Panel members prior to 9
Nov meeting.

AP2-6: More detail about the totality and components of the imputation methods initial y
planned and now being implemented was requested. Gareth to assign SNZ owner and identify,
with Richard, the meeting where this matter can be discussed.

AP2-7: Next time data is presented, include like for like comparisons between ERPs and Census
usual y resident counts – we have used 2013 census usual residence counts and 2018 ERP as
benchmarks in same tables. Gareth to note for subsequent meetings.

AP2-8: Next time maps are produced, try to use consistent colours and scales. Also would like to
see similar response rate data for Auckland Local Boards. Gareth to note for subsequent meetings.

AP2-9: Ensure labels on al  graphs are clear and any footnotes noted with graphs. Gareth to note
for subsequent meetings.

AP2-10: Request for data on birthplace of the population in 2018. Gareth to assign SNZ owner and
identify, with Richard, the meeting where this matter can be discussed.

AP2-11: Discussion needed about the decisions to impute data for a larger range of variables and
to remove the ‘not stated’ category in some key ones like ethnicity. Gareth to assign SNZ owner
and identify, with Richard, the meeting where this matter can be discussed.

AP2-12: Send summary table and links to Census Transformation papers (in SNZ archive site) as
background information for the next meeting. Gareth to fol ow up for meeting on 9 Nov.

AP2-13: More information needed about IDI household matching including whether there wil  be
any more specific targeting of areas with the threshold cut-offs (have we considered using
different cut-off rules for different parts of the population?). Gareth to assign SNZ owner and get
information sent out before 9 Nov.

AP2-14: Stats NZ needs to consider the public perception of the extended use of admin data and
use of name, age and address to match to admin sources of personal information. Kathy to action
and respond back before 9 Nov.

AP2-15: Need more information about the quality of the ethnicity data in the IDI-ERP. Gareth to
assign SNZ owner and identify, with Richard, the meeting where this matter can be discussed.

AP2-16: Re electoral implications, more information is required about the representativeness of
the data and impacts of imputations. A range of scenarios, including constitutional impacts for
Māori, would be useful. Gareth to work out when more information about the electoral data wil
be available and agree, with Richard, the meeting where this matter can be discussed.

3pm to 3:15 Summary of day
There was not time for the presentation on the quality assurance framework – prioritising the
data and methods discussions. There was a detailed discussion about the approaches Stats NZ is
taking to improve data quality by extending the intended use of administrative data - referred to
as “Plan B”. Other strategies to improve census information using the IDI-ERP referred to as “Plan
C”, were only discussed briefly. This wil  require more time at a subsequent meeting.

AP2-17: Request for more information about Plan C in advance of next meeting. Gareth to assign
SNZ owner and identify, with Richard, the meeting where this matter can be discussed

There was also an interest in talking about customer uses and the date and content of the product
releases – including the idea of a possible provisional release.

Kathy introduced the possibility of going back into the field to col ect more census data. The Panel
questioned whether going back into the field may also help measure the accuracy of our
imputation. A preliminary discussion about this wil  take place at the meeting with Stats NZ staff in
Christchurch on 30 October.

AP2-18: Need more information about the Post Enumeration Survey methods and outcomes
under the different imputation scenarios. Gareth to assign SNZ owner and identify, with Richard,
the meeting where this matter can be discussed.

AP2-19: Gareth to arrange travel to Christchurch for Len, Ian, Thomas and Barry by 26 Oct.

AP2-20: The panel said they would like to get a better sense of the roadmap going forward.
Gareth to assign SNZ owner and identify, with Richard, the meeting where this matter can be
discussed.

AP2-21: Circulate a list of standard releases from the census. Gareth to assign SNZ owner and
identify, with Richard, the meeting where this matter can be discussed

The next meetings dates are:
- 9 November, Auckland
- 7 December, Auckland
- 14 February, Wel ington
- 6 March, Auckland
- April date to be confirmed by 9 Nov meeting.

Action log
Ref  Date
Description
Owner
Date
Progress
Date
raised
required
closed
AP
29/8/18
TOR: Amend section of report
Gareth
Before
23/10 TOR was finalised,
Closed
1-1
where it states that the GS wil
and
Oct
distributed, reviewed and
14/9/18
sign off on the panel report.
Richard
meeting  published 14 Sept. Can be closed.
  Include reference that advice on
B
admin data is wanted - make this
more explicit.
  - Add line to TOR around
statutory requirements.
  - Add line to TOR around impacts
of census data on iwi and Māori.
AP
29/8/18   Send independent ABS report to
Gareth
Before
23/10 ABS report not yet sent
Open
1-2
members.
Oct
given the information is sensitive.
meeting  Gareth to work with Kathy and
Richard on how to give access or
present at future meeting. Update
at Nov meeting
AP
29/8/18
Set up some time to talk in next
Gareth
Before
23/10 Links to electoral calculation
Closed
1-3
meeting to talk about electoral
Oct
added to Oct meeting agenda.
23/10/18
legislation and how it's calculated
meeting  Electoral data update delivered in
Oct meeting.

- and prepare something to send
out.
AP
29/8/18
Create and send out template for  Gareth
Before
23/10 No existing template, but
Closed
1-4
conflicts of interest.
Oct
discussion at Oct meeting has led
23/10/18
meeting  to AP 2-2 – to create a register.
Wil  close this one.
AP
29/8/18
Give information on how Stats
Gareth
Before
23/10 Adele has prepared some
Open
1-5
NZ has met legal obligations.
or at Oct  information, to be discussed at the
meeting  informal meeting on 30 Oct.
AP
29/8/18
Complete declaration of secrecy
Gareth
At Oct
23/10 Ian and Tahu signed
Closed
1-6
docs for Tahu and Ian
meeting  declarations at the Oct meeting.
23/10/18

Close
AP
29/8/18
Confirm dates of next meetings.
Richard
At Oct
23/10 Nov, Dec, Feb and Mar dates  Open
1-7
meeting  confirmed. April date to be
confirmed. Leave open
AP
29/8/18
Col ate and send out information
Gareth
Before
23/10 Summary not provided at
Open
1-8
about languages (what was
Oct
Oct meeting. Wil  target Nov
considered & priorities).
meeting  meeting.
AP
29/8/18   Find out the response rate of the  Gareth
At Oct
23/10 Ran out of time for PES
Open
1-9
PES
meeting  update. To be prioritised in next 2
meetings.
AP
29/8/18
Find out the methodology of the
Richard
Before
23/10 Summary of information
Closed
1-
independent agency that
S
Oct
attached to October meeting
23/10/18
10
measured the 'awareness' KPI.
meeting  agenda.
AP
29/8/18
Prepare information for critical
Gareth
Before
23/10 Not specifical y discussed at
Open
1-
triggers.
Nov
meeting. Gareth to discuss
11
meeting  whether this fits with the panel or
better with the external review
and advise panel.
AP
29/8/18
Provide information of bias.
Gareth
At Oct
23/10 presentation of method and
Open
1-
meeting  the pro and cons of approaches
12
only touched on at Oct meeting.
This wil  be subject to Nov meeting
content. To close after that
meeting
AP
29/8/18
Provide rough indication of Māori  Gareth
At Oct
23/10 Māori descent and ethnicity
Closed
1-
response rates for next meeting.
meeting  counts presented at October
23/10/18
13
meeting. Wil  be subject to future
discussions, but wil  close this
action point.
AP
23/10/18  Revise minutes from the first
Richard
By 9

Open
2-0
meeting and present to for
and
Nov
confirmation at the meeting on 9  Gareth
November.
AP
23/10/18  Work out way to give access to
Kathy
By 9

Open
2-1
sensitive information outside of
and
Nov
the meeting room
Gareth
AP
23/10/18  Create and distribute a conflict of  Gareth
By 9

Open
2-2
interest register
Nov
AP
23/10/18  More information requested
Gareth
At 9 Nov  Topic for 9 Nov meeting.
Open
2-3
about the components and
meeting
quality of the dwel ing frame
including mail-out and list leave
approaches, what was added and
deleted during operations.
AP
23/10/18  More information requested
Gareth
By Dec

Open
2-4
about what investigations we are
meeting
doing into 2013 counts and
response rates – are we
assuming that 2013 is correct?

AP
23/10/18  Send apartment strategies (and
Gareth
By 9

Open
2-5
other targeted strategies) as
and
Nov
background to next meeting.
Steph
AP
23/10/18  Need more detail about the
Gareth
By 9
Sessions at 30 Oct meeting and
Open
2-6
totality and components of the
and
Nov
priority at ful  9 Nov meeting.
imputation methods initial y
Richard
planned and now being
implemented.
AP
23/10/18  Next time data is presented,
Gareth
For Nov
25/10 – Wil  note for future
Closed
2-7
include like for like comparisons
and
meeting  meetings
25/10
between ERPs and Census usual y  Adele
resident counts – we have used
2013 census usual residence
counts and 2018 ERP as
benchmarks in same tables.
AP
23/10/18  Next time maps are produced,
Gareth
By 9
25/10 – Wil  note for future
Closed
2-8
try to use the same colours and
Nov
meetings
25/10
more detailed response rate
breakdowns. Also ensure furture
TA breakdowns include Auckland
Local Boards in future.
AP
23/10/18  Ensure labels on al  graphs are
Gareth
By 9
25/10 – Wil  note for future
Closed
2-9
clear and any footnotes noted
Nov
meetings
25/10
with graphs. To note for next
meetings.
AP
23/10/18  Request for data on birthplace.
Gareth
By Dec
Topic for December meeting
Open
2-
meeting
10
AP
23/10/18  Discussion needed about the
Gareth
By 9
Discussion planned for 30 Oct
Open
2-
decisions to impute a larger
Nov
meeting and then at 9 Nov meeting
11
range of variables and impact on
not stated. Include ethnicity and
Maori descent.
AP
23/10/18  Send summary table and links to
Gareth
For 9

Open
2-
Census Transformation papers (in
Nov pre
12
SNZ archive site) as background
reading
to next meeting.
AP
23/10/18  More information needed about
Christine  By 9

Open
2-
IDI household matching including
Nov
13
whether there wil  be any more
specific targeting of areas with
the threshold cut-offs (have we
considered using different cut-off
rules for different parts of the
population?).
AP
23/10/18  Stats NZ needs to consider the
Kathy
By 9

Open
2-
public perception of the
Nov
14
extended use of admin data and
use of name and address to
match to admin sources of
personal information. Kathy to
action and respond back before 9
Nov.
AP
23/10/18  Require more information about
Christine  By 9
Pre reading to be sent and
Open
2-
the qualities of the ethnicity data
Nov
Christine to discuss at 9 Nov
15
in the IDI-ERP.
meeting.
AP
23/10/18  Re electoral implications, more
Kathy
By 9
Discussion planned for 9 Nov
Open
2-
information is required about the  and
Nov
meeting.
16
representativeness of the data
Gareth
and impacts of imputations. A

range of scenarios, including
constitutional impacts for Māori,
would be useful
AP
23/10/18  Request for more information
Gareth
By 9
Discussion planned for 9 Nov
Open
2-
about Plan C in advance of next
Nov
meeting.
17
meeting. Gareth to assign SNZ
owner and agree priority with
Richard.
AP
23/10/18  Need more information about
Kathy
By Dec
Topic for December meeting
Open
2-
the Post Enumeration Survey
and
meeting
18
methods and outcomes under
Brooke
the different imputation
scenarios.
AP
23/10/18  Only some panel members are
Gareth
By 26
25/10 Travel being booked by
Closed
2-
available to go to Chch on the
Oct
Brenda
26 Oct
19
30th. Confirm Ian, Len, Thomas
and Barry wil  be going. Gareth to
arrange travel with the four.