2018 Census External Data Quality Panel: Meeting 1 Minutes
Date and time
29 August 2018, 9am to 1pm
Location
Stats NZ, Auckland Office
48 Greys Avenue,
Auckland Central
Present -
Richard Bedford - Chair- AKL
panel members Alison Reid - AKL
Ian Cope – Video Conference (VC)
Barry Milne - AKL
Thomas Lumley - VC
Tahu Kukutai – audio conference until 11:20 then VC
Len Cook - AKL
Present – Stats Liz MacPherson - AKL
NZ
Kathy Connol y - AKL
Vince Galvin - AKL
Gareth Meech - AKL
Stephanie Prosser - AKL
Michel e Feyen - from 11.20 - AKL
Aimee Byrne (minute taker) - AKL
Denise McGregor - from 11.20 - WLG
Michael Berry - from 11.20
Christine Bycroft - from 11.20
Richard Stokes - from 11.20
Abbreviations
EDQP: External Data Quality Panel
TOR: Terms of Reference
GS: Government Statistician
Meeting minutes
09.00 - Welcome and introduction to the panel session
The Chair welcomes everyone and discusses the agenda for the day. He asks everyone to give
their name and background as an opportunity to talk about skil s and experience each panel
member brings.
Each panel introduced themselves and some took the opportunity to expresses concerns about
data quality. Panel members confirm they are happy to be part of the group and to contribute
moving forward.
Liz MacPherson, the Government Statistician takes the opportunity to thank panel members for
attending. Asserted that Stats NZ wil be open with the panel. Liz has expectation the panel wil
provide an independent report when it's finished. She appreciates having the people around table
to help get them get them where they need to. Hopes to produce high quality data that is fit for
purpose.
The chair opens the floor for comments, questions and suggestions.
Terms of Reference discussions:
Panel report:
There was a discussion about the panel terms of reference - specifical y to the description of the
report, and who signs off the final report. It is confirmed the chair wil sign off the report.
Error in document noted stating that the Government Statistician and chair wil sign-off report.
It was also noted about the reference to greater use of admin data. Agreed to amend both points.
Discussion of disagreement between panel members:
There was a discussion about what would happen if the panel members were to unable to agree.
The chair acknowledges the panel need to al ow for a situation where there maybe differences of
view and they can be acknowledged in the report
Discussion of conflict of interest:
The panel raised that it may be difficult to dissociate advice from their other interests. The chair
responds emphasizing there is information that wil be discussed during EDQP meetings that
cannot go back to the other organisation.
Liz described that the panel member was selected due to their specific experiences along with
census experience. Suggests it would be useful for member to have conversation with employer if
concerned.
Liz goes on to day that Stats NZ are looking for diversity of thought for the EDQP. Wants the panel
to work through issues and states this wil take testing from whole lot of different perspectives.
AP 1-1: Gareth agrees to panel member suggestion of having a conflict of interest template which
he wil send out.
Gareth explains that the Stats NZ Terms of Reference (TOR) was closely based on the Office of
National Statistics (ONS) UK Census independent panel TOR. Also explains that he spoke to
Australia Bureau of Statistics (ABS) however theirs had a different focus in that they only looked at
final data with good independent report. Iterates the key difference for the Stats NZ panel is that
we want advice and wil take it into account.
The chair suggests that the TOR 'in scope' section can be established as the group goes forward.
The panel wil discuss operational strategies going forward, although this is not a current focus.
Panel members sign letter of appointment.
AP 1-2: Gareth to send independent ABS report to panel.
Māori and general electorates
The panel discussed the potential implication for Māori electoral population & Māori electorates.
It is a distinctive issue due to statutory requirement.
AP 1-3: Set up some time to talk in next meeting to talk about electoral legislation and how it's
calculated and prepare something to send out.
Liz states that Stats NZ wil have two external reviews for the 2018 Census. As Government
Statistician she has three approaches:
1. The External Data Quality Panel (EDQP) set up to obtain best possible quality census data.
There wil be lessons learned along the way.
2. Understand how 2018 Census got to this outcome. Investigate what did and didn't go as
planned - to inform what we can be done in the future as Stats NZ are currently planning the
next census.
3. External review of the 2018 Census. It is expected that the external reviewers wil want to talk
to the EDQP and learn their views along the way about quality of data. Both groups work wil
go alongside each other.
Liz emphasizes the focus of EDQP group is on the data and the external review focus is learning
lessons of entire process.
The panel describes the data quality chal enges wil include:
- smal areas, deprivation indexes, health board funding, meshblock data, and availability and
quality of admin data.
9.45 - Declaration of secrecy and census embargo constraints discussion.
Packs of the panel papers wil be kept at Stats NZ in between meetings - which includes
notebooks. Declaration of secrecy is explained, and al present panel members are asked to sign
copies. Remaining panel members wil sign at the next meeting.
Gareth emphasises the importance of declaration for protecting confidentiality of respondent’s
data. It is unlikely for members of the panel to see unit record data; Stats NZ wil be avoiding this
if possible.
If the panel members were to see any information that could identify someone, they are not
al owed to talk to anyone about it. Unrounded numbers wil often be used. The panel is not
al owed to take any data away or talk to anyone.
The however chair explains that Stats should anticipate other people wil be asked about the
panel and what the panel is for. The chair has the opinion the panel should work with a very high
level of trust. Stats NZ need to be prepared that panel ists wil be frank about the nature issues
being talked about and thinks it is important the panel feel responsible about what can or can't be
said.
From time to time however, a panel member may need to ask a wel -informed col eague for
advice – the chair doesn't believe Stats NZ should not be concerned about this. This is not
breaching confidentiality but exercising the opportunity to bring the best advice to the panel.
Discussion of trust and embargoed data.
Gareth confirms over next couple of months there wil be data presented to panel that is not final
information. Numbers wil change during processing and while asking advice from panel.
The chair points out that Stats NZ is asking for a judgement and that it's like dealing with the
process of different views, ensuring to bring best advice. The panel is a group of informed people
who are already conscious that the current situation but may not have al the information.
Liz clarifies for panel to work wel there needs to be trust. Explains as Gareth mentioned, the
panel are being brought in within usual embargo. Stats NZ wouldn't ordinarily put this information
(or information out currently) out public until the point of publishing data. Stats NZ have thought
long and hard about what has been published, and the results that are out are published as
'interim results'. The panel wil be working through where the census is now- which wil change
along course. Liz expresses issue that panel ists do need to be upfront that they are on panel, why
they are on panel and the sorts of issues the panel wil be considering, for example 'electoral
data'. Advises the panel can say 'yes we wil be looking at this issue and the impacts.' However,
she clarifies she is uncomfortable going out now and saying what is put out in public domain in
the interim, is now different.
The chair clarifies the panel does not intend to go out to do that. However, the question was if
panel members talk with people - while retaining confidence with people they work with, can
there be a good and open discussion? States the panel wil be careful with what they say. Stats
have also acknowledged (critical y) in their document there is a historical process, of greater
under enumeration over time. The chair believes Stats NZ needs to maintain this level of honesty
going forward. Ability to have conversation that could be relevant. The ability to have
conversation with people around issues relevant to EDQP.
Liz ultimately wants the group to give confidence for the decisions to be made. Within the panel -
encourages discussions to be frank. As wel as openness, the panel need to maintain
confidentiality.
Another member agrees with discussion. Answers there is no black and white. Understands the
need to exercise best judgement at al times.
There wil be expectations at some stage that the network (Māori Data Sovereignty) wil be
making comment on the report when it is published. Expresses that they don’t see issue
maintaining confidentiality leading up to publication of report. There are also higher-level issues
that people have ongoing interest in and panel may want to engage on some level without
revealing details. May need to have conversation about that later. Doesn't see problem about
being on the panel.
The other panel members are happy with principles.
Suggestion to have key messages to use.
The panel recommends Stats prove a set of key messages the panel could use – similar to what
the ONS provided.
Documentation to be signed.
Gareth requests that panel members re-read and sign the appointment letter, to sign the
declaration of secrecy and sign the IRD declaration form. Gareth to witness signatures.
Discussion of how Stats NZ wil store panel members notes/packs.
A member questions ask whether the information/notes left by panel members wil be locked up
securely by Stats NZ. Stats confirmed that the information wil be taken to Wel ington and wil be
secured with a lock until next meeting. Some concerns are noted about needing to hand back
notes as Stats NZ wil have the only copy.
Discussion of paper with fee information:
Request to panel members to provide al details for payment depending on tax status etc.
Minor corrections are made to name titles.
Liz informs panel of proposed media release.
Liz highlights that there wil be high level of interest in the panel. Expresses interest in Stats NZ
releasing a media release to saying what panel is, who the members are and also to publish the
TOR. Liz confirms panel members can clarify to people that they are on panel if asked.
Suggestion that panel members be given a set of key messages after each meeting. The panel
would like to see media release before publishing. The chair wil be only spokesperson if needed.
The chair asks group are there any other issues to raise.
Request for more information about address frame checks
A member questions how Stats NZ are sure they have fol owed the obligation of the Statistics act
in the census electronic system.
Liz indicates that this is something the external review would do. She informs Stats NZ worked
with legal counsel team around legal obligations.
Another member requests feedback about Stats NZ meeting its legal obligations. They would not
like to give advice on the panel unless she knows the legal obligations had been met. Based on
panel comments - Stats NZ agrees to provide something to panel on legal / statutory obligations.
The chair raises discussion of the final report.
The chair states it won't be confidential and wil be available to wider audience. Liz proposes to
release report alongside first publication of census results.
A member does not think a day for report can be confirmed yet. Raises that if there is a very big
delay for results, there may need to publish the report ahead of data release. There would usual y
be an accompanying methodology report for people to understand the report. The release date is
something the panel and Stats NZ can work through.
Another member questions whether there is required/ statutory date that the data has to be
released? Stats confirms there is not, but needs to meet electoral and boundary review timelines
– which have some flexibility at the moment.
Gareth explains Stats NZ are happy to provide resource for the writing of the report.
Liz informs the panel that Stats NZ also have an internal quality panel that wil be available for
EDQP. The panel appreciates having the offered by expertise Stats NZ however raises the EDQP
need to be mindful of the independence of their report. Liz strongly agrees but wants the panel to
feel they can cal on Stats NZ people if they need to.
11:20 (Thomas left and the following joined the meeting: Tahu on VC, Christine VC, Mike &
Denise VC , Michel e Feyen joined, Richard Stokes CHCH)
2018 Census model strategy and answer Q&As
Kathy and Steph present to panel.
Issues / questions raised during presentation:
Processes:
A member questions recruitment as it was different for 2018 Census. Asks were field staff trained
by Stats NZ? Stats NZ Recruitment company employed staff - (Organised time sheet etc) - however
Stats NZ stil trained the staff.
The chair is interested in the processes of the different stages of establishing whether someone
has fil ed in the census. (Gives example of someone absent from usual residence on night).
Questions about what was the control over ensuring that people had completed the census? Stats
NZ – notes there were some chal enges. Explains part of the problem of processing and timing
was that it was not in real time. This was a reflection of not being able to test this system prior.
Important to note great level of change for this time that is outside the tested experience.
Targeted operations:
The chair questions whether the census general y got good support from local communities given
the importance of data for them. Stats NZ – responds that it did & named communities. Support
also from migrant communities and local government advocate programme which connected
throughout census.
A member questions was there ever a thought offer other language forms other than Te Reo.
Stats NZ - was considered, but it would have been a complicated process. Instead the cal centre
offered language help. Another member clarifies that the language approach NZ used was very
consistent with UK and Australian census. These chal enges existed even in the traditional model.
Stats confirmed 7 different languages pamphlets were produced. Cal centre had 9 languages
available. Tablets had 25 languages.
Stats NZ worked with ethnic organisations to help provide support. Prioritized parameters were
around Te Reo, and security. Had a lot of questionnaire problems to solve, and choices had to be
made. Action point to col ate and send out information around languages for 2018 Census.
Targeted areas:
One member states that looking at targeted areas would imply a lot for Auckland. Stats NZ -
informs maps of targeted areas wil be shown in another meeting.
PES:
A member questions requests the response rate for the 2018 PES? Stats NZ – informed PES has
not been processed.
KPI's:
Stats presented that the communications campaign for Stats NZ performed wel . Panel members
question the methodology of independent agency used for KPI of awareness. Action point to send
more information.
A member expresses that they finds it problematic that Māori did not have the same KPI as rest of
population especial y given they are a targeted group. Does not understand rationale that Stats
NZ would build inequity in targets.
The panel questions whether the KPIs were low, so that the target could be met. Stats NZ -
explains that the research was done to establish a baseline for KPIs. So, for the case of Māori the
baseline was set for a lower level of awareness. Additional y, the baselines were set 3 years before
census, and they were measured November 2017 to see the effect the campaign had.
Paper forms:
The chair questions the number of people who requested and completed census by paper.
A member raises that in the past when census was not online, the census didn't have rely on
people’s initiatives. The 2018 Census has introduced quite a few trigger points. Suggests it would
be helpful to get a flow chart to show approach to trigger responses for people to complete the
census.
Household forms & field officers
A member points out that NZ is unique to other countries that do the census (UK/Australia) in that
they don't have just household forms, have personal form. Questions whether this had an impact.
Stats NZ - it did made it harder this time.
Another member discusses that having field officers on the ground is the only way from changing
a non-response to respond. Understand change in staff number 2018 Census, questions whether
this is evidence of not having enough staff.
Triggers for risks:
A member questions what the critical triggers were - that provided a trigger or information that
there was a problem. Would like deeper understand outcomes. Stats NZ - agrees Stats can
prepare the triggers where decisions were made, e.g. staying longer in the field & assisted
completion events. The MIS (management information system) as a way to track target response
groups.
Kathy provides overview of what things didn't go wel .
Kathy explains ABS have visited and the would be happy to provide presentation of their thoughts
and gave examples of things that didn't go wel .
The chair comments that by using new system for recruitment, and suggested Stats NZ had lost a
lot of knowledge that previous field staff had. Suggests that Stats NZ underestimated the skil s the
old enumerators had in getting forms completed.
Vince presents the current numbers for processing
It is re-iterated that the numbers shown are stil shifting as processing is ongoing. Panel informed
that the next meeting wil focus on data.
A member points out numbers imply a low growth rate between 2013 and 2018, which is not
credible. Stats NZ - Stats have been monitoring numbers in categories - numbers have jumped
around. Stil a lot of cleaning up to do.
The member questions whether Stats NZ can break down partial information from people who did
it on web vs paper system. Stats NZ - Stats are questioning everything – including online and
paper.
1pm – The chair concludes the meeting and thanks members for participation:
The chair explains it has been useful to get frank summary of how things went- how you thought
things and it has been helpful to see high level data. Hopes to see more data at the next meeting.
If people attend meetings Stats NZ should set up one-on-ones and they wil be ful y briefed.
The chair shares concern that the panel shouldn’t get too broad if looking at quality of data.
Agrees a high-level summary is needed as the number of changes to this year census was
big. Some members are interested in understanding the bias.
Another member questions if there is sense for timing of indicative response rates for Māori. Stats
hopes to have high level information of this for next meeting.
Action log
Ref
Date
Description
Owner
Date
Progress
Date
raised
required
closed
AP 1-
29/8/18
TOR: Amend section of report
Gareth
Before
Open
1
where it states that the GS wil
and
Oct
sign off on the panel report.
Richard B meeting
Include reference that advice
on admin data is wanted -
make this more explicit.
- Add line to TOR around
statutory requirements.
- Add line to TOR around
impacts of census data on iwi
and Māori.
AP 1-
29/8/18 Send independent ABS report
Gareth
Before
Open
2
to members.
Oct
meeting
AP 1-
29/8/18
Set up some time to talk in
Gareth
Before
Open
3
next meeting to talk about
Oct
electoral legislation and how
meeting
it's calculated - and prepare
something to send out.
AP 1-
29/8/18
Create and send out template
Gareth
Before
Open
4
for conflicts of interest.
Oct
meeting
AP 1-
29/8/18
Give information on how
Gareth
Before
Open
5
respondents were to meet
or at Oct
their legal obligations.
meeting
AP 1-
29/8/18
Complete declaration of
Gareth
At Oct
Open
6
secrecy docs for Tahu and Ian
meeting
AP 1-
29/8/18
Confirm dates of next
Richard
At Oct
Open
7
meetings.
meeting
AP 1-
29/8/18
Col ate and send out
Gareth
Before
Open
8
information about languages
Oct
(what was considered &
meeting
priorities).
AP 1-
29/8/18 Find out the response rate of
Gareth
At Oct
Open
9
the PES
meeting
AP 1-
29/8/18
Find out the methodology of
Richard S Before
Open
10
the independent agency that
Oct
measured the 'awareness' KPI.
meeting
AP 1-
29/8/18
Prepare information for critical Gareth
Before
Open
11
triggers.
Nov
meeting
AP 1-
29/8/18
Provide information of bias.
Gareth
At Oct
Open
12
meeting
AP 1-
29/8/18
Provide rough indication of
Gareth
At Oct
Open
13
Māori response rates for next
meeting
meeting.
