1 September 2022

Scott
[FYI request #20128 email]

Tēnā koe Scott

Official Information Act request: Year One Review of the Algorithm Charter
Thank you for your email dated 4 August 2022 in which you requested, under the Official
Information Act 1982 (the Act), the following information:
I would like to make a request for information relating to the recently released Taylor Fry
Year One review of the StatsNZ Algorithm Charter.

I would like to request:

ONE: Copies of all aides-mémoire, memoranda, briefings, and reports produced by
StatsNZ for the responsible minister on the topic of the review and StatsNZ’s response to
it.

TWO: If an action plan, work plan or implementation plan has been prepared for
StatsNZ’s response to the review I would like to request a copy of it.

THREE: A StatsNZ memorandum prepared for Minister Clark dated September 2021
refers to a “maturity assessment” StatsNZ was working on to assess its own use of the
Charter:

“Since signing the Charter, Stats NZ has begun work on a maturity assessment to
understand what gaps and issues around algorithm use might exist relating to the
Charter. When this is completed, options to address the gaps will be developed. Stats NZ
intends to publish a report on its findings and solutions early next year to ensure
transparency and to help agencies who are undertaking similar reviews”

The memorandum I am referring to can be accessed here (quote on page 6):

https://fyi.org.nz/request/19545/response/74734/attach/5/OIA%20DC%2043%202022%2
0Documents.pdf

I would like to request a copy of this maturity assessment. I would also like to request a
copy of whatever document sets out the “options to address the gaps” referred to above,
as well as any action plan, work plan or implementation plan prepared for this, if any.

In response to your first question please find attached a copy of Report to the Minister of
Statistics - Report on the Algorithm Charter Review. This is the only document produced by
Stats NZ for the responsible minister, on the topic of the review and Stats NZ’s response to
it. You will note within the document that some information is withheld under section
9(2)(f)(iv) of the Act as it is under active consideration. The release of this information is
likely to prejudice the ability of government to consider advice and the wider public interest
of effective government would not be served.

You can also find more information about the charter, including a copy of the review
published online here: Algorithm charter for Aotearoa New Zealand - data.govt.nz.

In response to your second question, this part of your official information request is refused
under section 18(e) of the Act as this information does not exist or, despite reasonable
efforts to locate it, cannot be found.
In response to your third question, the Algorithm Maturity Assessment is still currently under
development.  Two  internal  case  studies  were  put  through  a  draft  Algorithm  Maturity
Assessment  to  test  the  framework.  A  workshop  to  develop  solutions  and  options  from
identified gaps was planned during August 2021. However, the workshop was cancelled due
to  the  COVID-19  lockdown,  and  work  was  then  put  on  hold  more  widely  due  to  internal
reprioritisation of resources towards unanticipated high priority work.

Work on the Algorithm Maturity Assessment has only recently resumed. Documents relating
to the case studies remain in a draft working state, and the work to identify options to address
the gaps has not yet been undertaken. As such I am withholding this information under section
9(2)(g)(i) of the act to protect the effective conduct of public affairs through the free and frank
expression  of  opinions.  I  believe  the  greater  public  interest  is  in  the  ability  of  individuals  to
express opinions in the course of their duty.

However, it is still our intention to produce and publish a report on the findings and solutions
of the Algorithm Maturity Assessment, when this work has been completed. We will notify you
via this email when the report becomes publicly available. Additionally, please find attached a
copy of the Algorithm Maturity Assessment framework  that we used, noting that it is still in
draft form.

If  you  are  not  satisfied  with  this  response,  you  have  the  right  to  seek  an  investigation  and
review  by  the  Ombudsman.  Information  about  how  to  make  a  complaint  is  available  at
www.ombudsman.parliament.nz or 0800 802 602.
It  is  our  policy  to  proactively  release  our  responses  to  official  information  requests  where
possible.  This letter, with your personal details removed, will be published on the Stats NZ
website.  Publishing  responses  creates  greater  openness  and  transparency  of  government
decision-making, and helps better inform public understanding of the reasons for decisions.

Ngā mihi nui

Blair Cardno
General Manager, Data Management and Operations

Algorithm Maturity Questions for XXX
Charter
Questions
What has been
Links to evidence
Gaps identified
commitment
done?
Transparency
Has Stats NZ provided information as to the purpose and benefits

- clearly explain
of the relevant algorithm/methods for the communities who may
how decisions are
be affected by decisions made outside Stats NZ?
informed by
Has Stats NZ provided plain English documentation of the

algorithms.
relevant algorithms/methods for researchers who will use the
data?
Has Stats NZ provided plain English documentation of the

relevant algorithms/methods for the communities in the data?
Has Stats NZ made information about the data and processes

available (unless a lawful restriction prevents this)?
Has Stats NZ published information about how data are collected?

Has Stats NZ published information about how data are stored

securely?
Has Stats NZ published information about where the data are

stored?
Partnership
Has Stats NZ embedded a Te Ao Māori perspective in the

– deliver clear
development and use of these algorithms/methods consistent with
public benefit
the principles of the Treaty of Waitangi?
through Treaty
Does Stats NZ have established systems, policies, and

commitments
procedures to support ethical, responsible, and culturally
appropriate practices when working with algorithms/methods?
Has Stats NZ identified people, communities and groups who

have an interest in these algorithms / methods, both technical
users and communities in the data?
Focus on people
Has Stats NZ identified the benefit of these algorithms for Stats

NZ and other agencies and whether they are in line with the goals
of the communities of interest?
Has Stats NZ identified the potential harm of these algorithms if

they are miss-used by others?

1

Charter
Questions
What has been
Links to evidence
Gaps identified
commitment
done?
Has Stats NZ actively engaged with the people, communities, and

groups who were identified above?
Has Stats NZ understood and documented the limitations of the

data used or produced by the algorithms / methods?
Make sure data is
Has Stats NZ understood the cultural context of these algorithms?

fit for purpose
Has Stats NZ understood the research context of these

algorithms?
Has Stats NZ identified any bias introduced by the algorithms/

methods?
Has Stats NZ got a plan for managing this bias?

Has Stats NZ established a process to identify opportunities for

new algorithms?
Has Stats NZ commissioned any peer reviews of these

algorithms/ methods?
Privacy, ethics, and
Did these peer reviews cover privacy, ethics, and human rights

human rights
concerns?
Has Stats NZ considered the privacy concerns of whānau and

community groups as well as those of individuals?
Have any unintended consequences of the algorithms / methods

been identified? If so, what has been done to act on this
information?
Does Stats NZ have a nominated point of contact for public

inquiries about algorithms/methods?
Human oversight
Can citizens find and use this point of contact to ask questions or

make complaints?
Have any question or complaints been received by Stats NZ?

Were they dealt with appropriately?
Does Stats NZ provide a channel for challenging or appealing

decisions informed by algorithms/methods?
Has Stats NZ clearly explained the role of humans in decisions

informed by algorithms/methods?

2

Report to the Minister of Statistics:
Report on the Algorithm Charter Review
Date
27 July 2022
Priority
L
Ref number
MM2319
Timeline and next steps
Decision or action
N/A
required by:
Purpose
This briefing provides you with an overview of the independent report ‘1-year
review of the Algorithm Charter for Aotearoa New Zealand’, prior to its publication.
Linkages

Publicity
The 1-year review will be published on data.govt.nz.
Recommended action
It is recommended that you:
1.
Note that an independent review of the Algorithm Charter’s first year of operation has been completed
by the consultancy Taylor Fry.
         NOTED
2.
Note that the review found that there is almost universal support for the Charter amongst government
agencies and subject matter experts, but to realise the desired shifts in the ethical use of algorithms,
agencies require additional guidance and supports to effectively implement the Charter.
                                                                              NOTED
3.
Note that the independent review of the Algorithm Charter will be published on the data.govt.nz
website.
                         NOTED
4.
Agree to share a copy of this briefing with the Associate Minister of Statistics given her portfolio
responsibilities for Māori data capability.
      AGREE / DISAGREE

Caleb Johnstone
Hon Dr David Clark
General Manager – Data System Policy
Minister of Statistics
Data System Leadership
Date:

Stats NZ

In confidence

Background
1.  In July 2020, the Algorithm Charter for Aotearoa New Zealand (attached as Appendix 1) was
released. The intention of the Algorithm Charter was to increase public confidence and visibility
around the use of algorithms within the public sector. The Charter demonstrates a commitment
to ensuring New Zealanders have confidence in how government agencies use algorithms.
2.  The Charter is voluntary and applies solely to the public sector.  The Algorithm Charter has 28
signatory agencies so far. Some current non-signatory agencies have expressed an interest in
signing up in the future. A list of the signatories is attached as Appendix 2.
3.  Conducting  a  review  of  the  Algorithm  Charter  after  its  first  year  was  agreed  as  part  of  its
establishment.
4.  An  independent  review  of  the  Algorithm  Charter’s  first  year  of  operation  was  conducted  by
specialist consultants Taylor Fry. Taylor Fry was also the consultancy used by the New Zealand
Police to conduct a stocktake of their algorithms, and to provide advice on best practice to assure
safe and ethical development and use of algorithms.
5.  The  independent  review was  conducted  in  consultation  with  signatories,  non-signatories  and
subject matter experts.
Review findings
6.  The  review  found  that  there  is  almost  universal  support  for  the  Algorithm  Charter  amongst
government agencies and subject matter experts, and that the Charter has been influential in
changing practice.
7.  As  a  result  of  the  Charter,  agencies  have  implemented  new  risk  management  policies;
established  ethics  committees  and  review  boards;  created  new  roles  to  oversee  emerging
technologies; performed stocktakes and reviews of their algorithms; evaluated the potential for
risks that would have previously been unforeseen; and been transparent with the public about
the types of algorithms that are being used.
8.  The review specifically pointed to several notable shifts as a result of the Charter, including:
•  increased  executive  visibility  of  the  importance  of  data  ethics  and  the  data
management practices within their agencies – with data teams also reporting that they
have an improved remit to raise issues at an executive level; and
•  improved  capability  when  interacting  with  third-party  vendors  –  with  data  teams
asking more questions and providing greater scrutiny over models being considered
for purchase.
9.  The review also found that agencies would not have made the move towards publishing their
algorithms online, if it had not been for the Charter.
10.  However, the review also identified a number of areas where agencies require additional support
to  implement  the  Charter.  The  review  established  five  themes,  each  with  a  series  of
considerations focussed on improving progress towards the Charter’s commitments. These are
summarised below:
Themes and considerations
Charter specifics
The review reported that most agencies see real value in the Charter, however its
application can be hampered by a lack of clarity about what should be considered an
algorithm, and the supporting risk matrix - which can be seen as a simplistic tool with
limited use.
The review recommends supplementing existing guidance with further information that
clarifies some of the points of confusion and provides more examples of algorithms, tools
and processes that should and shouldn’t be captured under the Charter.
Capability and capacity
In confidence

The review reported that agencies would welcome a community of practice and an
oversight body to support compliance with the Charter, as measuring bias is not something
that all agencies have expertise in.
The review recommends:
•  facilitating a community of practice for signatories of the Charter, enabling the
sharing of examples where agencies are delivering well on the principles;
•  creating an oversight body for the Charter; and
•  developing a more detailed resource guide for agencies – helping to form guidelines
and principles for best practice, including technical descriptions of bias assessment
protocols.
Engaging with Treaty Partners
The review found that many agencies are not clear on how to practically implement the
partnership commitment in the Charter and that the capacity of experts to support agencies
is limited.
The review recommends:
•  working with Māori data experts to develop a more detailed guidance white paper
on what the partnership commitment should consist of, and issuing subsequent best
practice.
•  helping agencies to fulfil the principle of parity when engaging in consultation and
ensuring government agencies have realistic expectations about timeframes for
consultation with experts.
Public Awareness
The review found that public reporting of the algorithms in use is fragmented and incomplete
and therefore public awareness of use by government agencies is limited.
The review recommends developing an annually updated register of algorithms to increase
public visibility and investigating novel forms of citizen participation such as citizen
assemblies and focus groups to increase the public’s awareness of algorithms.
Wider context
The review found that the “light regulatory nature” of the Charter places limitations on its
ability to offer public assurance and facilitate public trust. In addition, the review observed
that algorithms sit within a broader algorithmic system incorporating data sourcing, data use,
presentation of algorithm output to users, and decision-making – highlighting the importance
of taking a consistent and visible approach across the whole system.
The review recommended:
•  considering ways, such as a publicly available register of algorithms, to evolve the
Charter from its light regulatory nature to encourage compliance and best practice;
and
•  the provision of advice on system-wide data governance and establishing non-
binding audits of Charter compliance.
Next steps
11.  Stats NZ supports the findings of the review and is undertaking planning work  as part of the
wider work programme focussed on data ethics s(9)(2)(f)(iv)

3

s(9)(2)(f)(iv)

12.  Policy options on data ethics are in early development stages, with a progress update  due in
the coming months. A key part of this process is to test approach and seek direction from the
Digital Executive Board.
13.  In the interim, work is underway to determine a phased approach for implementing the remaining
recommendations.  This will involve working with others to reduce any duplication, make best
use of existing resource, and to build momentum. Progress updates will be provided regularly.
14.  A copy of the review will be published on data.govt.nz. There is an opportunity to start talking
more widely about the Algorithm Charter, its application, and the Taylor Fry review. Stats NZ
has been asked by RNZ to participate in a panel on Artificial Intelligence on 11 August 2022,
which will likely cover the Charter.
15.  We will continue to advise your Office as future opportunities arise.
4

Appendix 1: Algorithm Charter for Aotearoa New Zealand

In confidence

Appendix 2: Signatories
Current signatories to the charter are:
•  Ara Poutama Aotearoa — The Department of Corrections
•  Te Tāhuhu o te Mātauranga — The Ministry of Education
•  Te Manatū Mō Te Taiao — The Ministry for the Environment
•  Te Tūāpapa Kura Kāinga — The Ministry of Housing and Urban Development
•  Te Tari Taake — Inland Revenue
•  Te Tāhū o te Ture — The Ministry of Justice
•  Toitū Te Whenua — Land Information New Zealand
•  Te Puni Kōkiri — The Ministry of Māori Development
•  Oranga Tamariki - The Ministry for Children
•  Te Manatū mō ngā Iwi o te Moana-nui-ā-Kiwa — The Ministry for Pacific Peoples
•  Te Manatū Whakahiato Ora — The Ministry of Social Development
•  Tatauranga Aotearoa — Stats NZ
•  Te Manatū Waka — The Ministry of Transport
•  Te Kāhui Whakamana Rua Tekau mā Iwa—Pike River Recovery Agency
•  Manatū Wāhine — The Ministry for Women
•  Toi Hau Tāngata — Social Wellbeing Agency
•  Te Ope Kātua o Aotearoa — New Zealand Defence Force
•  Te Kaporeihana Āwhina Hunga Whara — Accident Compensation Corporation
•  Te Tari Taiwhenua — Department of Internal Affairs
•  Te Arawhiti — The Office for Māori Crown Relations
•  Waka Kotahi — The New Zealand Transport Agency
•  Te Tari Arotake Matauranga — The Education Review Office
•  Hīkina Whakatutuki — The Ministry of Business, Innovation, and Employment
•  Manatū Aorere — The Ministry of Foreign Affairs and Trade
•  Manatū Hauora — The Ministry of Health
•  Ngā Pirihimana O Aotearoa — New Zealand Police
•  The Mental Health and Wellbeing Commission
•  Te Kawa Mataaho — Public Service Commission

8

Document Outline