Appendix A
From:
Adam Jackson
Sent:
Wednesday, 29 January 2025 11:18 am
To:
Pip Van Der Scheer
Cc:
Isabelle Sin
Subject:
Privacy
Hey, circling back on privacy. Sounds like it is very low risk. It’s covered in the contract we’re using
and s 9(2)(h)
of course, we were expecting them
to comply with their obligations under the Privacy Act and contract and keep personal information
confidential and appropriately secure.
Let me know when you think we have our ducks in a row enough for a further conversation with the
provider and I’ll line that up.
Adam Jackson
Chief Advisor to the Chief Executive
Ministry for Regulation
Mobile: s 9(2)(a)
| Email: [email address]
www.regulation.govt.nz
1
From:
s 9(2)(a)
@publicvoice.co.nz>
Sent:
Thursday, 30 January 2025 1:39 pm
To:
Adam Jackson; Pip Van Der Scheer; Isabelle Sin
Subject:
RE: RSB Submission Analysis
Hi All,
Good to catch up. Some more information about some of AI tools we use to improve productivity.
MAXQDA (PublicVoice is a certified reseller in New Zealand) is one of the AI tools we use to speed the analysis
process up. Here is some more information about the security - AI data protection - MAXQDA and AI Assist for
qualitative data analysis - MAXQDA
Most importantly
No AI model training on user data
SSL/TLS 1.3 encryption for data transmission
Zero data retention
Cheers,
s 9(2)(a)
-----Original Appointment-----
From: Adam Jackson <[email address]>
Sent: Thursday, 30 January 2025 10:02 am
To: Adam Jackson; s 9(2)(a)
; Pip Van Der Scheer; Isabelle Sin
Subject: RSB Submission Analysis
When: Thursday, 30 January 2025 12:30 pm-1:00 pm (UTC+12:00) Auckland, Wellington.
Where: Microsoft Teams Meeting
Hi s 9(2)(a) ,
I’ve had something come up between 12 and 12.30 so let’s grab the second half of that slot. Talk
soon.
Thanks
Adam
________________________________________________________________________________
Microsoft Teams Need help?
Join the meeting now
Meeting ID: 410 041 277 41
Passcode: K5ni6YX2
Dial in by phone
1
+64 4 889 8046,,399072101# New Zealand, Wellington
Find a local number
Phone conference ID: 399 072 101#
For organizers: Meeting options | Reset dial-in PIN
________________________________________________________________________________
..........................................................................................................................................
Confidentiality notice: This email may be confidential or legally privileged. If you have received it by mistake, please tell the
sender immediately by reply, remove this email and the reply from your system, and don’t act on it in any other way. Ngā mihi.
2
From:
Aimee Riddell
Sent:
Thursday, 30 January 2025 3:26 pm
To:
Jeremy Shoebridge; Adam Jackson
Cc:
Pip Van Der Scheer; Laura Fair; Isabelle Sin
Subject:
RE: AI usage by provider
Thanks Adam, nothing further to add at this point.
Agree s 9(2)(h)
retention and disposal after completing work, suggest adding under item
(a)
Cheers,
Aimee
From: Jeremy Shoebridge <[email address]>
Sent: Thursday, 30 January 2025 3:19 pm
To: Adam Jackson <[email address]>; Aimee Riddell <[email address]>
Cc: Pip Van Der Scheer <[email address]>; Laura Fair <[email address]>;
Isabelle Sin <Isabel [email address]>
Subject: RE: AI usage by provider
s 9(2)(h)
Cheers,
Jeremy
Jeremy Shoebridge (he/him)
Acting Head of Legal
Ministry for Regulation
īmēra: [email address] | waea pukoro:s 9(2)(a)
– call only, not text
www.regulation.govt.nz
This email (including any attachment) may be confidential or subject to legal privilege. Please do not forward
it outside the Ministry for Regulation without checking with a member of the Legal team first.
If you are not the intended recipient of this email, do not read, copy, use, forward or disclose the email or any of its
attachments to others. Instead, please immediately report this by replying to this email and then delete it and the reply
from your system.
1
From: Adam Jackson <[email address]>
Sent: Thursday, 30 January 2025 3:04 pm
To: Aimee Riddell <[email address]>; Jeremy Shoebridge
<[email address]>
Cc: Pip Van Der Scheer <[email address]>; Laura Fair <[email address]>;
Isabelle Sin <[email address]>
Subject: AI usage by provider
Hi both,
As discussed with Aimee earlier, we’re exploring whether our service provider can use AI to help us
with submissions analysis.
Aimee thought it sounded fine, so long as we cover o the following questions:
(a) Will the data we provide to our Service Provider (Public Voice) be used to train the model for
future use? (No is the answer we want.)
(b) Where are the servers for the AI located? (Hopefully not some dodgy place.)
(c) How will they construct the prompts used to analyse the data? (Ideally suggesting a human-
like analysis.)
(d) How will they mitigate the risk of AI error (eg hallucination)?
s 9(2)(h)
If not,
I’ll send the provider an email inquiring about these things shortly – other than the first one, for which
they have already told us the answer is “no” and confirmed in writing. (They’ve also confirmed that
the model has SSL.TLS 1.3 encryption for data transmission and zero data retention.)
The answer to (c) will almost certainly be that they are constructed using the tags that we (humans)
provide, which Aimee is happy with.
Andrew is on board with AI use, subject to the above, and we know the Minister supports us using it.
Thanks
Adam
Adam Jackson
Chief Advisor to the Chief Executive
Ministry for Regulation
Mobile: s 9(2)(a)
| Email: [email address]
www.regulation.govt.nz
2
From:
Aimee Riddell
Sent:
Friday, 31 January 2025 1:46 pm
To:
Adam Jackson
Cc:
Isabelle Sin; Pip Van Der Scheer; Laura Fair
Subject:
RE: RSB Submission Analysis
Thanks Adam, also ok with these responses. Endorsing AI usage as per MfR requirements.
Cheers,
Aimee
From: Adam Jackson <[email address]>
Sent: Friday, 31 January 2025 1:40 pm
To: Aimee Riddell <[email address]>
Cc: Isabelle Sin <[email address]>; Pip Van Der Scheer <[email address]>;
Laura Fair <[email address]>
Subject: FW: RSB Submission Analysis
Hi Aimee,
I’m happy with these responses. Are you? If so, I think we have a green light on the AI usage.
Adam Jackson
Chief Advisor to the Chief Executive
Ministry for Regulation
Mobile: s 9(2)(a)
| Email: [email address]
www.regulation.govt.nz
From: s 9(2)(a)
@publicvoice.co.nz>
Sent: Friday, 31 January 2025 11:34 am
To: Adam Jackson <[email address]>
Subject: RE: RSB Submission Analysis
Hi Adam,
Here are our responses to your questions about information handling and AI use:
(a) Confirmed - information will not be used to train AI models.
(b) Confirmed - all information will be destroyed upon project completion.
(c) Data processing will occur in New Zealand, European Union or United States.
(d) Maximum of 3 PublicVoice staff members will have access to the information.
(e) All staff accessing the data will be based in New Zealand. Zero data retention policies ensure only
PublicVoice staff have access.
1
(f) Our approach to AI prompts will be:
Converting your tag framework into structured analysis prompts
Using standardised templates to ensure consistent analysis
Testing prompts extensively before full implementation
Regular quality checks of AI outputs against human validation
No sharing of prompts or methodology outside the project team
(g) To mitigate AI error risks:
AI outputs will be human-reviewed by our analysts
Implementing multi-step validation process
Using MAXQDA's built teamwork tools
Regular cross-checking between team members
Maintaining audit trail of all analysis steps
Immediate flagging and correction of any inconsistencies
(h) Confirmed - all information will be kept strictly confidential in accordance with our privacy and
confidentiality policies. See Privacy Policy | PublicVoice Research & Consultation NZ
Best regards,
s 9(2)(a)
From: Adam Jackson <[email address]>
Sent: Friday, 31 January 2025 8:40 am
To: s 9(2)(a)
@publicvoice.co.nz>
Subject: RE: RSB Submission Analysis
Thanks s 9(2)(a) . s 9(2)(g)(i)
Adam Jackson
Chief Advisor to the Chief Executive
Ministry for Regulation
Mobile: s 9(2)(a)
| Email: [email address]
www.regulation.govt.nz
From: s 9(2)(a)
@publicvoice.co.nz>
Sent: Friday, 31 January 2025 8:38 am
To: Adam Jackson <[email address]>
Cc: Isabelle Sin <[email address]>; Laura Fair <[email address]>; Pip Van Der Scheer
<[email address]>; Aimee Riddell <[email address]>
Subject: RE: RSB Submission Analysis
Good morning, Adam,
Thanks for sending through these questions. I'll get back to you early this afternoon with responses to these AI
and privacy questions, along with the other information we discussed around approach and pricing.
Best,
s 9(2)(a)
2
From: Adam Jackson <[email address]>
Sent: Thursday, 30 January 2025 4:58 pm
To: s 9(2)(a)
@publicvoice.co.nz>
Cc: Isabelle Sin <[email address]>; Laura Fair <[email address]>; Pip Van Der Scheer
<[email address]>; Aimee Riddell <[email address]>
Subject: RE: RSB Submission Analysis
Hi s 9(2)(a) ,
I’ve just been working through a few issues around the use of AI with our IT and legal gurus and have
set out a few questions/requests for you below. I’ve also covered off a couple of more general
privacy issues. I have a pretty good idea of your answers to most from our conversations and
correspondence so far, but just need to check and get your replies in writing for assurance.
(a) Please confirm that the information we provide will not be used to train your AI model. (I know
you’ve already answered this, but if you could reply “confirmed” so we have it all in one place,
that would be helpful.)
(b) Please confirm that the information will be destroyed when the job is finished. (Again, I know
you’ve already confirmed ….)
(c) Where will the information be held/processed by the AI – as in which country(ies)? This
question is just to make sure that we don’t have personal information, etc, held in countries
with poor data protections.
(d) How many people are likely to have access the information? (A broader privacy question,
rather than an AI one.)
(e) In which country(ies) will people who have access to the information be based? (Again, a
broader privacy question rather than an AI one.)
(f) How will you construct the prompts to get the AI to analyse the data? (For example, will you be
taking our tags and turning them into prompts?)
(g) Can you give us a brief description of how you will mitigate the risk of AI error, including by
hallucination?
(h) Can you please confirm that you will keep the information we provide to you confidential, for
privacy and general confidentiality reasons?
Based on what you’ve told us already, I don’t anticipate your answers throwing up any issues, though
we haven’t previously talked about (c)/(e) – in which case we’ll be comfortable with you using AI to
help do the analysis.
Kind regards
Adam
Adam Jackson
Chief Advisor to the Chief Executive
Ministry for Regulation
Mobile: s 9(2)(a)
| Email: [email address]
www.regulation.govt.nz
3
20250309 Rules for including submissions and other useful stuff
Email submissions
• Email submissions were included only if they were sent within 32 hours of
submissions closing or if an extension was explicitly granted.
• Inclusion of submissions:
o Only one submission was retained from each submitter (based on
combination of email address and name).
o Where instructions were given on which of multiple submissions to
include, we followed these.
o Where multiple submissions were identical or very similar and no
instructions were given, we kept the last one.
o Where multiple submissions differed substantially and no instructions
were given, we kept the one with the most content.
o Where the same submitter submitted by email and CS with different
content, we combined the two into one submission.
o Where submitters sent corrections to earlier submissions, we made
these changes.
o All emails from each email address that sent more than one email were
manually examined to implement this process, and emails and submitter
names were compared across email submissions and CS submissions.
o Where a submission was sent in a format we could not access, we invited
the submitter to resubmit and included their resubmission even if it was
sent after the closing date. Such submissions were identified through an
AI procedure combined with selected manual examination.
• We made every reasonable effort to exclude from consideration emails that were
not RSB submissions.
o An AI model was also used to identify potential non-RSB submissions, all
of which were then verified manually.
o Senders who submitted on a different bill/issue were notified and invited
to resubmit on the RSB. We dropped such submissions unless the
submitter instructed us to retain them and consider them RSB
submissions.
• Treatment of multiple submissions
o Where one email contained multiple separate submissions, we treated
each as a separate submission. These were identified primarily by an AI
model, and each case was confirmed by a human.
o Where one submission was on behalf of multiple people/contained
multiple signatures, we treated it as a single submission.
Citizen Space submissions
• Where a person filled out the survey more than once (based on name and email),
we kept only their last response.
• Submissions where none of the questions were answered were retained.
• Content was considered regardless of which field it was entered in.
Coding of identity
• Identity was a collected field in Citizen Space. Submissions where this field was
missing were considered individual submissions if views were expressed in first
person singular. In the remaining 24 cases, we assumed the submission was on
behalf on an individual, but could not verify this.
• For emails, the process was as follows:
o AI was used to assign individual, iwi/hapū, or organisation to each
submission.
o The coding of identity was checked by a human in every case where AI
identified the submission as not coming from an individual. Corrections
were made where necessary.
o The small number of cases where AI could not give a definitive answer
were coded by humans.
Coding of overall stance for emails
• Emails were given preliminary classifications using a large language model (LLM)
following the logic of the classification flowchart.
• A random sample of 197 emails (where the probability of inclusion was higher for
those not classified as 'oppose') was manually coded.
• The LLM was refined until it produced results that closely matched the manual
coding of the 197 emails. Emails the LLM classified as Oppose, Support, or
Unclear had a very high probability of being classified the same way manually,
but those the LLM classified as partial support were classified manually as
partial support only about half the time, and as unclear the rest of the time.
• This version of the LLM was run on the full set of emails.
• Emails classified by the LLM as partial support in the previous step were
manually coded.
• Wherever an email had been manually coded, its manual classification was
used. In other cases, the LLM classification was used.
Coding of overall stance for CS submissions
• Emails were given preliminary classifications using a large language model (LLM)
following the logic of the classification flowchart.
• We manually coded the 567 subs the LLM categorised as something other than
oppose plus a sample of 100 the LLM coded as oppose. Coding was done by two
separate human taggers, with high levels of agreement.
• Fine tuned the LLM model to achieve a high match rate of tagging with the
manually tagged submissions.
• Match between manual and LLM coding was lower for those the LLM coded as
partial support or support than for those the LLM coded as oppose or unclear, so
we manually coded the rest of the submissions the LLM called partial support or
support.
• We modified the AI’s coding using several straightforward deterministic rules to
increase agreement between the AI’s coding and our manual coding.
• Used manual tags for the submissions they were made for, and AI tags for the
rest of the submissions.
Substantive submission data set
• Email submissions are included if the length of the email text or length of the
attachment text is at least 10,000 characters OR if the submission is on behalf of
an iwi/hapū, or organisation.
• CS submissions are included if the combined length of all free text responses
plus the length of any associated email submission is at least 10,000 characters
OR if the submission is on behalf of an iwi/hapū, or organisation.
