Dear Ministry of Social Development,

I am concerned to learn that Work and Income is proposing to utilise Artificial Intelligence and automated decision-making systems in matters that directly affect the livelihoods of New Zealanders. This strikes me as a deeply concerning and dystopian direction to take. Decisions relating to a person's income, housing, and wellbeing should be made by accountable human beings, not outsourced to algorithms and systems that the public cannot meaningfully scrutinise.

While Ministers and politicians appear eager to embrace this technology, I am interested in understanding the extent of MSD's own expertise in this area, particularly regarding data privacy, security, and sovereignty. Many AI systems rely on infrastructure and providers located overseas, raising legitimate questions about where New Zealanders' data may be processed, stored, or accessed. Before the Government hands greater decision-making authority to automated systems, the public deserves to know whether those implementing them fully understand the risks and consequences, or whether this is simply another case of adopting the latest technological trend and hoping for the best.

Accordingly, I request answers to the following questions:

--- Data Storage & Sovereignty ---
Will any beneficiary data be stored outside New Zealand?
Will any beneficiary data be transmitted to servers located outside New Zealand?
Which countries could potentially host or process MSD beneficiary data?
Can the Government guarantee that no beneficiary information will be processed in foreign data centres?
If overseas processing occurs, what foreign privacy laws would apply to New Zealand citizens' data?
Could foreign governments legally compel access to data held in their jurisdictions?
Has the Government conducted a formal data sovereignty assessment?
Will all data remain under New Zealand legal jurisdiction at all times?
What contractual protections exist if overseas providers suffer a breach?
Has the Privacy Commissioner reviewed the international data implications?
Vendors & Commercial Relationships
Which companies are being considered to provide these automated decision-making systems?
Has any agreement already been signed with technology vendors?
Will foreign-owned corporations have access to MSD beneficiary data?
Will any data be used to train commercial AI models?
Can vendors retain copies of data after processing?
Can vendors use anonymised or aggregated beneficiary data for their own research?
Will vendor contracts be publicly released?
What penalties apply if a vendor mishandles beneficiary information?
What audit rights will New Zealand have over foreign technology providers?
Can subcontractors access beneficiary information?

--- Transparency & Accountability ---
If a benefit is reduced, suspended, or denied, how will a person know whether automation was involved?
Will every automated decision be clearly disclosed to affected individuals?
Will beneficiaries have a legal right to request a human review?
How quickly must a human review occur?
Can a beneficiary refuse automated assessment and request human assessment instead? - IMPORTANT
Who is legally responsible if an automated decision is wrong?
Is the Minister responsible????
Is MSD responsible?
Is the software vendor responsible?
Can an algorithm be cross-examined in court?
How can citizens challenge a decision if the reasoning is hidden inside proprietary software?
Will beneficiaries receive a full explanation of how a decision was reached?
Will the source code be available for independent scrutiny?
Will independent experts be allowed to audit the decision-making systems?
Will annual reports be published detailing errors and appeals?

--- Accuracy & Errors ---
What is the estimated error rate?
What level of error is considered acceptable when decisions affect access to food, housing and income?
What happens if the system incorrectly suspends a person's payments?
How many incorrect decisions would trigger suspension of the programme?
How will false positives be measured?
How will false negatives be measured?
What compensation exists for people harmed by incorrect decisions?
Has the Government modelled worst-case failure scenarios?
What safeguards exist to prevent a New Zealand version of Australia's Robodebt scandal?
Has the Government formally compared this proposal against the Robodebt inquiry findings?

--- Bias & Discrimination ---
How will MSD test for bias against Māori?
How will MSD test for bias against Pasifika communities?
How will MSD test for bias against disabled New Zealanders?
How will MSD test for bias against sole parents?
How often will discrimination testing occur?
Will bias testing results be publicly released?
If bias is discovered, who has authority to halt the system?
Will affected communities be consulted before implementation?
Has Te Puni Kōkiri been consulted?
Has the Human Rights Commission been consulted?
Security & Cybersecurity
Has a full cybersecurity risk assessment been completed?
What would happen if the system were compromised by hackers?
Has the Government modelled ransomware scenarios?
How quickly could payments be restored following a cyberattack?
Would a cyberattack potentially expose financial, health, disability or family information?
Has the National Cyber Security Centre reviewed the proposal?
Will penetration testing reports be released publicly?
How long will beneficiary data be retained?
Will data deletion policies be independently audited?

--- Democratic Process ---
Why is legislation of this significance being advanced under urgency?
Why was the select committee process bypassed?
Why were beneficiaries not consulted before introducing the legislation?
How many public submissions were received before drafting the bill?
What urgency justified limiting public scrutiny?
Why should New Zealanders trust a system that has not undergone full public consultation?

--- Human Impact ---
What evidence exists that vulnerable people actually want this change?
How will elderly beneficiaries navigate automated processes?
How will people with disabilities challenge automated decisions?
How will people without internet access interact with the system?
What protections exist for people experiencing mental distress?
How many frontline staff positions are expected to be reduced as a result of automation?
Is the primary objective better service, or cost reduction?
If a beneficiary loses housing, food security, or medical access because of an incorrect automated decision, who will be personally accountable?
If beneficiary data is discovered to being used by AI for learning models overseas, how can the New Zealand Government recall and delete that data upon a request?

---

From my experience with the Internet, once the footprint is out, it's out forever - if you have beneficiary data going overseas for AI systems - there's no putting the genie back in the bottle.

Yours faithfully,

James Welsh