What Happens When an Approved AI Chatbot Serves a Meth Recipe?

"Three prompts. That is how long it took for the AI in every NZ emergency room to walk researchers through identity theft, bomb making and the recipe for methamphetamine. Health NZ's director of digital innovation and AI called it 'a minor issue.'

That is what people say right before the drawbridge comes up."

The AI was Heidi, a clinical scribe Health NZ approved in October 2025 for national rollout. Minister Simeon Brown described it as among the fastest rollouts in the world to move an AI from pilot to nationwide frontline use in emergency departments. By March 2026 it was in the hands of 1,250 doctors and frontline ED staff.

Five months later, a UK-based AI security firm called Mindgard decided to see how hard it would be to break Heidi. The answer was: not very. With three typed prompts — no technical expertise required — they got Heidi to rewrite its own guardrails. Once loose, it produced a step-by-step guide for a doctor to steal a patient's identity and a recipe for methamphetamine. Heidi Health said the vulnerability had been identified and patched internally before Mindgard's disclosure.

The point is that the public test came from researchers outside the system, not from anywhere inside the rollout.

This was not the only one.

In the same window, the Department of Corrections discovered probation staff had been pasting confidential offender information into free Microsoft Copilot to draft Extended Supervision Order reports. The policy explicitly forbade it. About thirty percent of Corrections staff had been using the tool since November 2025.

A few weeks later, Health NZ caught its own staff using free ChatGPT and Gemini to draft clinical notes. Disciplinary action was to follow.

Three incidents. Three months. Three different failure modes. One common thread.

In each case, AI had been deployed broadly before it had been proven small. Heidi to every emergency room. Copilot to a third of Corrections staff. Free consumer tools to clinicians, with no supervision and no boundary. Then the headlines. Then the panic. Then every other leader in the country pulls up the drawbridge.

The drawbridge is the part you need to understand.

When a public AI failure hits the news, it does not just damage the agency in the headline. It damages every leader in the country who was about to greenlight a similar rollout. Confidence in the technology drops. Boards start asking different questions. The AI initiative that was about to ship gets pushed back a quarter. Then another. The capability gap grows while the conversation cycles back to risk.

One bad headline can stall AI momentum across an entire organisation overnight.

The reason this keeps happening is not stupidity. The race is on. Vendors are pitching, competitors are deploying, and the perceived cost of being slow is high. So leaders approve faster than they would for any other system that touched customer, staff or patient data. They mistake the demo for proof. They mistake the rollout for adoption. They skip the part that builds confidence.

This is the gap that "we've got it handled" hides.

Vendors handle tools. IT handles access. Training handles knowledge. None of them handle judgement.

The only way to build judgement, without first becoming the cautionary tale, is to deploy AI the way you would deploy anything else load-bearing in your business.

A controlled pilot. Humans in the loop. One part of the organisation. The value proved. The risks mapped. The guardrails built.

What that looks like in practice:

• Start small. One workflow. One team.

• Keep humans in the loop.

• Measure the value.

• Map the risks.

• Build guardrails from reality, not vendor promises.

• Then scale.

This is the boring version of AI adoption. It is also the only version that does not eventually generate a headline.

AI governance is now behaving the way cybersecurity did fifteen years ago. Back then, the businesses that treated it as an IT problem stayed one incident behind for a decade. The ones that treated it as a leadership capability got ahead and stayed ahead. AI is in the same place today, except the failure cycle is faster and the public consequences are bigger.

A question worth asking this week, of yourself, before someone else asks it:

If a journalist called tomorrow and said your customer-facing AI had said something it should never have said, or that one of your staff had pasted client data into a free AI tool, would you know? Would you know which tool? Would you know what the policy says? Would you know if the policy is being followed?

If the answer to any of those is "I assume so," you do not have it handled. You have it hoped.

The leaders winning at AI right now are not the ones moving fastest. They are the ones moving with confidence, because they built it.

Pilots build confidence. Headlines destroy it.

Start here

In five minutes find out if your business has AI under control, or just hopes it does.

Take the 5-Minute AI Risk Snapshot

--

About the Author 

Mark Lucas is Co-founder at Generation AI, where we help New Zealand business leaders build real AI capability.

Information provided is general in nature and does not constitute legal advice