New Zealand's Office of the Privacy Commissioner has emphasised that agencies remain responsible for decisions made using AI tools and should not assume reliance on automated systems displaces existing Privacy Act obligations. This includes periodically reviewing penetration testing summaries and security certifications for AI systems.
From 10 December 2026, Australia's Privacy Act requires disclosure in privacy policies when computer programs use personal information to make decisions significantly affecting individuals. New Zealand organisations that carry on business in Australia, or collect or hold personal information of Australians, may qualify as APP entities and be subject to these requirements regardless of where they are incorporated.
For NZ organisations with an Australian customer base or operations, the December 2026 deadline is not an Australian matter. It is a compliance obligation that requires knowing whether your AI system is doing what your privacy policy says it does. That answer requires testing.
What AI Red Teaming Is and Why Penetration Testing Cannot Answer the Same Question
Penetration testing answers the question "can someone get in?" and relies on a verification signal that a machine can produce: code executes or it does not. AI red teaming answers a different question: does the LLM system do what the organisation said it would do?
The practical distinction for NZ businesses: your AI system can pass a penetration test with no critical findings and still produce outputs that violate your own policy. A loan assessment AI that rates applications differently based on a demographic signal not in the training objective is not a penetration testing finding. It is an AI red team finding. A customer support AI that discloses information it should not disclose under adversarial prompting is not a network vulnerability. It is a model behaviour finding.
AI red teaming must assess intended behaviour as well as unintended behaviour. A traditional penetration test succeeds when it finds something the application was never supposed to do. AI red teaming must also assess whether the application correctly handles what it was designed to do: whether it stays within scope, whether it applies its guidelines consistently, whether edge cases in legitimate usage produce harmful outputs.
For how a standard penetration test covers the technical security of the infrastructure and APIs around your AI system, the guide to API penetration testing and securing the backbone of modern applications covers the technical security layer that AI red teaming does not replace.
When NZ Organisations Need AI Red Teaming
AI red teaming is appropriate when your product uses an AI model or agent to produce outputs that affect real decisions, real users, or real actions.
The specific triggers for NZ organisations in 2026:
1. Your AI system makes or substantially assists decisions about individuals
From 10 December 2026, Australian Privacy Act APP 1.7-1.9 requires disclosure of automated decision-making for organisations using AI in loan approvals, insurance pricing, job applications, or client risk assessments. Testing whether the system makes those decisions consistently with its stated logic is the evidence that supports that disclosure.
2. Your AI agent can take actions with real-world consequences
Any NZ SaaS product with an AI agent that has tool access needs to validate that its behaviour under adversarial prompting stays within its authorised scope. The guide to what Excessive Agency is and how to test for it covers the specific testing methodology for AI agents with real-world access.
3. Your AI product handles personal health, financial, or identity information
71% of organisations report AI connected to core business systems, and only 16% consider those connections effectively governed (2026 CISO AI Risk Report). For NZ healthtech and fintech, that governance gap is a Privacy Act compliance gap.
Does Your AI System Stay Within Policy Under Adversarial Pressure?
Validate decision-making integrity, harmful output rates, and prompt injection resilience with CREST-accredited AI security testing across NZ and Australia.
What AI Red Team Testing Actually Examines
AI red teaming tests the model or AI agent for harmful behaviour under adversarial prompting. AI penetration testing tests the technical system around the model for exploitable defects. Both are required for a fully tested AI product.
The specific areas an AI red team engagement covers:
Prompt injection through legitimate input paths. An AI customer support agent should not disclose other customers' data because a user inserted a manipulated input into a support ticket.
Scope adherence under edge cases. An AI loan assessment trained to ignore demographic signals should produce consistent results regardless of which demographic signals appear in adjacent fields.
Probabilistic Testing: Rating Findings by Frequency
The probabilistic nature of LLM testing requires running each test case multiple times and rating findings by frequency, not binary present or absent. An AI system that produces a harmful output five percent of the time is a vulnerability requiring remediation, even though 95% of interactions are safe. AI red team reports produce a versioned test suite alongside findings, not only a list of exploits.
Policy adherence under adversarial pressure. A NZ government AI tool that follows the Algorithm Charter for Aotearoa commitments under normal usage should follow them under adversarial usage too. Red teaming is becoming a de facto compliance requirement for high-risk AI in regulated industries globally.
Book a scoping consultation with Capture The Bug to confirm what AI red team testing your NZ product needs against the Privacy Act, OWASP LLM Top 10 2026, and your specific deployment context.
Frequently Asked Questions
What is AI red team testing and how is it different from penetration testing?
AI red team testing evaluates whether an AI system does what the organisation said it would do, including under adversarial prompting. Penetration testing evaluates whether an attacker can exploit a technical flaw in the system to gain unauthorised access. A system can pass a penetration test and still produce outputs that violate its own policy. AI red teaming is required to test model behaviour, policy adherence, and harmful output production under adversarial conditions. Both types of testing are required for a fully secured AI product.
When do NZ organisations need AI red team testing?
NZ organisations need AI red team testing when their product uses AI to make or substantially assist decisions about individuals, when an AI agent has real-world tool access, or when the AI product handles personal health, financial, or identity information. From December 10, 2026, Australia's Privacy Act requires disclosure of automated decision-making for organisations with an Australian nexus. NZ organisations with Australian customers or operations may qualify as APP entities. Red team testing provides the evidence that supports accurate automated decision-making disclosures.
Does the Australian Privacy Act December 2026 deadline apply to NZ organisations?
It can. From 10 December 2026, Australian Privacy Act APP 1.7-1.9 requires disclosure in privacy policies when automated systems use personal information to make decisions significantly affecting individuals. New Zealand organisations that carry on business in Australia, or collect or hold personal information of individuals located in Australia, may qualify as APP entities subject to these requirements. The OAIC has shifted to proactive enforcement since January 2026, and its reach is not limited to Australian-incorporated entities. NZ organisations with Australian operations should seek legal advice and test their AI systems before December 2026.
What does AI red team testing actually examine?
An AI red team engagement tests: prompt injection through legitimate input paths, scope adherence under edge cases, policy adherence under adversarial prompting, harmful output production rates (not binary present/absent but frequency across multiple test runs), agent behaviour under manipulated inputs, and whether the system correctly handles its authorised scope. Because LLM outputs are probabilistic, a system that produces a harmful output 5% of the time is a vulnerability requiring remediation even if 95% of interactions are safe. Testing must be run multiple times and findings rated by frequency.
Do I still need a regular penetration test if I run AI red team testing?
Yes. AI red team testing and penetration testing address different questions. A penetration test covers the technical infrastructure around the AI system: authentication, authorisation, API security, network controls, and data handling. An AI red team covers the model's behaviour under adversarial conditions. A complete AI security programme requires both. AI red teaming does not replace the penetration test on the APIs, databases, and services the AI system connects to. Both should be run before deployment of a new AI feature or agent, and after significant model updates.
