Over the past two years, almost every SaaS product picked up some kind of AI feature. A chat assistant that answers customer questions. A summarizer that reads support tickets. A helper that drafts replies based on account history. Most of these shipped fast, because customers expect them now, and the teams building them were focused on making the feature work, not on what happens when someone tries to misuse it.
Capture The Bug set out to check exactly that. Fifty AI features, built by SaaS companies across New Zealand and Australia, were tested for one specific weakness: prompt injection. The companies are not named here, because the goal is not to single anyone out. It is to show what happens when a feature ships before anyone asks what it will do with text it was never meant to trust.
Forty five of the fifty leaked something they should not have.
What prompt injection actually is
Most AI features work by reading text and following instructions inside that text. The trouble starts when the AI cannot reliably tell the difference between instructions from the company that built it and instructions hidden inside content it is simply supposed to read, like a customer's message, a support ticket, or an uploaded file.
Picture a support assistant that summarizes incoming tickets for a staff member. If a ticket contains a line that looks like an instruction rather than a complaint, something telling the assistant to ignore its previous task and reveal information instead, a poorly built assistant may simply follow it. It does not know the difference between the company's instructions and a stranger's. It just sees text, and text is what it was trained to act on.
This is not a theoretical risk. It is the single most common way these fifty AI features failed.
What actually leaked
The leaks fell into a small number of repeated patterns across the test group.
- Cross account data exposure. In several cases, a crafted message convinced the assistant to reveal details from a different customer's account, including names, order details, or support history that had nothing to do with the conversation at hand.
- Internal instructions exposed. A number of assistants could be talked into revealing the internal guidance they had been given, the equivalent of a staff handbook nobody outside the company was meant to see. That alone hands an attacker a map of what the system is allowed to do.
- Unintended actions through connected systems. A handful of the AI features were wired into other tools, like ticketing systems or account settings. In those cases, a crafted instruction occasionally got the assistant to attempt an action, such as changing a setting or triggering a workflow, that the person typing it had no real authority to request.
None of these required deep technical skill. They required patience, a willingness to phrase a request in an unusual way, and a feature that had never been tested against exactly that kind of input.
Your Last Pentest Is Already Out of Date
Every week you ship without continuous testing is a week a vulnerability goes unseen. See what Capture The Bug finds in your first engagement.
Book a demo
If a chat assistant, a summarizer, or any AI feature in your product has never been tested this way, it is worth finding out what it would actually do with the wrong kind of message. Book a demo with Capture The Bug and see how a real test against your AI feature works, with findings explained in plain language.
Why this keeps slipping through
Most teams test an AI feature the way they test any feature: does it answer correctly, does it sound right, does it handle the obvious edge cases. That testing rarely includes someone deliberately trying to manipulate it, because that mindset belongs to security testing, not product testing, and the two often happen on completely separate timelines, if the second one happens at all.
It also slips through because a traditional pentest, scoped before the AI feature existed or written without it in mind, may never touch the feature specifically. The login page gets tested. The payment flow gets tested. The chat widget that quietly has access to every customer's account history sometimes does not, simply because nobody updated the scope to include it.
How this should actually be tested
An AI feature is, underneath the interface, just another part of the application talking to a backend through an API. Treating it that way is the most useful shift a team can make. A focused api pentest service aimed specifically at the endpoints behind an AI feature, the data it can access, and the systems it is connected to, finds these issues the same way it finds any other access control flaw: by trying to make the system do something it should refuse.
This is also good news for budget. Testing an AI feature properly does not usually require a separate, specialized engagement bolted on top of everything else. It fits naturally inside the same scoped penetration testing service a SaaS company already needs for its application and APIs. The feature just needs to be named in scope rather than left out of it.
This connects directly to how penetration testing cost in Australia and New Zealand actually works. A separate, AI-specific security audit can sound expensive and complicated, which is part of why so many companies skip it entirely. Folding the AI feature into an existing scoped test, the same kind already recommended for penetration testing for startups working through early growth and compliance requirements, usually costs far less than treating it as its own project, and it gets tested by the same people who already understand the rest of the product.
The honest comparison, as always, is not the cost of testing against zero. It is the cost of testing against the cost of a support assistant handing a stranger someone else's account details, which is a conversation no founder wants to have with a customer.
What this means for your roadmap
AI features are not going away, and customers have made it clear they expect them. What this test shows is that shipping one without checking how it handles a hostile message is no different from shipping a login form without checking what happens when someone tries the wrong password on purpose. Nine times out of ten in this exercise, nobody had checked.
The fix is not to slow down on building AI features. It is to make sure the same scrutiny applied to every other part of a product, the kind already built into a standard penetration testing service, gets applied here too, before a customer finds the gap by accident or someone with worse intentions finds it on purpose.
Plan Your Annual Pentesting Strategy the Right Way
Learn how modern SaaS companies structure pentesting across the year to reduce risk, stay compliant, and avoid last-minute panic before audits.
FAQ
What is prompt injection in simple terms?
It is when someone hides an instruction inside text an AI feature is meant to read, like a support message or uploaded file, and the AI follows that hidden instruction instead of simply processing the content normally. It happens because many AI features cannot reliably tell trusted instructions apart from text written by anyone else.
Why did 9 out of 10 AI features fail this kind of test?
Most teams test whether an AI feature works correctly, not whether it can be manipulated. Without a deliberate test for that specific weakness, gaps like this tend to stay invisible until someone, intentionally or not, stumbles onto them.
Can a normal penetration test catch prompt injection issues?
Only if the AI feature is explicitly included in scope. A pentest scoped before the feature existed, or written without it in mind, often misses it entirely. A proper API pentest service that names the AI feature's endpoints and data access as part of the scope will test for exactly this.
Does fixing this require rebuilding the AI feature from scratch?
Usually not. Most fixes involve tightening how the system separates trusted instructions from untrusted input and limiting what data or actions the AI feature can reach in the first place. It is a scoping and access control problem more than a full rebuild.
Is testing an AI feature expensive compared to a normal pentest?
Not when it is folded into an existing scoped engagement rather than treated as a separate project. Naming the AI feature's endpoints in the scope of a standard penetration testing service usually costs far less than a standalone AI security audit, and it gets tested alongside everything else.
