Most pentest vendor reviews never get said out loud. A contract ends quietly, the company moves on, and nobody writes down why. Capture The Bug decided to ask the question directly instead of guessing. Thirty CTOs at New Zealand SaaS and tech companies were asked one simple thing: why did you stop using your last pentest provider.
The answers were strikingly consistent. Almost none of them were about the vendor missing a real vulnerability. The complaints were about everything around the testing itself, the parts that never make it into a sales pitch.
The report took weeks and arrived stale
This was the single most common complaint, raised by more than half the group. A test would run, then nothing would happen for two to four weeks while the report got written. By the time it landed, the product had already moved on. New features had shipped. The exact version that got tested was no longer the version running in production.
One CTO put it simply: the report told them what was wrong with an app that did not exist anymore. That gap between testing and delivery is exactly what continuous testing is built to close, since findings show up as they are confirmed instead of arriving weeks later as a single document.
Retesting cost extra, every single time
Several CTOs brought up the same frustrating pattern. A vulnerability gets found, the team fixes it, and then confirming that the fix actually worked requires another invoice and another wait. What should have been a quick follow up check became its own mini project, with its own delay and its own line item.
This single issue pushed more than one company to look elsewhere, not because the original test was bad, but because verifying a fix should not feel like starting over.
Findings read like a generic checklist
A few CTOs described reports that felt copied from a template, long lists of low-risk items with little judgment about what actually mattered. Dozens of findings, almost none of them realistically exploitable, buried the two or three that genuinely needed attention. Engineers spent more time arguing about which findings were real risks than fixing the ones that were.
This complaint came up most often from companies whose products lean heavily on APIs. A generic web application scan style report often misses the specific logic flaws that matter most in an API, which is exactly why a focused api pentest service, one that understands how a product's endpoints are actually meant to behave, tends to produce a shorter, sharper list instead of a long one nobody trusts.
Your Last Pentest Is Already Out of Date
Every week you ship without continuous testing is a week a vulnerability goes unseen. See what Capture The Bug finds in your first engagement.
No one to talk to mid engagement
A handful of CTOs described a kickoff call, a few emails, and then silence until the final report arrived. When a question came up about a specific finding, there was no quick way to ask it. Everything had to wait for the next scheduled check in, if there even was one.
For a fast moving engineering team, that silence is often worse than a slow report. A finding that could have been clarified in five minutes instead sat unresolved for days, simply because nobody could reach a real person to ask.
Scope crept and so did the invoice
The last common thread was around cost. Several CTOs described starting with one quoted price, then watching it grow once testing actually began, as new systems got added to scope or extra hours got billed for clarification work that should have been included from the start. This is also where penetration testing cost in Australia and New Zealand gets misunderstood. A clear, fixed scope agreed before testing starts protects both sides from this exact problem. Founders evaluating penetration testing for startups should treat a vague, expandable quote as a warning sign, not a normal part of the process.
Book a demo
If any of this sounds familiar from a past engagement, the fastest way to see the difference is to look at one directly. Book a demo with Capture The Bug and see what a focused, plain language report actually looks like before committing to anything.
What this means for choosing the next one
None of these five complaints were really about whether a tester could find a vulnerability. They were about whether the engagement respected a team's time, budget, and need to actually act on what got found. That is a useful filter for evaluating any provider, current or future.
A few questions are worth asking before signing anything. How long between testing finishing and the report landing. Is retesting included or billed separately. Will findings be prioritized by real exploitability or just listed in full. Is there a way to ask a question mid test and get an answer the same day. And is the scope, and the price tied to it, locked in before work begins.
A penetration testing service built around continuous delivery rather than a single static report tends to answer most of these well by default, since findings appear as they are confirmed, retests are part of the same engagement rather than a new one, and a live dashboard replaces the radio silence between checkpoints.
What this means for your roadmap
Firing a pentest provider rarely means the testing itself failed. Most of the time, as this group of thirty CTOs showed clearly, it means the process around the testing failed to keep up with how fast the company needed to move. A slow report, a hidden retest fee, a noisy checklist, a silent inbox, and a quote that grew after the fact were the real reasons, not a missed vulnerability.
The next penetration testing service a company chooses is worth judging against all five of these, not just against a CREST certification on a website. The certification proves the testing is credible. These five answers determine whether the experience around it actually works.
None of this means a CTO has to take any vendor's word for it either. Asking a prospective penetration testing service to walk through exactly how they handle report timing, retests, and pricing before signing anything is a reasonable request, and a provider that answers clearly and specifically is usually the one worth trusting with the next engagement.
Plan Your Annual Pentesting Strategy the Right Way
Learn how modern SaaS companies structure pentesting across the year to reduce risk, stay compliant, and avoid last-minute panic before audits.
FAQ
Why do companies switch penetration testing providers so often?
Based on this group of thirty CTOs, the most common reasons were slow report delivery, extra charges for retesting, generic checklist style findings, poor communication during the engagement, and scope or cost creep after testing started. Very few switches happened because the original tester missed something real.
How much does penetration testing cost in Australia and New Zealand, and why does it sometimes grow mid engagement?
Cost depends on scope, but price growth mid engagement usually happens when scope was not clearly fixed before testing began. Agreeing on exactly what is being tested, and at what price, before work starts is the most reliable way to avoid this.
What should a startup ask before hiring a pentest provider?
At minimum, ask how long it takes to receive findings, whether retesting is included in the price, how findings get prioritized by actual risk rather than volume, and how questions get answered during the engagement itself rather than only at the end.
Is an API pentest service different from a general penetration test?
Yes, in practice. A general test often follows a broader checklist, while an API pentest service focuses specifically on how a product's endpoints are meant to behave, which tends to surface the logic flaws that matter most for API-driven SaaS products instead of a long list of low-priority items.
Does switching providers mean starting the testing relationship from zero?
Not necessarily. A new provider can usually review previous findings and pick up testing where the last engagement left off, especially if the goal is closing specific gaps like slow turnaround or unclear retesting rather than rebuilding the entire security program.
