DEFINITION: What choosing a penetration testing provider actually involves
Choosing a penetration testing provider is a procurement decision assessed across four dimensions: verified tester competence, scope alignment with actual risk, finding delivery and remediation workflow, and commercial structure including what is excluded from the quoted price.
Most vendor evaluations fail because they assess only the fourth dimension. Price is the easiest to compare and the least predictive of outcome. Two providers quoting similar figures can deliver materially different value depending on who conducts the testing, what the scope covers, and whether findings arrive in a form an engineering team can act on.
This checklist is structured as questions to ask, with an explanation of what a strong answer sounds like and what a weak answer reveals.
Section 1: Verifying tester competence
Question: Is the firm CREST accredited, and can that be verified independently?
CREST is an international accreditation body that independently assesses penetration testing organisations and their individual testers against verified standards of methodology and competence. Accreditation is publicly verifiable through the CREST member directory.
A strong answer: Names the accreditation and invites verification. A weak answer: Describes testers as experienced or highly skilled without external validation, which is a self-assessment rather than an independent one.
Question: Who specifically will conduct the testing, and what are their qualifications?
Some providers sell using senior consultants and deliver using junior staff. A strong answer: Identifies the actual testing team and their certifications. A weak answer: Refers only to the firm's general capability.
Question: Does the provider carry professional indemnity insurance?
Testing carries a small but real risk of service disruption. A strong answer: Confirms coverage and its limits without hesitation, because any established provider has this documented.
Section 2: Scope alignment
Question: Does the proposed scope cover APIs and authorization logic, or primarily the web application front end?
For most modern SaaS products, the highest-risk surface is the APIs behind the product and the access controls governing who can reach which data. A scope focused mainly on the front end will miss the vulnerability classes most likely to cause a serious breach.
A strong answer: Discusses endpoints, authorization models, and user role boundaries specifically. A weak answer: Describes generic application testing.
Question: Does the methodology explicitly include business logic testing?
Business logic flaws, such as workflows that can be completed out of order or transactions that can be manipulated, are not detectable through pattern-based approaches. They must be named in the methodology.
A strong answer: Explains how testers develop an understanding of the application's intended behaviour before attempting to violate it.
Question: How is scope adjusted when the product changes?
A strong answer: Describes a defined process for bringing new systems into coverage. A weak answer: Implies scope is fixed at contract signing, which guarantees that anything built afterward is untested.
Section 3: Finding delivery and remediation
Question: When are findings delivered relative to when they are discovered?
A strong answer: Confirmed findings reach the engineering team as they are validated. A weak answer: Findings are compiled into a report delivered after testing concludes, which means the team learns about a critical issue weeks after a tester found it.
Question: Is every finding validated by a human before delivery?
Unvalidated output transfers triage cost to the customer. Engineering teams spend time investigating issues that are not genuinely exploitable.
A strong answer: Confirms human validation before any finding reaches the customer. A weak answer: Describes findings as surfaced directly from tooling without a validation step.
Question: Is retesting included, and how many retests?
This question separates a service from a transaction more reliably than any other. A finding is not resolved until a retest confirms the fix worked.
A strong answer: Retesting within the covered scope is included. A weak answer: Retesting is available at additional cost, which quietly creates a financial disincentive to verify fixes thoroughly.
Question: Can the engineering team ask a tester a question during the engagement?
A strong answer: Provides a direct channel with a defined response expectation. A weak answer: Routes questions through account management or defers them to a scheduled call.
Your Last Pentest Is Already Out of Date
Every week you ship without continuous testing is a week a vulnerability goes unseen. See what Capture The Bug finds in your first engagement.
Book a demo
Any provider under consideration should be able to answer every question in this checklist directly and without qualification. Book a demo with Capture The Bug and use this list as the agenda.
Section 4: Commercial structure
Question: What is explicitly excluded from the quoted price?
This is the most useful commercial question available, and it is rarely asked directly. Common exclusions include retesting, out-of-hours testing, additional systems discovered during the engagement, and report formats required for specific compliance frameworks.
A strong answer: States exclusions plainly. A weak answer: Is vague, which usually means exclusions exist and will surface as variations later.
Question: How is scope creep handled contractually?
A strong answer: Defines a change process with agreed rates before work begins. A weak answer: Defers the question, which is the mechanism through which quoted prices grow after testing starts.
Question: What compliance frameworks can the report evidence?
A strong answer: Confirms which frameworks the reporting structure addresses and whether findings are mapped to specific control expectations. A weak answer: Offers a generic report and leaves the mapping work to the customer.
A penetration testing service structured around continuous delivery generally answers the delivery, retesting, and scope adjustment questions in this checklist by design rather than by exception.
The three answers that should end an evaluation
Three responses indicate a provider is not suitable regardless of price.
- Accreditation that cannot be verified: If a claimed certification does not appear in the issuing body's public directory, the claim is unsupported.
- Refusal to identify the testing team: A provider unwilling to name who will conduct the work is either subcontracting without disclosure or staffing differently from how it sold.
- Findings delivered without human validation: This transfers the entire triage burden to the customer while charging for a testing service.
When a smaller scope is the right answer
An honest checklist should acknowledge that the most comprehensive option is not always correct.
An organisation with limited remediation capacity gains little from expanded testing coverage, because findings will accumulate faster than they can be addressed. In that situation, a narrower scope focused on the highest-risk systems, with capacity reserved for remediation, produces better security outcomes than broad coverage that generates a backlog.
Any provider that recommends the largest possible scope without first understanding remediation capacity is optimising for contract value rather than for the customer's security position. A penetration testing service worth engaging will ask about remediation capacity during scoping.
What this means for your roadmap
The determining factors in a penetration testing engagement are who conducts the testing, whether the scope covers the systems that actually carry risk, whether findings arrive in time and in a form the team can act on, and what the price excludes. Price comparison alone predicts none of these. Running the same structured checklist across every provider under consideration, including any penetration testing service already in use, produces a decision based on capability rather than on which proposal document was most persuasive.
Plan Your Annual Pentesting Strategy the Right Way
Learn how modern SaaS companies structure pentesting across the year to reduce risk, stay compliant, and avoid last-minute panic before audits.
FAQ
What should a CISO ask when choosing a penetration testing provider?
The essential questions cover four areas: whether the firm holds independently verifiable CREST accreditation and who specifically will conduct testing, whether the scope covers APIs and authorization logic rather than only the front end, when findings are delivered and whether they are human-validated, and what is explicitly excluded from the quoted price.
How can a company verify a provider's CREST accreditation?
CREST publishes a public member directory. Any firm claiming accreditation can be checked against it. A claim that cannot be confirmed through the issuing body's directory should be treated as unsupported.
Why does retesting matter when comparing providers?
A vulnerability is not resolved until a retest confirms the fix worked. When retesting is billed separately, it creates a financial disincentive to verify fixes thoroughly. Whether retesting is included is one of the largest differences in total cost and security outcome between providers.
What are the warning signs of an unsuitable penetration testing provider?
Three responses indicate unsuitability: accreditation that cannot be verified in the issuing body's directory, refusal to identify who will conduct the testing, and findings delivered without human validation, which transfers triage cost to the customer.
Is the most comprehensive testing scope always the best choice?
No. An organisation with limited capacity to remediate findings gains little from broad coverage, because findings accumulate faster than they can be addressed. A narrower scope covering the highest-risk systems, with capacity reserved for remediation, often produces better security outcomes.
