Only 48% of all vulnerabilities identified in penetration tests are ever resolved. For Australian organisations, that number has a compliance consequence that compounds it: an unresolved finding from last year's test is both a security risk and a gap in the evidence trail your APRA supervisor, Essential Eight assessor, or SOC 2 auditor will check this year.
What your organisation does before, during, and after the engagement determines whether the investment produces compliance evidence and closed vulnerabilities, or a PDF that sits in a shared drive. This checklist covers the three stages where AU organisations most commonly lose that value.
Before the Engagement: Scope and Preparation
The scope document is the most important artefact your organisation produces in the process. A scope document that clearly lists in-scope and out-of-scope targets, an approved testing window, an incident response contact list with phone numbers, and a data classification and confidentiality agreement creates the accountability that prevents misunderstandings from derailing an engagement.
Before any engagement starts, confirm these items are in place:
- Asset inventory is current and signed off. The tester can only test what is in scope. Walk through the inventory with the provider before signing the statement of work, not after the report arrives.
- Rules of engagement are documented and signed. Rules of engagement cover testing hours, systems that require care (production databases, payment processors), notification procedures if the tester identifies active exploitation by a third party, and emergency stop procedures. APRA CPS 234 requires careful exclusion documentation for systems where downtime is not acceptable, with a business justification for each exclusion.
- CREST certification of the provider is verified before engagement, not assumed. The guide to how CREST penetration testing works in Australia covers how to verify CREST International and CREST ANZ status separately, because they are two different bodies and both registries must be checked directly.
- Compliance framework is mapped before testing begins. If the engagement needs to produce APRA CPS 234, Essential Eight, ISO 27001, PCI DSS, SOC 2, or IRAP evidence, confirm with the provider that the report format maps findings to those frameworks. Not every AU provider produces compliance-mapped reports as their default output.
- Existing findings from the prior engagement are reviewed. A penetration test conducted against an environment with unpatched high-severity findings from the previous test will find those findings again. Reviewing and remediating prior findings before the new engagement makes the new test more productive and prevents the report from repeating the same finding list.
During the Engagement: What Your Team Should Do
Many AU organisations treat the testing window as a passive period. The tester is working. The internal team is waiting. This approach wastes the most valuable operational intelligence the engagement can produce.
- Do not suppress all IDS and IPS alerts during testing. If you suppress all alerts during testing, you will not learn whether your detection capability would catch a real attacker. A better approach is to log but not block test traffic, then review detection coverage as part of the engagement findings.
- Maintain a point of contact who is reachable during testing hours. A tester who finds an active third-party compromise needs to be able to notify your team immediately. An emergency contact list that rings to voicemail does not satisfy this requirement.
- Track the testing window against your change freeze calendar. Infrastructure changes made during a live penetration test can invalidate findings or create new exposure. Freeze changes to in-scope systems for the testing duration.
Your Last Pentest Is Already Out of Date
Every week you ship without continuous testing is a week a vulnerability goes unseen. See what Capture The Bug finds in your first engagement.
After the Engagement: The 48% Problem
This is where the majority of AU organisations lose the value of their penetration test. The penetration test itself is rarely the problem. The methodology is sound. The testers are competent. The report is thorough. And then nothing happens.
A structured post-engagement process closes the gap between findings and resolution.
- Assign each finding to an owner with a remediation deadline. A finding list without assigned ownership is a list that sits in a queue. Every Critical and High finding should have a named owner and a deadline before the report is distributed internally.
- Document the remediation evidence for each finding. The evidence matters as much as the remediation itself for compliance purposes. Remediation evidence should include patch notes, configuration changes, or code commits, and the retest must confirm fix effectiveness, not just that the vulnerable service was taken offline.
- Request and archive the retest report separately from the initial report. An APRA supervisor or Essential Eight assessor reviewing your testing evidence wants to see that findings were retested and confirmed closed, not just that the test was conducted. The retest report is a separate compliance artefact with its own archival requirement.
For how the remediation and re-testing cycle produces the specific evidence regulators and auditors check, the guide to what happens after a penetration test covers the exact steps that convert a finding list into confirmed compliance evidence.
Map closed findings to the compliance framework before the next cycle. APRA CPS 234 and ISO 27001 both expect the testing programme to demonstrate improvement over time. A finding that appeared in last year's report and was remediated this year is evidence of a functioning security programme. A finding that appeared in last year's report and appears again is evidence of a remediation gap that an assessor will note.
For organisations approaching their next engagement or setting their annual testing schedule, a scoping consultation with Capture The Bug covers how to structure each engagement for maximum compliance evidence and remediation closure within the testing cycle.
Plan Your Annual Pentesting Strategy the Right Way
Learn how modern SaaS companies structure pentesting across the year to reduce risk, stay compliant, and avoid last-minute panic before audits.
Frequently Asked Questions
What should an Australian business prepare before a penetration test?
Before a penetration test, an AU business should have: a current, signed-off asset inventory, a scope document listing in-scope and out-of-scope targets, rules of engagement covering testing hours and emergency procedures, verified CREST certification of the provider, the compliance framework the report must map to (APRA CPS 234, Essential Eight, ISO 27001, PCI DSS, SOC 2, or IRAP), and a review of unpatched findings from the prior engagement. The scope document is the most important artefact the organisation produces in the process.
Why do only 48% of Australian penetration test findings get resolved?
The most common reasons are: findings are not assigned to an owner with a deadline, remediation is assumed rather than verified, retest evidence is not documented separately, and the compliance connection between findings and the organisation's regulatory framework is not made explicit. A structured post-engagement process that assigns each finding to a named owner, documents remediation evidence, and archives the retest report separately is the mechanism that closes the gap between the finding and the resolved control.
What is a rules of engagement document for a penetration test?
A rules of engagement document covers: approved testing hours and dates, systems that require special care (production databases, payment processors), notification procedures if the tester identifies active exploitation by a third party, emergency stop procedures, the incident response contact list with direct phone numbers for testing hours, and exclusion documentation for systems where downtime is not acceptable. For APRA CPS 234-regulated entities, system exclusions require a business justification and must be documented. The document should be signed by the CISO or equivalent before engagement starts.
What evidence should an Australian business archive from a penetration test?
For APRA, Essential Eight, ISO 27001, and SOC 2 purposes, archive: the scope document and rules of engagement, the initial penetration test report with findings, severity ratings, proof-of-concept evidence, and remediation guidance, the remediation evidence for each finding (patch notes, configuration changes, or code commits), and the retest report separately confirming that each finding was retested and the fix confirmed effective. The retest report is a separate compliance artefact from the initial report and must be archived separately.
How often should Australian businesses conduct penetration testing?
APRA CPS 234 requires systematic testing calibrated to threat exposure, implying at minimum annually and after material changes. The Essential Eight at Maturity Level 2 and above requires annual testing at minimum. ISO 27001 requires regular testing as part of the risk management cycle. PCI DSS requires annual penetration testing and testing after significant infrastructure or application changes. SOC 2 auditors expect at least annual testing. For AU businesses with active development or infrastructure change cycles, targeted testing after major releases or changes supplements the annual engagement.
