All organizations should perform penetration tests, yet many worry about not receiving the full value of their investment. Organizations have two choices: perform penetration tests with their internal teams, or hire an external vendor and find ways to lower costs.
Either case presents challenges, but to maximize the value of a penetration test, the organization must balance cost savings with quality. A low-quality penetration test can be an enormous waste of money that leaves the organization unaware of exploitable vulnerabilities.
Use the links below to jump ahead to a specific topic of interest:
- Penetration Testing Value and Return on Investment
- Self-Administered Tests: Challenges and Maximizing Value
- Third-Party Vendor Tests: Challenges and Maximizing Value
- Criteria and Requirements for a Pentest Quote
- Cost Cutting Cautions
- Bottom Line: Execute a Penetration Test to Obtain Maximum Value
Penetration Testing Value and Return On Investment
The value of a penetration test will be different for every organization and will be based on the calculation of the return on investment (ROI). ROI compares the cost of the benefit against the cost of the investment, or in the case of penetration testing, the exploitation cost divided by the test cost:
ROI = Expected Cost of Exploitation / Cost of Penetration Test
Here, the expected cost of exploitation would be the cost of data breaches, business operations disruptions, and IT system damages from any potentially successful attack on the systems tested. This number will be different from organization to organization and system to system.
Consider the wide variance in the potential costs for the following examples:
- An hour of disrupted internet for a grade school on vacation
- An hour of disrupted internet for Amazon on Black Monday
- A data breach on a steel plant’s time card database (does not include employee information)
- A data breach from a drug researcher’s laboratory computer
- Command and control access to the one computer in a dry cleaning retail store
- Command and control access to a laptop belonging to the VP of finance for a fortune 500 company
The more valuable the company resources, the more investment that should be made in penetration testing — even if those tests result in no discoveries. Scott Lyons, CEO of Red Lion notes, “The value of any test is less about if the pentester can get into the system and much more about if the system can prove its value to the business in resilience, ability to protect the system, or ability to provide alerts to security teams.”
“The value of any test is less about if the pentester can get into the system and much more about if the system can prove its value”
– Scott Lyons, CEO of Red Lion
An organization might be tempted to use in-house IT teams to perform penetration testing to save money. In some cases this will be reasonable, and when in-house expertise is available, it should certainly be done to reduce the vulnerabilities before hiring outside expertise.
Yet, outside teams bring new perspectives, different experience, and specialties that can uncover overlooked vulnerabilities. Outside vendors should be engaged periodically and may even be required under some compliance standards such as for PCI DSS.
Self-Administered Tests: Challenges and Maximizing Value
Organizations seek to use internal resources for penetration testing because they assume that the testing will be less expensive. However, the inherent challenges of self-administered tests conceal hidden costs to the organization that can only be offset by careful considerations.
Self-Administered Test Challenges
The primary challenges to self-administered penetration tests consist of hidden costs, internal expertise recruiting challenges, internal expertise myopia, penetration tool selection, and conflicts of interest. Unless an organization addresses these issues, penetration tests will likely be of lower quality than desired.
Hidden Costs of Self-Administered Tests
Managers often pursue self-administered penetration tests because the employee costs are considered to be “paid for” by salaries. When comparing against the costs of external penetration tests, these managers will use a zero for internal penetration testers because they have no additional expenses.
However, while valid for cash flow, the perspective is short sighted. The costs should at least include the allocated salaries of the internal teams and also include opportunity costs of projects that must be sidelined.
For example, conducting a penetration test might prevent the team from working on IT improvements or security improvements, or even monitoring and responding to security threats. These delays can increase the risk to the organization which increases the potential for breaches and other damages.
Internal Expertise Recruiting Challenges
A profound shortage of IT and cybersecurity expertise continues to increase recruitment and retention costs for all organizations. Unfortunately, this leads to most internal IT and security teams being composed of generalists because the increased costs to obtain expertise in a specific area, such as penetration testing, cannot be justified unless the expert will be using their expertise full time. When does it ever make sense to use a $300 per hour resource to do a $100 per hour job?
However, this also means that when the time comes to apply expertise, the generalists will need to perform more research to understand tools, take longer to set up and conduct tests, and may even pursue tests that don’t make sense because they don’t recognize it. The lack of expertise creates expensive inefficiencies that cannot be anticipated or even measured without comparison to an expert performing the same task.
Generalists will make more mistakes, be unable to recognize mistakes, and lack familiarity with compliance and other special needs. An organization will not recognize the hidden costs of mistakes and non-compliance until it is too late.
Internal Expertise Myopia
Even the largest organizations with the resources to hire penetration testing expertise will only be able to hire experts in a few areas. These experts will deeply understand only their focus, which will lead to penetration testing myopia — they really know what they know but may not be familiar with techniques or vulnerabilities outside of their expertise.
For example, a network and firewall penetration testing expert will be unlikely to also have expertise to test web applications for SQL injection, or to understand internet-of-things (IoT) firmware hacking. While it is perfectly acceptable for this expert to handle internal penetration testing on the corporate network, it would be unfair and unreasonable to hold this expert responsible for testing other types of infrastructure.
Penetration Tool Selection
Most organizations only have the resources to perform penetration tests periodically, so the cost of penetration tools will be spread out over a much smaller number of tests than a penetration testing firm. This may cause an organization to lean toward open source tools, but these tools tend to be more barebones and require more expertise or time to use.
Most internal penetration testing teams lack the time to master and deploy a full range of network and application penetration testing tools. An organization will need to pick a limited set of tools that tend to enable testing on a subset of possible tests.
Conflicts of Interest
The IT and DevOps staff created the environment, so will they be willing to honestly present that the IT resources or applications were deployed incorrectly? Will there be flaws hidden from management and secretly fixed?
Another type of conflict appears when managers apply pressure to cut tests short and check-box validate the results either because of impatience, operations concerns, or a lack of devotion to a secure posture. Both conflicts of interest tend to be hidden and difficult to detect.
Self-Administered Penetration Test Considerations to Maximize Value
For many organizations, the challenges of self-administered penetration tests can make outsourcing penetration testing less expensive and more efficient in the long run. However, for those organizations pursuing self-administered penetration testing, the value of the tests can be maximized by alignment of testing with resources and careful planning.
Align Testing with Resources
To align testing with resources an organization will realistically match their expectations with the budget, security staffing, penetration testing tools, etc. This will be a moving target particularly sensitive to budget and security staff workload. An organization must not force staff to perform beyond their capabilities or it will drive conflicts of interest and myopia.
Careful Planning
Careful planning involves splitting up or rescheduling penetration testing to overcome resource constraints. Even when pursuing testing, make sure sufficient resources are available to maintain key cybersecurity functions (incident response, monitoring) and to make progress with key maintenance projects.
The most obvious example would be peak operations. During peak operations, the organization cannot tolerate downtime and most staff will be fully consumed with supporting the delivery of the organization’s product or service.
Also read:
- 9 Best Penetration Testing Tools
- 7 Best Penetration Testing Service Providers
- 24 Top Open Source Penetration Testing Tools
- How to Implement a Penetration Testing Program in 10 Steps
Third-Party Vendor Tests: Challenges and Maximizing Value
Many organizations prefer to perform in-house penetration tests because of the challenges with third-party vendors. However, outsourcing to experts saves enormous time and overcomes IT expertise hiring constraints.
An organization can improve the value and reduce the cost of their penetration testing by executing IT fundamentals, adopting long-term contracts, prioritizing partial penetration tests, or performing rolling penetration tests.
Third-Party Vendor Test Challenges
When deciding to work with a third-party vendor, most organizations will be primarily concerned with confidentiality, cost uncertainty, and how to select a penetration testing vendor. Fortunately, to some degree, these issues present themselves with selecting any vendor (elevator maintenance, office cleaning, etc.), so purchasing can provide basic guidelines and the organization can focus on the penetration-testing-specific aspects of these issues.
Confidentiality
All organizations protect a variety of secrets such as proprietary technology, secret processes, personal identifiable information (PII), credit card information, and IT security details. Using a third-party penetration tester runs the risk of exposing those secrets through the hacking process.
While contracts provide a mechanism for legal consequences, the more important issue is trust. Organizations need to thoroughly investigate potential partners to verify a professional track record.
Cost Uncertainty
The cost of a penetration test can vary wildly. Although our research has determined an average penetration testing cost of $18,300, the range of pricing spans from several hundred dollars to $100,000.
The difference in pricing relies upon key factors such as the number of systems tested, the experience of the penetration testers, the type of systems tested, the type of penetration test, and more. Buyers should expect to have several rounds of communication with the potential penetration testing vendors to lock down the details needed for clear and predictable pricing.
How to Select a Penetration Testing Vendor
Buyers also stress about trying to evaluate the expertise of penetration testers without expertise of their own. Fortunately, buyers have very clear options for evaluation that do not require expertise.
References and penetration testing credentials provide an excellent starting point to evaluate a vendor’s history and qualifications. Buyers can then request proposals and sample reports to evaluate if these resources can be easily understood by both executives and the in-house cybersecurity team.
When trying to decide between a handful of highly qualified vendor candidates, the buyer can also pursue a test run where each vendor performs the same initial task. These evaluation options are explored in more detail in How Much Does Penetration Testing Cost? 11 Pricing Factors.
Third-Party Vendor Tests: Maximizing Value
An organization may be well prepared to limit costs through careful control of the factors that affect penetration testing costs and still have many opportunities to further reduce penetration testing costs. Some techniques cut the costs of a test, such as executing IT fundamentals or long-term contracts. Other techniques control costs or enable low cost penetration tests such as prioritizing partial penetration tests or executing rolling penetration tests.
Execute IT Fundamentals
Penetration testers are much more expensive than internal IT teams and therefore should not be used to perform basic vulnerability scans or asset discovery. Yet many organizations fail to execute these fundamentals and waste money overpaying hacking experts to provide basic asset discovery and vulnerability discovery reports.
To save money on a penetration test, an organization’s IT and security teams should ensure the completion of IT fundamentals such as:
- Regular asset discovery to capture new or rogue devices
- Maintained and current inventory of IT infrastructure components, devices, and applications
- Track changes to IT infrastructure components
- Regular vulnerability scans to detect known vulnerabilities and misconfigurations
- Effective patch and vulnerability management to eliminate or mitigate known security risks rapidly
- Regular security education regarding phishing, social engineering, and other attacks targeting the organization’s employees
Long-Term Contracts
Long-term contracts, sometimes called managed penetration testing services, offer a term contract for a certain number of tests or types of test for an extended time period. These contracts, often annual, can specify the specific penetration tests conducted or simply offer a number of tests to be performed per month. These contracts offer discounted rates in exchange for the committed spending.
An organization should definitely consider their long-term penetration pest needs. For example, compliance typically requires regular penetration tests, so a long-term contract can help control the costs.
Similarly, a growing organization frequently adds new network infrastructure, makes significant modifications to its applications, or establishes new offices. Each of these events would merit a penetration test, so a long-term contract with controlled pricing ensures the company will not delay needed penetration tests because of budgetary issues or lengthy contract negotiations.
Organizations with lower resources can obtain a different type of long-term contract that uses a retainer fee for a limited number of tests at a negotiated price over a fixed time period — for example, eight network tests over a one-year period with an option for more tests at listed prices. The company can expand testing as needed without the commitment for a large number of monthly tests.
Prioritizing Partial Penetration Tests
If an organization cannot afford a penetration test of the full environment, partial testing can be an option to reduce costs. However, organizations will then be faced with the decision of how to define the subset of the environment to test.
Priority should be based on the risk profile of the business. For organizations without an effective risk management program, the organization should focus on the most valuable resources that are the most likely to be targeted by malicious hackers.
Other factors that apply can be priorities to test:
- Publicly accessible IT infrastructure components
- Key security tools or controls protecting the most systems or protecting regulated or proprietary data
- Systems with known vulnerabilities to test the controls protecting them.
Rolling Penetration Tests
Instead of testing everything at once and incurring the huge bill, the organization can do rolling engagements of quarterly or monthly tests. This will be more easily applicable for testing discrete web applications than for complex websites and IT environments where the test scope can be more clearly defined.
However, when testing a more complex environment, a pentester can always reach a defined milestone in the testing (certain number of IP addresses, certain number of applications, etc.) and provide the organization with a list of untested components for a website and assets or network segments for a network. The organization can then determine what will be in scope for the current test and delay testing other components for the future.
Criteria and Requirements for a Pentest Quote
To ensure clear and reliable pricing, buyers should also prepare clear criteria and requirements for the penetration test. To start, buyers can consider the factors that affect penetration test costs within the control of the organization. Then prepare guidelines for the penetration test company to enable faster and more accurate quotes, particularly for the following factors:
- Scale and Scope: the size of the organization to test, especially the number of IP addresses t, network segments, applications, and people (for social engineering) in scope
- Penetration Test Type: black box, gray box, or white box penetration testing
- Compliance Requirements: any tests or reports required to meet compliance standards (PCI DSS, HIPAA, etc.)
- System Type: types of systems to be tested, such as endpoints, mobile devices, applications, networks, databases, cloud infrastructure, etc.
- Future Opportunities: disclose any needs for multi-month, multi-year penetration testing engagements
- Special Requirements: outline any needs for off-hour testing, physical security tests, social engineering, and similar special requirements
- Contract Type: disclose if the bid should be for a fixed cost bid or a time and materials bid
For larger engagements, a helpful aspect to consider is key performance indicators (KPI). An organization can define stages, KPIs, and intermediate status reports to check for progress and results during the penetration test. These intermediate status reports can determine if the testing should proceed or if the test should be paused so the organization can remediate problematic systems or make other accommodations.
For example, for a large ecommerce website, the penetration testing team might be asked to report any dangerous vulnerabilities before attempting to verify them through active exploitation. If the pentest team reports that they might be able to erase data in the transaction database, the company will likely want a chance to create a test environment for the penetration test team to test the exploit rather than risk the production database.
Cost-Cutting Cautions
Some cost cutting options, such as executing IT fundamentals, strengthen the security profile of the organization. However, performing partial or rolling penetration tests to cut costs narrows the scope of the penetration testing, which risks exposing the organization to unknown vulnerabilities hidden in the excluded IT systems.
For example, an oil refinery may decide to focus on the IT systems of the main office building containing all employee computers and servers and exclude any devices or network segments running to the refinery or pumping facilities of the plant. This exclusion may prevent the penetration testing team from discovering rogue computers and industrial internet of things (IIoT) devices such as wi-fi enabled pump controls installed within the facility.
Every organization needs to temper their cost cutting against the possible risks. These risks can be minimized by ensuring that penetration testing will eventually be performed within an appropriate time period (quarterly, annually, etc.) on all company systems and network segments.
Bottom Line: Execute a Penetration Test to Obtain Maximum Value
Although organizations might want to maximize their ROI from a penetration test, buyers must avoid delaying projects for incremental gains. In the end, execution of the penetration test delivers value and additional optimization will only leave unknown vulnerabilities exposed for exploitation by attackers.
Organizations need to move forward with the best option possible at the time the penetration test needs to be performed. During each test, the organization will gain improved security and experience that can be used to further refine and improve the ROI of future penetration tests.
For more reading on penetration testing, consider:
