Are you paying too much for your Penetration Test? By, James DeLuccia IV

As a security professional I have spent many an evening evaluating a client’s defenses using myriad of attack vectors and tools. Over time tools have replaced much of the art and the work from the effort. The only component really missing from the field of common knowledge was the craftsmanship in bringing these tools together to provide a thorough and accurate test. I would submit that this final component has been addressed through the brilliant development and documentation provided by the folks at VulnerabilityAssessment.co.uk.

They have published a template that includes nearly every step, tool, configuration, command line, execution instructions, and report formating necessary to deliver a very reasonable penetration engagement. They have the report available via HTML and PDF. I strongly recommend visiting their site, downloading the materials, and comparing their template with your most recent penetration test. As someone in the industry, you will find incredible similarities and a great Thank You should escape your lips as you marvel at the thoroughness. The reason this is so valuable is that conducting “standard” security assessments upwards to penetration assessments requires a very natural path that includes discovery and expoloitation. Of course, the specific vulnerability discovered and the exploit code used will vary, but not so much the tools anymore and certainly not the result. The end result is a checklist that provides a very precise approach that can allow a junior security professional to evaluate any wired enterprise.

Does this excuse the need for a penetration test? NO. It does however, place a bit of responsibility on those who require a penetration test and those who are paying for such tests. The only reason to pay for these types of engagements are two fold. One, the independent 3rd party set of eyes is not as sensitive as those running the network and WILL find items that were considered “business as usual”. This external tester is not bounded by the company culture and the societal side of the organization, and will present information that may have not been reported due to allegiances in the organization. Two, a checklist (even as good as this one) is not an adequate replacement for a truly talented security penetration professional. This checklist only raises the bar, and shines a bright light on what is typically considered a black-box product (money in – report out).

So, are you paying too much for your penetration test? Is your provider merely following these steps and charging you full bill rate for something that has become automated, but charge based on the manual effort it used to take? These are important questions, because we are talking about an evaluation that is intended to rigorously check our environments to proactively eliminate threats. Of course, there is the opposite truth that states you receive what you pay for in the world, and demanding pennies for a service that a senior professional who has spent at least 10 years honing their craft is unlikely to deliver the quality you are seeking.

How can you get a better value on your NEXT penetration test?

• First, review the last penetration report and find the methodology. Compare this methodology to that provided in the link above.

• You want to see that your provider’s methodology is more robust with more tools and more checks. For all gaps – highlight them, and skim the report to see if they were done in addition, but separate from the methodology.

• Well, if you find your report is lacking chalk it up to a learning experience – no sense crying over spilled milk (that is unless you haven’t paid said “service provider”, and haven’t officially accepted the work –> then you can get more accurate information on the work they conducted)

• Ideally, your report will have at least as much as listed in the checklist with perhaps some good substitutes to the tools and steps implemented. Remember we are not trying to throw darts in their hard work, but instead trying to ensure that everyone comes out better (they get paid for good work, and you get an accurate risk score).

• Now you want to check who did the work on your report. This is sometimes an overlooked item, and probably THE most critical component. YOU must require the CV / Resume of the individual actually conducting the penetration test from your provider. Get this in writing – i.e. Rocco or Vinny (my dogs – who by the way are not available for consulting work) will conduct the penetration test for X client. The BIO of this individual should be indicative of someone passionate about the technology hosted in your environment. For instance – if you have a complete Microsoft (soup to nuts) implementation, a CV showing strong Unix and Java is worthless, but someone with .NET, Microsoft training experience, and membership with groups that cater towards this type of work is preferred. Essentially you want to vet whomever is doing this work, and this should comply with the rigorous standards you place on anyone touching (or hopefully not touching) your critical applications. The Institute of Internal Auditors (IIA) has a great listing of how you evaluate individuals, but simply – check they do not have criminal backgrounds, are in good standing with peer groups, are current on their security clearances and certifications, and are vouched for by the company.

• Albeit checking vendors and the service provider personnel is usually done within the Vendor-Approval process, but it is key to consider with these types of engagements, because you know what to look for and they will normally submit their most senior persons for review by the vendor board. Check – Check – and Re-Check, these penetration assessments are VERY important and you should embrace them (both to secure your company’s intellectual property, your customer’s and your own personal sensitive information – unless you want a letter about identity theft, and protecting the jobs of everyone in the organization – too many companies to list that have gone underwater as a result of a penetration), but do so with confidence.

• As an educated consumer, you now realize that the majority of the work provided on the checklist and potentially 60% of the service provider’s methodology that is to be conducted can be done through automation, and should not cost the rate of the senior level practitioner you have approved. This is where you, the client, requests greater value. Have the practitioner focus their custom scripting and manual checks on the key systems that matter to your organization (i.e. those that represent ingress points, are publicly available, are part of your remote-office architecture, and, of course, your 3rd party outsourced operations). This portion of the effort is only possible if they have open attack vectors (meaning – they are your web server, and you can open a web browser to the server). Said another way – the practitioner can only use their incredible talents if there is something vulnerable, and therefore you only pay for what they actually discover and penetrate. Consider paying a low fee for the discovery, and then allow them to expand it as able, and bill for the normal hourly rate. This places an incentive to the vendor to provide the practitioner enough time. It gives the practitioner a real challenge – not just a fill in the box, but an honest challenge to demonstrate his/her talents. Best of all, you get a very customized product that provides a more realistic measurement of the risk profile.

The above may be coined by service providers as a “Whitebox or Blackbox” service. The simple explanation is that a Blackbox service is conducted with zero to minimal information on the attack targets. This is great if you have secret data centers and you want to see if they can be discovered and possible attacked (such as those found in your remote office of one in Auburn Indiana). A Whitebox (the preferred method and in line with the theme of this article) engagement has the client providing the asset information and even the type of services. This provides the service provider with sufficient information to skip the basic discovery techniques and move straight into the assessment portion of their methodology.

A final point when commissioning a penetration test – these take considerable effort (as listed in the check list) + the custom script development and execution, and therefore tend to cost nearly double a vulnerability assessment (i.e. run nessus hit print). The common response to selecting the sample of devices is to choose those that are most valuable, based upon a risk profile. [Here is where I throw a stone at the "business as usual" concept in security where penetration tests should be locked into the critical assets. My reasoning on this is as follows - you are paying someone to attack your most important assets. (These are probably your most watched, patched, secured, and backed up devices in the entire environment.) The likelihood that these are vulnerable is significantly low. That doesn't mean a risk based approach to conducting a penetration assessment isn't valuable - it is extremely valuable, if done appropriately and for the right reasons] Unfortunately as these systems are interconnected, the simplest appliance (Linksys Router) in a remote office may open the same data, as the main firewall / web server. Therefore, it is even more important to have a full penetration assessment on all digital-ingress points using the criteria discussed above. If this is not possible, then request a vulnerability assessment on every public device, and then pull those vulnerable from the list for the penetration test. In this manner the penetration test is really testing your LAYERED security. Consider this concept – if you had a vulnerability assessment and patched everything, and then had a penetration test what would happen? If the scoped devices were those that were found vulnerable were patched the vendor would conduct another vulnerability scan to determine if there were any NEW openings, and then try and exploit those. If they are able to then they will demonstrate how they can hop into your environment via these exposed systems. What is the point? What are you looking to determine? If it is to validate thoroughly that systems are secure and the architecture supports this fact – have a penetration test based on the payment schedule mentioned. Otherwise, consider a different service that addresses your needs.

Tests can be conducted internally also – which are far quicker and better for the client looking to secure all devices. The reason is you bypass the security devices and test at the host level. If the hosts are secure then the security devices are adding a layer of confidence, but if not then the host will most probably be vulnerable from the external segment (especially if we are considering this a patching issue). Yes, to those screaming, it does negate the firewall (add twenty acronyms of additional technology) from stopping everything, but then again isn’t it simpler to secure the single device correctly, and then layer the security to limit breaches (as would be validated in a penetration-beach head building engagement)?

Bottom line – consider the reason for the penetration / assessment, and be sure you are getting these needs addressed. I personally know many talented security professionals that use automated tools for about 10% of their penetration engagements and create on-the-fly code to exploit systems. These are the individuals you want conducting your engagements, and I want you to have them too. Avoid cheap imitations, limit risk, and demand quality.

Best regards,

James DeLuccia IV
A special thanks to my great friend Clement for his kind review of this post – his site contain vast expertise on both Security testing and the leading site for CISSP preparation.

The CISSP and SSCP Open Study Guides Web Site http://www.cccure.org
The Professional Security Testers Warehouse http://www.professionalsecuritytesters.org