Are You Giving Out Cheat Codes if You Whitelist Pen Testers?

At some point during the scoping process with new clients, the subject of whitelisting comes up. Most of our clients choose to whitelist our IP addresses during engagements, although it may seem counterintuitive at first. You might be thinking, “Why would I ever whitelist your testers? We want to simulate how real attackers would target us and whitelisting would make that task unrealistically easy for you. No thank you.”

It’s an understandable perspective at first glance, but the choice to whitelist is actually a consequence of larger decisions you’ll make about the project, not a motivating factor.

We’ve jumped ahead by starting with whitelisting, so let’s take a step backwards to look at the larger motivation behind engaging with a third-party testing company like Bishop Fox. Our scoping process begins with asking broader questions to help inform the details of the upcoming assessment. In the big picture, the most important questions are:

• Why am I testing?
  • What am I most concerned about in my environment? What are the most likely avenues of attack? What’s the worst case scenario?
• How do I plan to use the results of the testing?
  • Do I have the capacity to implement changes? Will this test produce issues that I can remediate, or will it reveal third-party vulnerabilities that require changing the security products I use?

By stepping back to this larger scale, it becomes evident that the decision to whitelist (or not) is one variable that is affected by the high-level understanding of your project goals. The project priorities should be your guide, and whitelisting decisions are a later consequence of those goals.

So what does whitelisting mean in this context?

WHAT IS WHITELISTING?

In general, whitelisting (also called safelisting or allowlisting) is a strategy that designates an explicit list of data that is allowed access or permissions. (The list could be of anything - passwords, types of media, keywords, people’s names, IP addresses, etc.) The opposite of whitelisting is called blacklisting or blocklisting, which explicitly lists the items that should be denied access or permissions.

In pen testing, clients can choose to whitelist the IP addresses that the pen testers will be using to explore and access their network during the engagement, or they can leave them off the list so that the testers need to bypass external security controls like web application firewalls (WAFs) to explore a client’s internal surface.

WHAT ARE GOOD REASONS TO WHITELIST?

There are several good reasons that companies choose to whitelist. Let’s take a look at a few of them.

Warp Us to Level 99

Let’s say you’ve hired beta testers to find glitches in your video game. You could have them start a new game at Level 1 like a regular player would, but if you’re especially curious about a potential bug in Level 99 of your game, you’ll need to wait a lot longer to get the information you’re looking for if they start at the beginning.

Giving pen testers the ability to warp to the high-risk areas you care about is not cheating if that’s what you want to know the most about. Instead, you’re enabling them to spend more of their time on your priority areas, saving time and money. It’s true that they might learn something insightful by going through every level individually, but their work will take longer and more of their time will be spent away from your target area. Without whitelisting, the assessment becomes a test of the pen testers’ abilities, not your environment.

Avoid False Alarms

In another scenario, let’s say you’re interested in improving the security of your physical office building. You ask pen testers to come meet you to talk about a new project, but you don’t give them credentials to get inside. This choice will likely cause delays in your kickoff as the testers lockpick and socially engineer their way through doors that you could have given them creds for. This choice may also use up your security team’s time and energy on a fake threat, which means alerts and paperwork that divert your resources away from real threats.

Learn Weaknesses Beyond the Castle Walls

Another aspect to consider is that external security controls like WAFs have already been through QA with their own vendor’s security budget. The results of the pen test will therefore confirm their stated security posture, involve reconfiguring their default settings, or require you to choose a new WAF because of known vulnerabilities.

If pen testers instead focus on the vulnerabilities behind the WAFs, they will uncover issues that you have more complete control over to remediate – your apps, your credentials, your internal configurations. WAFs may seem like sturdy castle walls, but also may seem more like speed bumps to attackers during their exploit chain, meaning that greater focus should be spent on how resources are organized behind those walls. Prioritizing discoveries in the areas you value and (more importantly) have control over can vastly improve the usefulness of the remediation steps you receive at the end of testing.

WHAT DIFFERENTIATES ATTACKERS FROM PEN TESTERS?

Attackers have advantages that testers do not. Each side will inherently approach your network in a different way based on their abilities, resources, and goals. Whitelisting is one way to even the playing field a bit. Let’s look at some other factors that separate real attackers from hired pen testers.

Attacker Advantages

• No time limit for reconnaissance and planning.
• No rules of engagement.
• No limits about where and how they connect to your environment.
• If they find a vulnerability in a commonly used platform or app, they can attack multiple organizations simultaneously and then hone in on the most vulnerable targets.

Tester Advantages

• Shared team expertise and experience exploring similar environments.
• No inherent need to obfuscate attacks to avoid detection and legal consequences.
• Internal intel from clients about their goals, valuable assets, and internal structure.
• Opportunities for casual reconnaissance during remote or in-person kickoff meetings.
  • About applications and platforms in use, office floorplans, and company culture.
• The potential to work from whitelisted IP addresses to get around more quickly and easily (hint, hint).

WHAT ARE GOOD REASONS NOT TO WHITELIST PEN TESTERS?

Again, the answer to this goes back to what your goals are for the test. If the part of your environment that you’re most curious about, lack the most information about, or want to rigorously test is the external security controls, then not whitelisting your pen testers makes sense. The same is true if you most want to test your internal team’s response process to unknown threats, or to test policies you have on paper that you are not sure are being maintained or enforced. Even if you intend to test your security operation center without telling them, we’d still discuss that fact in scoping to prioritize that aspect and to know who to loop in if our work causes alarm. We’re trying to simulate an attack, not distract the security team from real threats. In many ways a pen testing team is like an EICAR file – a benign file that is recognized by antivirus as malicious. Pen testing is a way to test the structures in place without causing real destruction.

And What’s a Bad Reason Not to Whitelist?

Here’s one bad reason – choosing not to whitelist because your only goal is to pass compliance and you hope to slow us down with WAFs so we don’t find as much. Fewer identified issues means less to remediate, but hiding your head in the sand is not a security strategy. Aiming low to check the box for compliance means that our testing won’t help you better understand your attack surface or strengthen your security posture. It might feel like a quick win, but that lack of understanding muddies decision making down the line and exposes your company, product, and users to more unknown risks.

WHY NOT BOTH?

If you want to both test your external controls and learn about as many vulnerabilities as possible, you could also ask your pen testers to first test one way and then the other.

By whitelisting first, pen testers can find deeper internal vulnerabilities, and then add the layer of external security controls to learn if the issues are still exploitable. This also helps isolate the vulnerabilities caused by configuration issues in third-party products from the issues caused by permission or segmentation in your internal networks. If this split approach sounds useful, pen testers can “turn off” whitelisting on their side by attacking from different IP addresses to streamline the process.

Choosing both ways provides more answers to more questions as you compare the before and after of your perimeter security, but the extra effort will translate into a longer and more expensive engagement overall.

EVEN IF YOU DON'T WHITELIST

Even if you decide not to whitelist us, we’ll still provide you with a list during the beginning of engagements of the static IP addresses we’ll be using. This means that if your team does receive alerts from those locations, the issue will escalate to our point of contact, who will know not to waste time investigating them as real threats.

ASK GOOD QUESTIONS, GET GOOD ANSWERS

Pen testers are not attackers. They do recon, then invent and use tools to augment their work, but the difference between a real attack and a pen test is much more involved than just a matter of whitelisting. You can choose to whitelist or not, just make sure that choice propels you toward the answers you’re really looking for.

Let Your Goals Guide You

Ultimately, having specific goals for a pen test help determine how the choice to whitelist (or not) will affect those goals. Do you have tight budget and time constraints? Do you have a small, overworked security team? Keep your goals focused to get what you need without overwhelming your internal resources. Here are some reasons to decide one way or the other:

• To learn what an internal or external threat could do to your environment
• To learn how effective your legacy and third-party systems are against a dedicated attacker
• To learn about as many vulnerabilities as possible during the testing window
• To pass compliance

Don’t Worry, We’ll Figure This All Out Together During Scoping

Off the shelf, pre-packaged pen tests are no good for many reasons, but one is that they only check what the pen testing firm thinks clients will value for compliance, or will make them appear thorough. Not us. We dig into the details with every client during scoping to make sure our approach is relevant to their business needs, whatever they are. By clarifying goals at the start, we avoid overwhelming small businesses or under-serving companies with mature security programs. It’s about the balance of trade-offs: time, information, money. Each variable in the engagement has an impact, whitelisting is just one aspect of addressing the health of a client’s security.

To help pen testers find more vulnerabilities in a short window, you can give them inside info about sensitive areas, while attackers get unlimited time and can play dirty. In the end, no assessment is going to be 100% realistic, but being on the same page about goals can result in more findings that actually improve your security program.

So, whitelist or don’t. Whichever way you decide, just make sure you do it for the right reasons.