IAM Vulnerable - Assessing the AWS Assessment Tools

In my previous post, I introduced IAM Vulnerable, walked through how to set it up in a playground AWS account, and demonstrated how to practice exploiting the types of privilege escalation vulnerabilities identified by Spencer Gietzen and demonstrated by Gerben Kleijn.

In this post, I talk about the identification aspect of IAM privilege escalation within a target account. I share my favorite IAM privesc assessment tools and conclude that while each of these tools is invaluable to a penetration tester, none of them can identify all known privesc paths while at the same time removing all false positives.

To help you understand why I make that claim, I take a deep dive into areas each tool would need to support on its path to perfection and show how our favorite tools stack up against those difficult challenges. To support this research, I’ve included some false positive and false negative test cases in IAM Vulnerable.

The art of identification

You can’t exploit what you can’t identify. Also, our goal as penetration testers and security practitioners is to identify ALL of the unintended privesc paths, not just some of them.

The good news is that there are some amazing open source tools that will help you! Here are some of the tools I find particularly helpful for tracking down vulnerable configurations:

• Principal Mapper (PMapper)
• AWSPX
• Pacu (iam__privesc_scan module)
• Cloudsplaining

The tools listed above are each very good at finding IAM misconfigurations, and they are the four tools I cover in the deep dive later in this post. The elephant in the room is that any tool that tries to help us with this task must understand and attempt to recreate AWS’s IAM authorization logic. This is difficult, and in some edge cases, it’s an impossible task.

As an operator, you need to know the strengths, and more importantly, the limitations of each tool you use. In the next section, we’ll talk about some of the specific reasons it is so hard to identify all of the potential privesc paths.

Why is this so hard?

The way AWS makes IAM authorization decisions is complex. There are multiple layers of authorization in place. For example, the success or failure of a certain privilege escalation path might depend on policy-defined resource restrictions or conditional statements that make it valid only in certain cases. In addition to per-account IAM policies, a privilege escalation path might also depend on factors like service control policies (SCPs) that are applied at an organization level, or resource-based policies that are attached to resources and evaluate on individual resources.

The following chart from the AWS article linked above provides details about how these IAM decisions are made:

Brigid Johnson’s talk Become an IAM Policy Master in 60 Minutes or Less is an excellent primer on this topic.

Deep dive into IAM privesc tool capabilities

First, let’s outline some of the areas that an AWS IAM assessment tool needs to consider in order to capture all valid exploitation paths while preventing false positives. Here’s the list of the high-level capabilities we’ll cover in this deep dive:

• Does the tool analyze both users and roles, or just one or the other?
• Does the tool take a principal-centric or policy-centric approach?
• Does the tool handle resource constraints?
• Does the tool consider the permissions of service roles?
• Does the tool handle transitive privesc paths (i.e., attack chains)?
• Does the tool handle the DENY effect as expected?
• Does the tool handle NotAction as expected?
• Does the tool handle Condition constraints?
• Does the tool handle service control policy (SCP) restrictions?

The tables below represent a point-in-time assessment of where the tools stand at the time of this blog’s initial release (September 2021). This analysis will become outdated as soon as the tools improve their detection capabilities in a particular area. That said, the feedback I’ve gotten from my Bishop Fox colleagues is that this analysis is helpful, even if only as a single point-in-time assessment that will soon be outdated. If nothing else, it illustrates how important it is to test the tools you use to make sure they work as you expect.

Capability Summary (September 2021):

In the sections below, I further define each of the capabilities, provide an example detection scenario, and list any relevant IAM Vulnerable resources for testing each capability.

Does the tool analyze both users and roles, or just one or the other?

An IAM user is an entity that can authenticate and interact with AWS resources. Users authenticate with a username and credentials (console password or CLI access keys). An IAM role can also interact with AWS resources, but it does not need to be uniquely associated with one person. A role can be assumed by one or more principals. You can access a role through SSO, via another role, or via an IAM user.

You’ll want to make sure that at least one tool you run is capable of identifying privesc paths that originate with a user as well as those that originate with a role. For example, one of the first tools created to identify AWS IAM privesc, AWS_escalate.py, only checks users, not roles. If you use that tool today, you’ll be missing important privesc paths. The good news here is that all tools we evaluate in this post address roles.

Detection Example:

Will the tool alert you if a user can privesc via iam:CreatePolicyVersion? How about a role with the same permission? If the tool does not detect both, then there is a false negative.
IAM Vulnerable Test Cases:

All — Every privesc path has a user and a role associated with it.

Does the tool take a principal-centric or policy-centric approach?

A principal-centric approach would identify any principals that have the permissions required for successful privilege escalation, even if the required permissions come from two or more IAM policies. Meanwhile, a policy-centric approach examines only one policy at a time, which causes missed detections (false negatives).

You’ll want to make sure that you have a tool that is capable of taking a principal-centric approach to identifying privesc paths.

Detection Example:

Will the tool alert you if a principal gets iam:PassRole from one policy and ec2:RunInstances from a second policy? If not, it is a false negative.

IAM Vulnerable Test Cases:

False negative (exploitable) – Privesc through multiple policies

False positive (non-exploitable) – Allow and Deny through multiple policies

Does the tool handle resource constraints?

Every IAM policy has a resource element that defines what object or objects the policy applies to. IAM policies that follow the principle of least privilege make heavy use of the resource element to limit the effect of granted permissions.

You want your tools to examine not only the permission itself, but also the resource section of the policy. AWSPX and PMapper do an amazing job at this.

Detection Example:

What if you have the permission to iam:CreatePolicyVersion, but you can only perform that action on a policy that isn’t applied to you directly, or transitively? Or, what if you only can perform iam:CreatePolicyVersion on a policy that doesn’t exist? While you have a high-powered permission in both examples, the resource constraint has limited where you can apply that permission, which prevents privesc. If the tool does not properly handle the resource constraints in policies and tells you that you can privesc, it would be a false positive.

IAM Vulnerable Test Cases:

False negative (exploitable) – Resource constraint

False positive (non-exploitable) – Resource constraint

Does the tool consider the permissions of service roles?

A role can only be passed to a service if the role trusts that service via the role’s trust policy. For example, if you can create a Lambda instance and want to pass a role to it, the only roles you can pass are ones that already trust lambda.amazonaws.com. If there are no roles that trust the Lambda service, then your privesc path is blocked even if you have iam:PassRole.

PMapper and AWSPX both do an excellent job of handling this, and this is really what sets them apart from the rest from the perspective of a penetration tester. If the tool does not handle this for you, you’ll have to manually look at what roles trust the service you’d like to exploit.

Detection Example:

Even if you have iam:PassRole and ec2:RunInstances, and even if there are some instance profiles, if there are no high-privileged roles assumable by the target service, there is no privesc path. Does the tool alert on the non-exploitable path? If so, it is a false positive.

IAM Vulnerable Test Case:

False negative (exploitable) – Role that can be assumed by multiple services

Does the tool handle transitive privesc paths (i.e., attack chains)?

A privilege escalation path is considered to be transitive if it requires more than one step. Usually, this involves a user or role first assuming a target role and then using the permissions assigned to that role to escalate their permissions.

You’ll want to make sure your tool handles transitive privesc paths. This might seem like a small issue, but in complex AWS environments, some of the paths to administrator take two or more hops. As you can see in the table above, AWSPX and Principal Mapper are currently the only tools that can identify these transitive privesc paths.

Detection Example:

What if the role test123 is not an admin yet but can privesc to admin? What if users Alice and Bob can assume the role test123? Does the tool only tell you that role test123 can privesc and omit the fact that this also means Alice and Bob can privesc to admin as well? That’s something you’ll want to know.

IAM Vulnerable Test Case:

False negative (exploitable) – AssumeRole chain

Does the tool handle the DENY effect as expected?

Every IAM policy has an effect element, which defines whether the listed permissions should be allowed or denied. If the effect in a policy statement is DENY, the permissions listed in that statement are blacklisted, rather than whitelisted. AWS gives the DENY effect precedence when evaluating policies, as shown in the flowchart earlier in this post.

As shown in the table above, AWSPX has chosen to ignore DENY effects altogether at this point, and Cloudsplaining has the logic to handle DENY effects but will result in a false positive if a second policy is doing the DENY effect because it only takes a policy-centric approach.

Detection Example:

What if you have a role with a policy applied that says you can do something, but then it also it also says you can’t do it? As outlined in the flowchart, the DENY effect gets evaluated first, so even though there is an ALLOW, the privesc path is not possible. If the tool does not properly handle the DENY effect in policies and says you that you can escalate privileges in this situation, it would be a false positive.

IAM Vulnerable Test Cases:

False positive (non-exploitable) – Allow and deny via same policy

False positive (non-exploitable) – Allow and Deny via multiple policies

Does the tool handle the NotAction effect as expected?

Instead of defining what actions are allowed, NotAction defines what is not allowed. The problem is, every other action is therefore allowed. In other words, NotAction changes the policy statement from a whitelist to a blacklist, which often leads to unintended privilege escalation.

As every penetration tester knows, there’s always a way around a blacklist. The use of NotAction effects in policies is generally a bad idea and not recommended, but we still see it quite often. It is an area that often leads to privilege escalation and is definitely something you’ll want your tools to be aware of.

Detection Example:

What if you have a role with a policy applied that uses the NotAction effect to say that you can do everything but a list of specific actions? Unless the NotAction statement covers every single combination that allows for privesc, you might be able to escalate privileges. If the tool does not properly handle the NotAction effect in policies, you will have false negatives.

IAM Vulnerable Test Case:

False negative (exploitable) – NotAction

Does the tool handle Condition constraints?

The optional condition element lets you specify conditions for when a policy is in effect. There are AWS global condition context keys and IAM and AWS STS condition context keys, and you can apply those conditions to any policy.

All of the tools either ignore conditions entirely or only provide manual support. For example, PMapper supports manually evaluating conditionals with the "who can" query type, but this is not fully evaluated when using they "preset privesc" query. The result is that there will be both false negatives and false positives, and we just have to be aware of this and manually investigate cases where conditions are used.

In my opinion, conditions are the hardest things for a tool that is not built into AWS to evaluate correctly. The tool would have to recreate the evaluation logic for each of the specific sets of conditions supported by AWS. In many cases, even with full API access to AWS, the tools don’t have access to the data needed to make some of these decisions (e.g., what IP address the caller is coming from).

Detection Example:

What if you have the permission to iam:CreatePolicyVersion, but there’s a condition that says you can only execute this action if the time is in the past? The condition applied would make this otherwise exploitable configuration non-exploitable, and if the tool alerts on this, it would be a false positive. However, the time is just one of many conditions that AWS supports, making this area particularly tricky. What if the condition is that the source IP of the caller must be a specific IP address? How will the tool know what IP address you are coming from?

If you are interested in digging deeper, I have two tests for this in IAM Vulnerable that you can modify or just play around with as is. I have one role with an attached policy that uses a condition that will always evaluate to true (the aws:TokenIssueTime needs to be after 2020-01-01), and I have another role with an attached policy that uses a condition that will always evaluate to false (the aws:TokenIssueTime needs to be before 2020-01-01).

IAM Vulnerable Test Cases:

False positive (non-exploitable) – Condition constraint

False negative (exploitable) – Condition constraint

Does the tool handle service control policies?

According to Amazon’s documentation, “Service control policies (SCPs) are a type of organization policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization. SCPs help you to ensure your accounts stay within your organization’s access control guidelines.”

In order for a tool to be able to even consider evaluating SCPs, a pen tester would need permissions at the organization level to view the SCPs. The tool would then need to analyze each individual AWS account and apply the SCPs like AWS would. PMapper is the only tool that supports SCPs, which is a pretty big differentiator.

Detection Example:

What if you have all the permissions you need to execute a privesc path, but there is an SCP at the organization level that prevents the attack? If the tool alerts on this, it is a false positive.

IAM Vulnerable Test Cases:

None

What privesc paths does each tool support?

Now that we’ve covered all of the aspects a comprehensive privesc tool needs to support, let’s finally get into the specific attacks. I’ve used Gerben’s five categories to sort the attacks and sorted the new attacks into the existing categories.

An important note here: The purpose of this table is to show which privesc paths are supported by the tool in their most basic manifestation – a single policy, and with all of the right prerequisites in place. In a real environment, you’ll need to consider the capabilities that each tool supports (shown above) to figure out where the tool might return a false positive or false negative.

Also, there are likely privesc paths that these tools identify that I have not accounted for yet in IAM Vulnerable.

What’s the best tool (for me)?

While my assessment criteria laid out above is all empirical, there are some other factors to consider, including ease of installation, ease of use, intended use cases, and the way the data is represented. For example, when I am on an objective-based cloud penetration test and I’m looking to find the path of least resistance to achieve the assessment objectives, PMapper and AWSPX are my favorites, and I always use both.

I love the way AWSPX gives you an interactive Neo4j graph-based interface and the ability to view the actual policies that enable the privesc path right within the tool. However, AWSPX only supports IAM, EC2, and Lambda. When it comes to completeness, there is no match for PMapper. PMapper supports the most unique privesc paths, and it is also the tool that correctly handles the greatest number of capabilities outlined above, which means fewer false positives and false negatives.

When I’m looking to identify as many potential privesc paths as possible, even ones that are not currently exploitable, Pacu's iam__privesc_scan module and Cloudsplaining are really helpful, as long as you are aware of their limitations. They both give you a list of all of the ways each principal can potentially escalate, even if the path is missing some required pre-conditions. Sometimes, even if the path is not exploitable today, you still want to know that you have a ticking time bomb on your hands.

What’s next for IAM Vulnerable?

I created IAM Vulnerable so that I could share my intentionally vulnerable AWS account playground with others. Then, I saw the opportunity to use it to better understand the AWS assessment tools I was using. Moving forward, I see two additional use cases.

First, I see IAM Vulnerable as a resource for IAM privesc tool developers to use in making sure their open source tools catch all of the things they are hoping to catch. During this journey, I’ve identified and submitted a bunch of bugs to the projects listed above, but with IAM Vulnerable open sourced, anyone can now do the same.

Second, I see IAM Vulnerable as a place where we, the community, can continuously add more AWS IAM privesc paths over time. When I started this journey, I was under the false impression that there were only 21 permission-based exploit paths possible in AWS. I was wrong. Those 21 exploit paths were just the foundation that Spencer laid down in 2018. There are many additional combinations of permissions that allow for privilege escalation that just have not been documented as thoroughly, or at all. Any AWS service that allows you to pass a role to it needs to be examined for potential privesc. Additionally, any service that might be using a highly privileged IAM role and that allows you to potentially backdoor existing resources is yet another opportunity for privilege escalation. I’d like to continue enumerating additional privesc paths in IAM Vulnerable, and I welcome your contributions!