Dufflebag: Uncovering Secrets in Exposed EBS Volumes

Get the Dufflebag source code on GitHub

Did you know that Elastic Block Store (Amazon EBS) has a "public" mode that makes your virtual hard disk available to anyone on the internet? Apparently, hundreds of thousands of others didn't either, because they're out there exposing secrets for everyone to see.

When we presented our findings in “More Keys than a Piano: Finding Secrets in Publicly Exposed EBS Volumes” back at DEF CON 27 (Slides,Video), we found all sorts of secrets and associated data—passwords, SSH private keys, TLS certificates, application source code, API keys, and anything else that might be stored on a server hard disk. Even more surprising, some of this sensitive information was found on “internal-only” resources that are hosted on AWS. So by searching exposed EBS volumes, an attacker can steal secrets from a server that isn’t even exposed to the internet!

To help identify these exposed EBS volumes and allow individuals and businesses to secure their secrets, the Bishop Fox team developed Dufflebag, an open source tool now available on GitHub. We waited until now to release the tool to give the affected parties some time to effectively deal with this issue and to polish the interface for broader use.

Finding the Loot

Although in theory all this data is just sitting out there on AWS for the taking, actually processing the contents of hundreds of thousands of hard drives isn’t so simple. There’s no search feature built into EBS, and there are petabytes of data to look through. So, how did we sift out the gems?

This is where Dufflebag comes in. This open source tool allows users to quickly rummage through public EBS volumes, shove what you want into a giant (duffle)bag, and make out with the loot. The logistics of actually reading from an EBS volume is quite complex—you have to copy the EBS snapshot, instantiate a new volume from it, attach the volume to a running AWS machine, and then mount the filesystem to search through each disk—and that to-do list doesn’t even account for region-specific data, cleaning up afterwards, or the fact that at any point, a step might fail for no clear reason. But Dufflebag takes care of all that so you can just explore found secrets
(e.g., /etc/shadow files) in the S3 UI, as shown below:

Dufflebag is an Elastic Beanstalk application. We set it up this way to make it easier to use and to remove the hurdle of setting up complex infrastructure that requires a lot of management. Dufflebag deploys and scales with ease and tears down just as easily. As a bonus, Elastic Beanstalk applications are fairly inexpensive to run on AWS.

To start your offensive security exercise, we suggest that you search the exposed EBS snapshots for references to your organization. This will give you an attacker’s view into public snapshots and data exposed on the internet. Even if you don’t see any public snapshots in your AWS account, you never know what might be lurking in the “shadow IT” infrastructure you didn’t know existed. Instructions on how to do this are in our technical guide on GitHub, where you’ll also find the full source code to Dufflebag. Then, with just a few small tweaks, you can change Dufflebag from searching for the worlds’ secrets to just yours.

You can see Dufflebag in action in this short video:

Securing Your Secrets

Ultimately, the solution is to make sure that all of your organization’s EBS snapshots are either private or encrypted (ideally both). In the AWS console, look at the Permissions tab for any snapshot and make sure it says “This snapshot is currently Private” as shown in the image below:

Additionally, you can look through the list of public snapshots to see if any of yours are included. If you see your snapshot in the list, assume that it’s compromised. Check out the list by selecting Public Snapshots from the drop-down below:

If you see your snapshot on the list and it’s not encrypted, it’s been publicly exposed. If that’s the case, make the snapshot private, then immediately initiate your incident response process to stop the bleeding:

1. Figure out what data has been exposed (passwords, keys, etc.)
2. Revoke all exposed credentials and keys and make new ones.
3. Investigate what caused the snapshot to be published (i.e., Is there some internal process at fault that would cause this to happen again? If so, lock that down either through process changes or user education).

There’s no good way to know if someone has read data from your public snapshots, so it’s prudent to take an aggressive line and assume that all of the data you find on an exposed snapshot has become public knowledge.

Taking Inventory

Keeping track of all this data is hard and if you’re not careful; sensitive business data could accidentally wind up on GitHub, Pastebin, and now EBS snapshots. Dufflebag can help you search for your data out there on the cloud and provide you with the visibility you need, but it’s up to you to stay vigilant. We’d love to hear what you think about Dufflebag and how you’re using it – hit us up on Twitter @bishopfox.