Summary
Bishop Fox researchers successfully reverse-engineered the encryption protecting SonicWall SonicOSX firmware, gaining access to the underlying file system for purposes of good-faith security research. They presented a walkthrough of their process and results at DistrictCon Year 0, the contents of which are described below. The researchers also released a tool, Sonicrack, to automatically extract keys from VMware virtual machine bundles and decrypt VMware NSv firmware images.
Background
SonicWall produces a range of next-generation firewall appliances that are frequently used by large commercial enterprises to protect their network perimeters. Appliance models include SOHO and TZ for consumers and small businesses, NSa for mid-size companies, NSsp and Supermassive for large enterprises, and NSv for hypervisor and cloud deployments.
These devices all run the SonicOS operating system, which currently has four major versions under active support (5 through 8). Prior to the release of version 7.1.1, firmware images for hardware appliances were distributed in a proprietary SIG encryption format, while NSv virtual machines used LUKS encryption to protect embedded volumes. In February 2022, Catalpa published a technique for extracting the encryption keys from memory and decrypting the NSv volumes. In December 2023, Praetorian improved on this technique by reverse-engineering the key derivation process and releasing a tool to decrypt static NSv images.
Starting with the release of version 7.0.1, SonicWall upgraded the operating system to SonicOSX, which brought significant changes to platform features, OS architecture, file system organization, and firmware packaging. Significantly, versions 7.1.1 and later of the NSv virtual machines use the same proprietary SIG encryption as the firmware for hardware appliances.
As a brief aside, it’s worth noting that a small proportion of firmware images for NSsp and NSv used a third proprietary encryption format, SWI. Our research team successfully reverse-engineered that format as well, and published a walkthrough of the decryption process on our blog in December.
Here’s a summary of encryption formats by appliance model and OS series:
Walkthrough Part 1: Extracting Keys
In this walkthrough, we’ll be using an Open Virtualization Application (OVA) version of SonicOSX version 7.1.1-7047.
Our goal is to unpack the VMware OVA image and get to the root file system so that we can analyze the files within it and gain an understanding of how the operating system functions. Typically, we would expect the process to look like this for an unencrypted file system:
The OVA bundle is just a tar archive – once unpacked, it contains several files that VMware uses to build the virtual machine, but all we care about is the primary virtual machine disk (a VMDK file). That file is also an archive, which can be unpacked using 7zip. Typically, this archive contains several volume images that are mounted when the virtual machine boots. One of these should contain the firmware, and we usually just need to mount the image to extract it. From there, the firmware could be in a variety of formats, but if it's not encrypted, then we just have to unpack it to get the root file system.
When the file system is encrypted, we must decrypt it before we can unpack the file system. To do that, we need to find encryption keys and some indication of how the encryption was performed. In a hardware appliance, we would expect the keys to be contained within some sort of secure storage mechanism, such as a Trusted Platform Module (TPM). In a virtual machine, however, no such mechanism is available, so we know the keys must be contained somewhere within the virtual machine image. We would first check all the volume images to see if the keys are just sitting there, but that doesn’t happen very often. They could also be stored in the NVRAM file, but what we commonly find is they are stored somewhere within the bootloader:
As a result, most of the fun in reverse-engineering encrypted firmware is in dissecting the bootloader to hunt for keys! Each vendor seems to take a different approach to this part and, as you’ll see, SonicWall chose a particularly complex one.
We start by unpacking the OVA file and then disk1.vmdk as described above, and we find several embedded volumes:
We mount these images, examine the volumes, and find that only two contain files: BOOT and INSTALL-CACHE:
The BOOT volume contains an EFI bootloader, and the INSTALL-CACHE volume contains firmware in SIG format. If we run binwalk on the firmware binary, it fails to identify any file signatures within:
$ binwalk currentFirmware.bin.sig Analyzed 1 file for 85 file signatures (187 magic patterns) in 1.4 seconds
Furthermore, binwalk’s entropy analysis feature shows high entropy throughout the file (note the horizontal yellow line), suggesting that it is encrypted:
Since there is only one other file contained within the virtual machine volumes, it seems safe to assume the encryption keys must be somewhere within the bootloader. Using binwalk to automatically unpack bootx64.efi produces the following files:
We inspect the Grub configuration file to discover the purpose of the other files:
$ cat grub.cfg
...omitted for brevity...
# SOURCE drop-in
source (memdisk)/dropin.cfg
###
### Boot Menu:
###
menuentry 'Installer' --unrestricted {
set fallback=1
linux (memdisk)/soniccorex.bin $linux_console $linux_quiet
$linux_append
initrd (memdisk)/initramfs.gz
}
We can see here that dropin.cfg is run prior to loading the Grub boot menu, and by examining the contents of that file, we learn that its role is to enforce the use of secure boot:
$ cat dropin.cfg
### Don't let prod builds proceed with out secure-boot
get_efivar -f uint8 -s secured SecureBoot
if [ "${secured}" != "1" ]; then
echo "Invalid firmware detected, please contact SonicWall Technical Support"
echo "Shutting down ...."
sleep 5
halt
fi
We also learn that soniccorex.bin is a Linux kernel and initramfs.gz is used as an initial ramdisk (initrd). You probably know that the primary function of a bootloader is to load the kernel into memory and start the operating system’s boot process, but you’d be forgiven for not knowing what initrd is. According to Wikipedia, an initial ramdisk is:
…a scheme for loading a temporary root file system into memory, to be used as part of the Linux startup process.…commonly used to make preparations before the real root file system can be mounted.
That sounds like a great place to handle pre-boot operations like, say, decrypting a file system, does it not? Since binwalk didn’t find any keys hiding directly in the bootloader, let’s unpack the initramfs archive and see what we find.
Is your Spidey sense tingling? It looks like we have a key sitting here in the initial ramdisk file system (hooray!), but there also appears to be an encrypted archive called sunup (uh oh). Before we jump to any conclusions, let’s examine init to see what’s happening during this part of the boot process:
$ cat init
...omitted for brevity...
ROOT_MOUNT="/stage1rootfs"
ROOT_IMAGE_ENC=/sunup.cpio.gz.enc
ROOT_IMAGE="/tmp/sunup.cpio.gz"
...omitted for brevity...
if test "$use_sunup_key_from" = "FS"
then
if is_key_in_fs
then
log "Found key in FS..."
# key=/onetime.key is present in FS
else
halt "Unable to setup stage2 (key not in fs) ..."
fi
fi
...omitted for brevity...
# decrypt stage2 (sunup)
openssl enc -a -d -aes-256-cbc -pbkdf2 \
-in $ROOT_IMAGE_ENC -salt \
-out $ROOT_IMAGE \
-pass file:${KEY}
...omitted for brevity...
# switch_root can only switch to a mount point (not a dir)
mount -t tmpfs tmpfs "$ROOT_MOUNT"
( cd "$ROOT_MOUNT" && gzip -dc "$ROOT_DISK/$ROOT_IMAGE" | cpio -dmi 2>/dev/null )
log "Changing root to stage2 (sunup) ... "
# boot the image
...omitted for brevity...
exec switch_root -c /dev/console $ROOT_MOUNT /init ||
fatal "Couldn't jump to stage2 initramfs ..."
First, we see that it’s a Bash script. The lines excerpted above indicate that the script mounts and boots a secondary initram file system, and the image it mounts is sunup.cpio.gz.enc. As we suspected, the image is encrypted, and when onetime.key is present in the primary initram file system, the key is used to decrypt the image before mounting it.
Another part of the script indicates that the sunup encryption key might instead be extracted from various secure storage mechanisms, which is good to know, but is irrelevant since we’re analyzing a virtual machine and secure storage is not available.
Let’s continue down the rabbit hole and use the openssl command above to decrypt the sunup image, then unpack it to see what files it contains:
$ openssl enc -a -d -aes-256-cbc -pbkdf2 -in sunup.cpio.gz.enc -salt -out sunup.cpio.gz -pass file:onetime.key $ gunzip sunup.cpio.gz $ cpio -i <sunup.cpio dev/console: Can't create 'dev/console' 244944 blocks
Here we have yet another file system, the stage 2 initramfs, also known as the sunup installer. A quick search reveals some keys that might be the firmware keys we’re looking for:
$ find . -name "*.key" ./usr/share/installer/FW-crypt-release.key ./usr/share/installer/SUNUP-crypt-release.key ./usr/share/installer/SCX-crypt-release.key
Once again, we need to analyze the init script to understand what actually happens during this part of the boot process. Before we continue, however, let’s recap what we’ve done so far. Here’s a flowchart illustrating the steps we took to get from the OVA package down to the sunup volume, with each distinct file system represented as a new row in the chart:
Walkthrough Part 2: Decrypting Firmware
Analyzing the init script within the sunup volume (stage 2 initramfs) points us to several Bash scripts contained within the init.d directory, which in turn point to many more Bash scripts in /usr/bin.
Eventually, we find a handful of files that are responsible for decrypting the firmware and installing the root file system:
/usr/bin/installer/usr/bin/fwDecrypt/usr/bin/multi-crypto/usr/lib/config/platform/system.sh
Within the /usr/bin/installer script, we find the primary functions used to decrypt firmware images:
fwEncrypt() {
CMD_TAG=fw-decrypt stdin: "$SOURCE_MONITOR" fwEncryptExtract
SCX_PACKAGING+=(SW-fwEncrypt)
}
...omitted for brevity...
sw-crypt-package() {
local format value type rest
read -r format value type rest
SCX_PACKAGING+=(SW-crypt)
SCX_SWP_HEADER="$format $value $type $rest"$'\n'$(dd 2>/dev/null bs=256 count=1 | base64)
CMD_TAG=scx-decrypt stdin: "$SOURCE_MONITOR" swpackage_decrypt "${SCX_SWP_HEADER#*$'\n'}"
}
These functions appear to decrypt the SIG firmware bundle and something referred to as “swpackage.” This is interesting, because it suggests that two rounds of decryption are necessary to get to the root file system (we hope).
We’ll start by analyzing the fwEncrypt function.
fwEncrypt() {
CMD_TAG=fw-decrypt stdin: "$SOURCE_MONITOR" fwEncryptExtract
SCX_PACKAGING+=(SW-fwEncrypt)
}
We see that fwEncrypt is calling another function, fwEncryptExtract, which we find inside /usr/bin/fwDecrypt. The first part of this function defines a header format which it parses from the start of the SIG firmware image:
fwEncryptExtract() {
local len=36 iv magic headerLen headerRev headerVer zero algoKeyId
...omitted for brevity...
readi4 algoKeyId &&
readi2 sskbIndex &&
readi2 rsaKeyLen && # len=20 characters
test $(( $headerLen - $rsaKeyLen )) = 36 &&
readxn 16 iv || # len=36 characters
installer_fatal "Can't unpack fwEncrypt header"
The fwEncryptExtract function extracts encrypted key material from the file header and calls another function, recoverAesX, to decrypt it and obtain an AES key:
local rsaKeyName="${sskb[$sskbIndex]}"
# we can now read rsaKeyLen bytes which is the encrypted aesKey
local aes
# shellcheck disable=SC2086
if test "$rsaKeyLen" = 0
then aes=$(recoverAesX $rsaKeyName < "$FWENCRYPT_KEY_DIR/aes-$algoKeyId.rsa")
else aes=$(dd bs="$rsaKeyLen" count=1 2>/dev/null | recoverAesX $rsaKeyName)
fi
The recoverAesX function shows that FW_DECRYPT_KEY is used to decrypt the AES key:
recoverAesX() {
installer_decrypt_key "${1:-${FW_DECRYPT_KEY}}" | hexdump -ve '1/1 "%.2x"'
test "${PIPESTATUS[*]}" = "0 0"
}
FW_DECRYPT_KEY therefore must be a key encrypting key, or KEK. Searching through the sunup file system, we find that FW_DECRYPT_KEY is defined in /usr/lib/config/platform/system.sh:
FW_DECRYPT_KEY='FW-crypt-release.key'
If you recall, FW-crypt-release.key is one of the files we guessed earlier was a firmware encryption key, so we seem to be on the right track! Now let’s take a closer look at how the installer_decrypt_key function works – we’ll find it in /usr/bin/multi-crypto:
# $1 contains decrypt-key name
# $2 has base64 encrypted payload (on or non-base64 on stdin)
# decrypt payload (encrypted key) using key in $1
installer_decrypt_key() {
local names name padding paddings="oaep pkcs1" decrypt decrypt_type decrypt_types="${SCX_INSTALLER_KEY_SEARCH:-ss ss_extra file}"
names="$1"
debug "Decrypt using $names"
# we grab payload base64 because we want multiple attempts to decrypt it
if test -n "$2"
then payload_b64="$2"
else # on stdin
payload_b64="$(base64)"
fi
for padding in $paddings
do for decrypt_type in $decrypt_types
do decrypt="$decrypt_type-rsa-$padding-decrypt"
for name in $names
do debug "Try $decrypt $name"
base64 -d <<<"$payload_b64" | "$decrypt" "$name"
if test "${PIPESTATUS[*]}" = "0 0"
then debug "Decrypted with $decrypt $name"
return 0
fi
done
done
done
}
This function essentially tries to decrypt the key material using the KEK with two different padding types: it tries oaep first, and if that fails, it tries pkcs1. Both of those function calls use openssl to perform the AES key decryption:
file-rsa-pkcs1-decrypt() {
openssl rsautl -decrypt -inkey "${SCX_KEY_PATH}$1${SCX_KEY_SUFFIX}" 2>/dev/null && return
}
file-rsa-oaep-decrypt() {
openssl rsautl -decrypt -oaep -inkey "${SCX_KEY_PATH}$1${SCX_KEY_SUFFIX}" 2>/dev/null && return
}
Alternative decrypt functions are called if the KEK was retrieved from secure storage, but in our case, because we’re analyzing a virtual machine, the KEK was retrieved from a file within the sunup volume.
Once the AES key has been decrypted, we return to the fwEncryptExtract function, which uses openssl to decrypt the rest of the firmware with the AES key, then validates the footer of the decrypted file:
openssl enc -d -nopad -aes-128-cbc -iv "$iv" -K "$aes" 2>/dev/null | fwDepadFooter
Let’s recap what we’ve learned so far with a flow chart:
- We begin with an encrypted SIG firmware file and a firmware KEK.
- We split the firmware file into two parts, referred to in the scripts as an SSDH header and encrypted SSDH package.
- We extract encrypted key material from the SSHD header.
- We decrypt the key material using the firmware KEK to obtain an AES key.
- We decrypt the SSDH package using the AES key.
That completes our analysis of the fwEncrypt function from /usr/bin/installer. Next, we’ll analyze the sw-crypt-package function from /usr/bin/installer.
sw-crypt-package() {
local format value type rest
read -r format value type rest
SCX_PACKAGING+=(SW-crypt)
SCX_SWP_HEADER="$format $value $type $rest"$'\n'$(dd 2>/dev/null bs=256 count=1 | base64)
CMD_TAG=scx-decrypt stdin: "$SOURCE_MONITOR" swpackage_decrypt "${SCX_SWP_HEADER#*$'\n'}"
}
Here we see that sw-crypt-package is extracting a 256-byte header from the SSDH package, base64-encoding it, and passing the result to a swpackage_decrypt function:
swpackage_decrypt() {
local KEY
# attempt to decrypt key from first 256 bytes
if KEY="$(installer_decrypt_key "$INSTALLER_DECRYPT_KEY" "$1")"
This function calls installer_decrypt_key again, which repeats the process of extracting and decrypting key material from the header, only this time it’s using a different KEK. Returning to system.sh, we find the definition for INSTALLER_DECRYPT_KEY:
INSTALLER_DECRYPT_KEY='SCX-crypt-release.key'
Another win – we previously identified that file in the sunup volume too! After decrypting the AES key, swpackage_decrypt uses openssl to decrypt the rest of the SSDH package:
{ printf "Salted__" ; test -n "$1" && base64 -d <<<"$1" ; cat /dev/stdin ; } | openssl enc -d -pbkdf2 -aes-256-cbc -pass fd:4 4<<<"${KEY}" 2>/dev/null
The scripts refer to the resulting file as an SW package, which we are hoping contains the root file system (our goal). Let’s recap the firmware decryption process once more:
- We begin with an encrypted SIG firmware file, a firmware KEK, and a software KEK.
- We split the firmware file into two parts, referred to in the scripts as an SSDH header and encrypted SSDH package.
- We extract encrypted key material from the SSHD header.
- We decrypt the key material using the firmware KEK to obtain an AES key.
- We decrypt the SSDH package using the AES key.
- We split the decrypted SSDH package into two parts, referred to in the scripts as an SW header and encrypted SW package.
- We extract encrypted key material from the SW header.
- We decrypt the key material using the software KEK to obtain an AES key.
- We decrypt the SW package using the AES key.
Now, all that remains is to analyze the decrypted SW package to see if it contains the root file system we’re looking for. Let’s run binwalk to see what it tells us:
$ binwalk -e sw-package
-------------------------------------------------------------------------------------------------------------------------------------------------------------
DECIMAL HEXADECIMAL DESCRIPTION
-------------------------------------------------------------------------------------------------------------------------------------------------------------
0 0x0 gzip compressed data, operating system: Unix, timestamp: 2024-02-05 22:50:37, total
size: 2943 bytes
2943 0xB7F gzip compressed data, original file name:
"soniccorex-image-release-nsv-vmware-20240205204943.rootfs.ext4", operating system:
Unix, timestamp: 2024-02-05 22:50:35, total size: 328547938 bytes
328550881 0x139549E1 gzip compressed data, original file name:
"soniccorex-image-release-nsv-vmware-20240205204943.rootfs.ext4.verity", operating
system: Unix, timestamp: 2024-02-05 22:50:37, total size: 19841152 bytes
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[+] Extraction of gzip data at offset 0x0 completed successfully
[+] Extraction of gzip data at offset 0xB7F completed successfully
[+] Extraction of gzip data at offset 0x139549E1 completed successfully
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Analyzed 1 file for 85 file signatures (187 magic patterns) in 6.1 seconds
Excellent! It looks like we found what we were looking for. Let’s unpack the rootfs image with 7zip (noting binwalk didn’t keep the original filename):
$ mkdir rootfs $ cd rootfs $ 7z x ../extractions/sw-package.extracted/B7F/decompressed.bin ...omitted for brevity... Folders: 980 Files: 11119 Size: 700436639 Compressed: 681230336
Finally, at long last, we can browse through the root file system of SonicOSX:
Just to prove to ourselves that we finally reached our goal, let’s search for sonicosv, the main binary that runs most of the firewall functionality:
$ find . -name sonicosv ./usr/share/lxc/containers/app-container-image-sonicos/rootfs/opt/sonicwall/sonicos/bin/sonicosv $ cd usr/share/lxc/containers/app-container-image-sonicos/rootfs/opt/sonicwall/sonicos/bin $ file sonicosv sonicosv: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-x86-64.so.2, BuildID[sha1]=cf5f4698ddd071c6733e681a85519fbdd2a21094, for GNU/Linux 3.2.0, stripped
Mission accomplished! This journey was a longer one than we anticipated, so let’s see if we can capture the whole process in a single flowchart:
Phew! After putting together this diagram, we couldn’t help ourselves – we had to compare it to Inception – and don’t you know, it worked out pretty well:
Sonicrack
After going to all the trouble of reverse-engineering the key extraction and firmware decryption process for SonicOSX, we decided to open-source the fruits of our labor. Our intent behind releasing Sonicrack is to make the current generation of SonicWall firewalls more accessible for good-faith security research. After all, if our small team of researchers can successfully decrypt the firmware, well-funded nation-state actors have certainly done the same. It is our belief that tools like this one help to level the playing field for those who work to protect consumers from malicious adversaries.
To use Sonicrack, you’ll need to start by running it against a VMware OVA image. We recommend building and running it in a Docker container since some of the dependencies can behave differently in other environments.
$ docker build -t sonicrack . ...omitted for brevity... $ docker run -it --privileged --rm -v .:/data sonicrack ./sonicrack.py --keys /data/keys --output /data /data/nsv-vmware.7.1.1-7047-R5557.ova [*] Extracting files from OVA archive [*] Decrypting firmware image [+] Successfully decrypted soniccorex-image-release-nsv-vmware-20240205204943.rootfs.ext4
Performing the first run against an OVA image is important so that Sonicrack can extract the keys:
$ ls keys FW-crypt-release.key SCX-crypt-release.key
Once you have the keys, you can decrypt any VMware SIG firmware (version 7.1.1 and later):
$ docker run -it --rm -v .:/data sonicrack ./sonicrack.py --keys /data/keys --output /data /data/sw_nsv_vmware_eng.7.1.3-7015-R6965.bin.sig [*] Decrypting firmware image [+] Successfully decrypted soniccorex-image-release-nsv-vmware-20241209200721.rootfs.ext4
Happy hacking! And as always, be sure to check out a few of our additional resources to see further research on SonicWall vulnerabilities and insights.
Subscribe to our blog and advisories
Be first to learn about latest tools, advisories, and findings.
Thank You! You have been subscribed.
