Introduction
This blog is based on previous research conducted by Carl Livitt, Bishop Fox alumnus.
The previous article in our Fortinet series, CVE-2023-27997 is exploitable, and 69% of FortiGate firewalls are vulnerable, described how to use intelligent Shodan queries to identify FortiGate SSL VPN endpoints exposed on the internet. By comparing the dates in their Last-Modified
response headers to patch release dates, we were able to estimate how many devices were vulnerable to a recently discovered heap overflow exploit allowing remote code execution.
In this blog, we will dive deeper into an issue we had to overcome to perform comprehensive research on FortiGate firmware. Our intent here is to expose more of the (often rigorous) process involved in performing security research, and to share with the wider security community what we learned along the way.
The Challenge
Performing comparative analysis on a particular software product requires obtaining several different versions of that software. In the case of FortiGate, this was relatively straightforward – all we had to do was register a free account on Fortinet’s support site and request a free trial license (note that Fortinet has since restricted trial access – more on this later). We could then download firmware images for a wide variety of versions, hardware appliances, and even product lines (not just FortiGate). The total number of available images numbered in the tens of thousands (!), so we wrote a Python script using Playwright to help us download as much of the available library as possible.
The trick, however, was this: although FortiGate virtual machines were distributed in the clear, most of the firmware images intended for bare-metal deployment were encrypted. Below is a table summarizing the distribution of cleartext vs. encrypted images for various Fortinet products. FortiGate and FortiProxy are called out because they were affected by CVE-2023-27997 and comprise nearly half of the available images. Other products distributed with encryption included FortiAnalyzer, FortiAuthenticator, FortiMail, FortiManager, and FortiVoice.
FIGURE 1 - Distribution of encrypted vs. cleartext firmware images across Fortinet products
To thoroughly compare different versions of FortiGate firmware, we needed to find a way to decrypt the encrypted firmware images.
Reversing the Encryption Scheme
Solving this problem required using reverse engineering to meet the following goals:
- Understand the firmware image file formats (cleartext and encrypted)
- Locate the logic (in a cleartext image) responsible for decrypting firmware (if possible)
- Reproduce the decryption function with our own code
- Write the corresponding encryption function
If we could accomplish this, then we would understand the encryption method well enough to identify any weaknesses in its cryptographic implementation.
Unwrapping the file formats
Achieving the first goal was straightforward. A simple run of the file command revealed that both image formats were compressed with gzip, e.g.:
❯ file FGT_* FGT_100D-v6-build9451-FORTINET.out: gzip compressed data, was "FG100D-6.04-FW-build1966-230310-patch09", last modified: Fri Mar 10 00:29:41 2023, from Unix, original size modulo 2^32 926036017 FGT_30E-v6-build0076-FORTINET.out: gzip compressed data, was "FGT30E-6.00-FW-build0076-180329-patch00", last modified: Thu Mar 29 03:19:25 2018, from Unix, original size modulo 2^32 926036017
Extracting the files required adding a couple of parameters to the gunzip command since it did not recognize the .out file extension:
❯ gunzip -cf FGT_100D-v6-build9451-FORTINET.out >FGT_100D-v6-build9451-FORTINET gunzip: FGT_100D-v6-build9451-FORTINET.out: trailing garbage ignored ❯ gunzip -cf FGT_30E-v6-build0076-FORTINET.out >FGT_30E-v6-build0076-FORTINET gunzip: FGT_30E-v6-build0076-FORTINET.out: trailing garbage ignored
Once the images were extracted, it was easy to see which were encrypted vs. cleartext by inspecting them with file and xxd:
❯ file FGT_* FGT_100D-v6-build9451-FORTINET: data FGT_30E-v6-build0076-FORTINET: DOS/MBR boot sector; partition 1 : ID=0x83, active, start-CHS (0x7,230,32), end-CHS (0xa,50,40), startsector 126976, 36864 sectors; partition 2 : ID=0x83, start-CHS (0xa,50,41), end-CHS (0xc,125,49), startsector 163840, 36864 sectors; partition 3 : ID=0x83, start-CHS (0xc,125,50), end-CHS (0x10,81,1), startsector 200704, 61440 sectors ❯ xxd -l 80 FGT_100D-v6-build9451-FORTINET.out 00000000: 90d0 b0f1 bcda 8be8 85bb f79a f6bc 4c40 ..............L@ 00000010: 7f6e 474e 3d2f 0001 2a10 3036 675c 4796 .nGN=/..*.06g\G. 00000020: 8ca7 ab8e f2af d78e ded2 a9f4 acd5 a3f7 ................ 00000030: 8ed6 aef7 d48a f0ab deb4 c095 bae8 a6e9 ................ 00000040: 86c6 a6e7 aacc 8eed 80be f29f f4be f89f ................ ❯ xxd -l 80 FGT_30E-v6-build0076-FORTINET 00000000: 0000 0000 0000 1100 0000 0000 ff00 aa55 ...............U 00000010: 4647 5433 3045 2d36 2e30 302d 4657 2d62 FGT30E-6.00-FW-b 00000020: 7569 6c64 3030 3736 2d31 3830 3332 392d uild0076-180329- 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
In this example, the FGT_30E image was identified as a bootable volume with a build name clearly visible in the 80-byte file header. The FGT_100D image, by contrast, was only identified as “data” with garbled text in place of a build name. From this vantage point, you may already see some indicators of weakness in the encryption scheme.
Accessing the file system
Having determined the minimal processing needed to differentiate encrypted images from cleartext ones, our next goal was to locate decryption code within the cleartext images. We first needed to gain access to the file system, and the easiest way to do that was to mount the disk image included with one of the freely distributed virtual machines.
Starting with a VMware image from the Fortinet support site (this example uses FortiGate version 6.4.13), we performed the following actions:
- Unzipped the download and looked for
fortios.vmdkwithin the OVF file structure - Instead of importing the FortiGate OVF, attached
fortios.vmdkto an existing Linux virtual machine (we used Kali Linux), then booted that VM - Logged in and opened a root shell
- Created a mount point, e.g.,
mkdir /mnt/fortigate - Listed attached volumes with
fdisk -land looked for the FortiGate volume (usually the second one listed) - Mounted the bootable partition of the FortiGate volume, e.g.,
mount /dev/nvme0n2p1 /mnt/fortigate - Browsed to
/mnt/fortiosto obtain any necessary files
This is what the process looked like in our case:
❯ fdisk -l ...omitted for brevity... Device Boot Start End Sectors Size Id Type /dev/nvme0n2p1 * 2048 526335 524288 256M 83 Linux /dev/nvme0n2p2 526336 4194303 3667968 1.7G 83 Linux ❯ mkdir /mnt/fortigate ❯ mount /dev/nvme0n2p1 /mnt/fortigate ❯ ls /mnt/fortigate boot.msg datafs.tar.gz datafs.tar.gz.bak extlinux.conf extract.flag filechecksum flatkc flatkc.chk image.src ldlinux.c32 ldlinux.sys lost+found rootfs.gz rootfs.gz.chk
We observed that the boot partition contained several archives that would be decompressed to create the file system during initial setup. With some analysis, we found that each of the .gz files was a cpio archive compressed with gzip. Extracting one of these archives looked like this:
❯ gunzip rootfs.gz ❯ file rootfs rootfs: ASCII cpio archive (SVR4 with no CRC) ❯ cpio -i <rootfs ...omitted for brevity... 194419 blocks ❯ ls bin.tar.xz boot data data2 dev etc fortidev init lib migadmin.tar.xz node-scripts.tar.xz proc rootfs sbin sys tmp usr usr.tar.xz var
After spending some time reviewing the file structure and noting the large number of symlinks used in place of various binaries, we determined that the vast majority of program functionality was handled by a single monolithic binary located at /sbin/init:
> ls -og sbin total 180 -rwxr-xr-x 1 22936 Jun 28 20:21 ftar -rwxr-xr-x 1 15016 Jun 28 20:21 init -rwxr-xr-x 1 141832 Jun 28 20:21 xz > file sbin/init sbin/init: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /fortidev/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=bfdb4a4bf008f20b0bf7c83474a2eeb7cbeacc70, stripped
Also noteworthy were the ftar and xz binaries in the sbin directory, as we found these were modified from the standard versions and were necessary to extract the .tar.xz archives that contained other parts of the file system.
Hunting for decryption code
Our next task was to examine the init binary to see if it contained functions to download and decrypt images. We loaded the binary into Ghidra and dug through the disassembly to find symbols relating to installation and/or upgrade processes. The fact that symbols were stripped from the binary made this analysis extremely time-consuming, but eventually we discovered this gem:
FIGURE 2 - Ghidra decompile view showing the firmware encryption/decryption algorithm
This function was called by others that were related to firmware upgrades and it contained multiple XOR operations, both strong indicators that the symbol’s purpose involved encryption and/or decryption.
Further analysis of the decompiled code confirmed this assumption, so we proceeded to reproduce the code in Python to have a clean working copy to play with.
Rewriting the decryption function
Ghidra generally does a very good job of decompiling code, and this case was no exception. We analyzed the code structure to understand how it worked, and identified several key points:
- The function accepts three inputs:
- A long integer acting as a pointer to some input data
- An unsigned integer indicating the size of the input data
- An integer that alters the function’s flow if equal to 0
- The size of the input data (in bytes) must be at least 512, and it must be divisible by 512.
- The function contains nested while loops that:
- Iterate through the input data in 512-byte blocks.
- Iterate through each block one byte at a time.
- Modify the value of each byte of input data in sequential order using one of two mathematical XOR operations.
- The two mathematical operations appear to be opposites.
- The function also contains an internal pointer to some data that:
- Must exist (the pointer cannot be null).
- Is used as input to the mathematical operations.
- Is accessed byte-by-byte within the nested while loops.
- Never has more than 32-bytes accessed, so it must be 32 bytes long.
- The third input value determines which of two mathematical operations is used to modify the input data.
- The two inner while loops appear to be nested, but upon closer inspection are a misrepresentation of an if/else statement.
In summary:
- The function takes arbitrary data as input, which must be evenly divisible into 512-byte blocks.
- It uses the input data along with a 32-byte key to perform mathematical operations that alter each byte of the input data in sequence (repeating for each block).
- Another input variable controls whether the function encrypts or decrypts the input data.
- The cipher is symmetric; it uses the same key for encrypting and decrypting.
With this understanding of the decompiled code, we wrote a functionally equivalent decryption function in Python:
def decrypt(ciphertext, key):
ptr = 0
num_bytes = len(ciphertext)
cleartext = bytearray()
while True:
key_offset = 0
block_offset = 0
previous_ciphertext_byte = 0xFF # IV is always 0xFF
# Decrypt one 512-byte block at a time
while block_offset != 0x200:
offs = ptr + block_offset
if offs >= num_bytes:
return bytes(cleartext)
# For each byte in the block, bitwise XOR the current byte with the
# previous byte (both ciphertext) and the corresponding key byte
ciphertext_byte = ciphertext[offs]
xor = (
previous_ciphertext_byte ^ ciphertext_byte ^ key[key_offset]
) - key_offset # subtract the key offset to undo obfuscation
xor = (xor + 256) & 0xFF # mod 256 to loop negatives
cleartext.append(xor)
# Proceed to next byte
block_offset += 1
key_offset = (
key_offset + 1 # increment key offset
) & 0x1F # mod 32 to loop around the key
previous_ciphertext_byte = ciphertext_byte
if block_offset == 0x200:
# Reached end of block
break
if ptr + block_offset > num_bytes:
# Reached end of file
return bytes(cleartext)
# Proceed to next block
ptr = ptr + 0x200
if ptr >= num_bytes:
# Reached end of file
return bytes(cleartext)
The corresponding encryption function only required changes to three lines: (lines ending in + key_offset, ^key[key_offset], = xor)
def encrypt(cleartext, key):
ptr = 0
num_bytes = len(cleartext)
ciphertext = bytearray()
while True:
block_offset = 0
previous_ciphertext_byte = 0xFF
key_offset = 0
while block_offset != 0x200:
offs = ptr + block_offset
if offs >= num_bytes:
return bytes(ciphertext)
cleartext_byte = cleartext[offs] + key_offset
xor = previous_ciphertext_byte ^ cleartext_byte ^ key[key_offset]
xor = (xor + 256) & 0xFF
ciphertext.append(xor)
previous_ciphertext_byte = xor
key_offset = (key_offset + 1) & 0x1F
block_offset += 1
if block_offset == 0x200:
break
if ptr + block_offset > num_bytes:
return bytes(ciphertext)
ptr = ptr + 0x200
if ptr >= num_bytes:
return bytes(ciphertext)
We then wrote some scaffolding to test the functions using an arbitrary 32-byte key, and confirmed that they successfully encrypted and decrypted a test string (validating our understanding of the mathematical operations):
❯ ./encrypt.py Key: 4b5659463738474e473435594f4e47453450344f384e344546364e4f45524739 Input text: I'm a little teapot, short and stout. Here is my handle; here is my spout. Ciphertext: fd83b5d0829faa94afe6a58cef2014219545f7878b4d07c408b3c7f43be8913a05230d3c39242d0f3268775a6a0935f8fcd5925c1cd39c8bf54273b1751ada711a22006525685a685350 Cleartext: I'm a little teapot, short and stout. Here is my handle; here is my spout.
At that point, we believed we had successfully reproduced the cryptographic algorithm used to decrypt FortiGate firmware images, but of course we could not confirm that without a valid key. Our reverse engineering effort had not revealed hard-coded decryption keys within the init
binary nor anywhere else on the system.
To obtain decryption keys, then, we needed to analyze the cryptographic scheme for weaknesses that might allow us to derive them.
Performing Cryptanalysis
Looking over the structure of the encrypt function above, we saw two nested while loops:
- The outer loop iterates through 512-byte blocks of data and processes them separately without passing any input from one to the next. This appears to be a basic block cipher like AES (Advanced Encryption Standard) using ECB (Electronic Code Book) mode.
- The inner loop iterates through the bytes within a block and processes each byte using a mathematical XOR operation that includes a key byte as well as the output from the operation on the previous byte. This looks like a basic stream cipher.
We then had to review best practices to recall which features of block and stream ciphers mitigate cryptographic attacks.
Reviewing block cipher security
ECB is the simplest block cipher mode in AES encryption. In this mode, the cleartext is divided into blocks of equal size, and each block is encrypted separately using a key as input:
FIGURE 3 - Electronic Codebook (ECB) mode encryption (image credit: Wikipedia)
This method of encryption can be performed very quickly because the blocks can be processed in parallel, but it does not hide code patterns well because identical cleartext blocks produce identical ciphertext blocks after encryption.
CBC (Cipher Block Chaining) mode improves on this method by XOR’ing the initial block of cleartext with a non-repeating, pseudo-random initialization vector (IV) before it is encrypted with the key. Each resulting block of ciphertext is XOR’d with the next block of cleartext before encrypting it. The improvements made in CBC ensure that any patterns in the cleartext are obscured after encryption, as highlighted by this comparison:
FIGURE 4 - Comparison of ECB and CBC encryption modes (image credit: Wikipedia)
Since we saw that the Fortinet encryption function was using a block cipher similar to AES in ECB mode, we assumed that it had the same weaknesses: it would produce recognizable patterns in ciphertext, and it would decrypt identical ciphertext to identical cleartext. This only constituted the outer layer of the encryption function, however, so we moved on to the stream cipher.
Reviewing stream cipher security
A stream cipher is a symmetric key cipher where plaintext digits are combined with a pseudorandom cipher digit stream (keystream). In a stream cipher, each plaintext digit is encrypted one at a time with the corresponding digit of the keystream, to give a digit of the ciphertext stream. - Wikipedia
In practice, a static key is typically used to seed an algorithm that produces the keystream, and XOR is the operation used to combine the keystream with the cleartext to produce ciphertext.
FIGURE 5 - Basic stream cipher encryption (image credit: The SSL Store)
This describes the inner loop of the Fortinet encryption function reasonably well, with one crucial difference: the “keystream” in this cipher consists solely of the 32-byte key repeated 16 times. Similar to what we saw with the block cipher, repetition of the key can produce recognizable patterns in the ciphertext. Furthermore, because of the mathematical properties of the XOR operation, the repetition allows us to conduct a known-plaintext attack against any 32 bytes of the ciphertext and recover the key (more on this below).
Identifying cipher weaknesses
To summarize what we learned about the Fortinet encryption cipher after reviewing standard encryption techniques:
- The block cipher used in the outer loop lacks chaining to prevent recognizable patterns between 512-byte blocks of ciphertext.
- The stream cipher used in the inner loop lacks a pseudo-random keystream to mitigate known-plaintext attacks against each 32-byte sequence within the ciphertext.
We therefore concluded that it was possible to perform a known-plaintext attack against any 32 bytes in the ciphertext to recover the encryption key, which we could then use to decrypt the entire file.
Since we had access to encrypted and cleartext firmware images, all we had to do was compare them and make an educated guess about where we could predict 32 bytes of cleartext that would be the same in every decrypted image.
Conducting pattern analysis
We returned to our earlier comparison of the file headers of two firmware images:
❯ xxd -l 80 FGT_30E-v6-build0076-FORTINET 00000000: 0000 0000 0000 1100 0000 0000 ff00 aa55 ...............U 00000010: 4647 5433 3045 2d36 2e30 302d 4657 2d62 FGT30E-6.00-FW-b 00000020: 7569 6c64 3030 3736 2d31 3830 3332 392d uild0076-180329- 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ ❯ xxd -l 80 FGT_100D-v6-build9451-FORTINET.out 00000000: 90d0 b0f1 bcda 8be8 85bb f79a f6bc 4c40 ..............L@ 00000010: 7f6e 474e 3d2f 0001 2a10 3036 675c 4796 .nGN=/..*.06g\G. 00000020: 8ca7 ab8e f2af d78e ded2 a9f4 acd5 a3f7 ................ 00000030: 8ed6 aef7 d48a f0ab deb4 c095 bae8 a6e9 ................ 00000040: 86c6 a6e7 aacc 8eed 80be f29f f4be f89f ................
As we anticipated, patterns were evident even within the first 80 bytes. The cleartext image began with a series of six null bytes, followed by a single byte value, five more null bytes, then four “magic bytes” that seemed to serve the purpose of a file signature. The next 32 bytes comprised part of the image name (with at least 30 printable characters) and were followed by 32 null bytes.
We compared the headers of several cleartext images and found this pattern to be remarkably consistent. We were able to confirm that the magic bytes were always present and immediately followed by the letters “FG” (if the product was FortiGate). The rest of the image name was not always formatted consistently, but always contained the word “build.” Null bytes from offsets 48 to 79 were consistent as well.
One important thing we noted, however, was that this “file header” did not always appear at the beginning of the file. Several firmware images had one or more blocks with other content (often just null bytes) preceding it, but this header always appeared at a 512-byte block boundary somewhere in the file.
Thus, our strategy became apparent:
- Read 32 bytes from the first 512-byte block of ciphertext, starting at offset 48.
- Encrypt each of these bytes with its corresponding known plaintext (in each case, a null byte) to produce a key.
- Use the key to decrypt the first 80 bytes of the block and validate that the content matches the standard file header:
- 4 “magic bytes” at offset 12
- 30 printable characters at offset 16
- The word “build” somewhere in that 30-character string
- Repeat the above for each 512-byte block in the file until a valid key is found.
- Use the valid key to decrypt the entire firmware image.
Recovering Keys
We wrote some Python code to perform the key recovery attack against bytes 48–79 from any 512-byte block. This started with a function to perform the byte-level XOR operation, which you will recognize as the same code used in our encrypt function above, but with the cleartext and key bytes swapped:
def derive_key_byte(
key_offset, ciphertext_byte, previous_ciphertext_byte, known_plaintext
):
key_byte = (
previous_ciphertext_byte ^ (known_plaintext + key_offset) ^ ciphertext_byte
)
key_byte = (key_byte + 256) & 0xFF # mod 256 to loop negatives
return key_byte
This works because the following XOR operations are all true (we’re using a caret to represent XOR here, since that is what Python code uses):
A ^ B = C
A ^ C = B
B ^ C = A
If we substitute the variable letters with the components of our encryption cipher, it gets a little more interesting:
key ^ cleartext = ciphertext (encrypt)
key ^ ciphertext = cleartext (decrypt)
cleartext ^ ciphertext = key (recover)
Thus, if we know the cleartext and ciphertext, we can derive the key simply by feeding them into the encryption algorithm. Note that this can become more complicated when the encryption cipher does more than a simple XOR operation: decryption must perform the mathematical inverse of the encryption operations, while recovery can simply reuse the encryption operations.
The next function we wrote implemented recovery of the full key:
def derive_block_key(ciphertext):
key = bytearray()
known_plaintext = 0x00
# Derive the key for this block
for i in range(32):
key_offset = (i + 16) % 32 # mod 32 to wrap around key
plaintext_offset = i + 48
ciphertext_byte = ciphertext[plaintext_offset]
previous_ciphertext_byte = ciphertext[plaintext_offset - 1]
key.append(
derive_key_byte(
key_offset, ciphertext_byte, previous_ciphertext_byte, known_plaintext
)
)
key = key[16:] + key[:16] # swap the first/second halves of the key
return key
You will note that we reversed the first and second halves of the key after recovery – this is because of the position of the known plaintext we chose. Offset 48 from the start of the ciphertext block corresponds to offset 16 from the start of the key, so we actually recover the last half of the key before the first half.
The encrypt function adds the offset from the start of the key to the value of the key byte before XOR’ing it with the other inputs (it is unclear to us why this was done), so we have to be sure we encrypt the correct key offset along with each ciphertext byte in order to derive the correct key byte.
After deriving the full key for the block, we used a modified version the decrypt function above to decrypt the first 80 bytes of the block, and wrote a validation function to check if it contained the expected content:
def validate_decryption(cleartext):
if (
# Length must be at least 80 chars
len(cleartext) >= 80
# Validate the file signature "magic bytes"
and cleartext[12:16] == b"\xff\x00\xaa\x55"
):
# Make sure the image name is readable
try:
image_name = cleartext[16:46].decode("utf-8", errors="strict")
except:
return False
# Make sure the word "build" is in the image name
if "build" in image_name.lower():
# Valid Fortinet image
return True
# Unknown format
return False
We ran the code against the first 512-byte block from one of the encrypted firmware images and confirmed that it successfully recovered a valid key:
❯ ./decrypt.py [+] Decrypting ./FGT_100D-v6-build9451-FORTINET.out [+] Loaded image data [+] Found key: oAbBIcDde7FfgGHhiIjJ7KlLmsnN3OPP [+] Validated: FG100D-6.04-FW-build1966-23031
As we discovered during our analysis, the “file header” did not always appear at the start of every file, so it was necessary to split the encrypted image into 512-byte blocks and attempt to derive a key from each block. Fortunately, the weak block cipher did not prevent us from processing the blocks in parallel.
The function below uses the Python multiprocessing library to do just that (and terminate as soon as it finds a valid key):
def derive_key(ciphertext):
# Determine the number of blocks to read
num_blocks = (len(ciphertext) + BLOCK_SIZE - 1) // BLOCK_SIZE
block_header_size = 80
# Create a pool of worker processes
with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
# Start the workers
results = [
pool.apply_async(
derive_block_key,
( # Each worker attacks the 80-byte header of a 512-byte block
ciphertext[
block_num * BLOCK_SIZE : block_num * BLOCK_SIZE
+ block_header_size
],
),
)
for block_num in range(num_blocks)
]
# Look for a successful result
for result in results:
key = result.get()
if key:
# Kill the workers as soon as we find a valid key
pool.terminate()
pool.join()
return key
return None
Decrypting Firmware
Having successfully cracked Fortinet’s encryption scheme, we just had to put all the pieces together into an easy-to-use decryption tool. We are pleased to release the fruits of our labor to the public: download FortiCrack today!
❯ ./forticrack.py FGT_100D-v6-build9451-FORTINET.out ___ __ __ ___ __ __ __ |__ / \ |__) | | / ` |__) /\ / ` |__/ | \__/ | \ | | \__, | \ /~~\ \__, | \ [+] Decrypting FGT_100D-v6-build9451-FORTINET.out [+] Loaded image data [+] Found key: oAbBIcDde7FfgGHhiIjJ7KlLmsnN3OPP [+] Validated: FG100D-6.04-FW-build1966-23031 [+] Decrypted: FGT_100D-v6-build9451-FORTINET.decrypted
Once we had a working decryption tool, all that was left was to derive keys for all the firmware images and decrypt them. For each image, after deriving its key, we used multiprocessing to decrypt each 512-byte block in parallel, then reassemble the decrypted file. As a result, most firmware images completed decryption in less than a minute.
Decryption of the FortiGate images proceeded smoothly, but we ran into hiccups with some of the other product lines that required us to tweak the decryption algorithm. Although most products used a similar 80-byte “file header,” small differences in the known plaintext for some, such as the offset of the first non-null byte or the order of the magic bytes (big endian vs. little endian for different architectures), required extra steps in the key recovery process.
In the end, our efforts were successful – here is the final tally of the Fortinet firmware images we decrypted:
FIGURE 6 - Total numbers of decrypted Fortinet firmware images
Some other interesting trends emerged at this stage. From the approximately 29k images we decrypted across 28 product lines, there were only 25 unique encryption keys, indicating rampant key reuse. Furthermore, we discovered that valid keys used only alphanumeric characters, so we added an additional key validation step (prior to decryption and content validation) to speed up the key recovery process.
In the end, we found that many of the keys were, in fact, hard coded within Fortinet firmware images, but we were unable to recognize them as keys before we put in the effort to determine their characteristics. Nevertheless, our work provided us with a reliable way to recover the keys whether or not they were hard coded.
Extracting File Contents
For those following along at home, we will also show how to extract the file contents from a decrypted firmware image. Typically, the easiest way is to mount the image file on a Linux system – most of the images have the base filesystem starting at offset 512, right after the MBR data. For example:
❯ mkdir firmwarefs ❯ sudo mount -o ro,loop,offset=512 FGT_100D-v6-build9451-FORTINET.decrypted firmwarefs ❯ ls -al firmwarefs total 48708 drwxr-xr-x 8 root root 4096 Aug 19 2020 . drwxrwxrwx 1 root root 4096 Jul 11 09:20 .. drwxr-xr-x 2 root root 4096 Aug 19 2020 bin drwxr-xr-x 2 root root 4096 Aug 19 2020 cmdb drwxr-xr-x 2 root root 4096 Aug 19 2020 config -rw-r--r-- 1 root root 6110 Aug 19 2020 devicetree.dtb drwxr-xr-x 11 root root 4096 Aug 19 2020 etc -rw-r--r-- 1 root root 86 Aug 19 2020 filechecksum -rwxr-xr-x 1 root root 3319135 Aug 19 2020 flatkc -rw-r--r-- 1 root root 256 Aug 19 2020 flatkc.chk drwxr-xr-x 2 root root 4096 Aug 19 2020 lib drwx------ 2 root root 16384 Aug 19 2020 lost+found -rw-r--r-- 1 root root 46431042 Aug 19 2020 rootfs.gz -rw-r--r-- 1 root root 256 Aug 19 2020 rootfs.gz.chk
Some images do not follow this convention, in which case you may need to manually locate the beginning of the filesystem. For example, the firmware image in the figure below has a filesystem that starts at offset 0x400000:
❯ xxd FMG_300F-v7.4.0-build2223-FORTINET.decrypted | grep -A10 ": 0000 0100 0000 04"
00400400: 0000 0100 0000 0400 0000 0000 c7fe 0200 ................
00400410: f1ff 0000 0000 0000 0200 0000 0200 0000 ................
00400420: 0080 0000 0080 0000 0020 0000 89aa 6164 ......... ....ad
00400430: 8baa 6164 0100 ffff 53ef 0100 0100 0000 ..ad....S.......
00400440: 89aa 6164 0000 0000 0000 0000 0100 0000 ..ad............
00400450: 0000 0000 0b00 0000 8000 0000 3c00 0000 ............<...
00400460: 0200 0000 0300 0000 d0a3 e29e 3e86 4a80 ............>.J.
00400470: acfa 042c 4472 6d8c 464f 5254 495f 424f ...,Drm.FORTI_BO
00400480: 4f54 5f44 4556 0000 2f63 6f64 652f 466f OT_DEV../code/Fo
00400490: 7274 694d 616e 6167 6572 2f73 6572 7665 rtiManager/serve
004004a0: 722f 6d6e 7400 0000 0000 0000 0000 0000 r/mnt...........
...omitted for brevity...
❯ sudo mount -o loop,ro,offset=0x400000 FMG_300F-v7.4.0-build2223-FORTINET.decrypted firmwarefs
❯ ls -al firmwarefs
total 221036
drwxr-xr-x 2 root root 4096 May 14 20:44 .
drwxr-xr-x 4 user user 4096 Jul 19 14:50 ..
-rw-r--r-- 1 root root 4066471 May 14 20:44 flatkc
-rw-r--r-- 1 root root 98350388 May 14 20:44 rootfs-ext.tar.xz
-rw-r--r-- 1 root root 70010824 May 14 20:44 rootfs.gz
-rw-r--r-- 1 root root 53657084 May 14 20:42 syntax.tar.xz
-rw-r--r-- 1 root root 122 May 14 20:44 widgets.tar.gz
The rootfs.gz file is a gzipped CPIO archive as discussed above and can be extracted like so:
❯ mkdir unpack ❯ gzip -d rootfs.gz ❯ cpio --no-absolute-filenames -D unpack -i <rootfs ❯ ls -al unpack total 33832 drwxr-xr-x 14 user user 4096 Jul 11 13:17 . drwxr-xr-x 7 user user 4096 Jul 11 13:17 .. -rw-r--r-- 1 user user 25817736 Jul 11 13:17 bin.tar.xz drwxr-xr-x 2 user user 4096 Jul 11 13:17 boot drwxr-xr-x 3 user user 4096 Jul 11 13:17 data drwxr-xr-x 2 user user 4096 Jul 11 13:17 data2 drwxr-xr-x 7 user user 4096 Jul 11 13:17 dev lrwxrwxrwx 1 user user 8 Jul 11 13:17 etc -> data/etc -rw-r--r-- 1 user user 256 Jul 11 13:17 .fgtsum lrwxrwxrwx 1 user user 1 Jul 11 13:17 fortidev -> / lrwxrwxrwx 1 user user 10 Jul 11 13:17 init -> /sbin/init drwxr-xr-x 2 user user 4096 Jul 11 13:17 lib -rw-r--r-- 1 user user 8737348 Jul 11 13:17 migadmin.tar.xz drwxr-xr-x 4 user user 4096 Jul 11 13:17 node-scripts drwxr-xr-x 2 user user 4096 Jul 11 13:17 proc drwxr-xr-x 2 user user 4096 Jul 11 13:17 sbin drwxr-xr-x 2 user user 4096 Jul 11 13:17 sys drwxr-xr-x 2 user user 4096 Jul 11 13:17 tmp drwxr-xr-x 3 user user 4096 Jul 11 13:17 usr -rw-r--r-- 1 user user 20360 Jul 11 13:17 usr.tar.xz drwxr-xr-x 8 user user 4096 Jul 11 13:17 var
As mentioned above, the .tar.xz files inside the archive are in a format customized by Fortinet, and so can only be unpacked using the ftar and xz binaries included in the sbin
directory.
Responsible Disclosure
After completing our decryption and analysis of Fortinet products, we reported the issue to the vendor. The response from Fortinet PSIRT was twofold. On the one hand, they informed us that they did not consider the weak encryption to be a vulnerability:
"Encryption here is not used for confidentiality because of the availability of VM images. Creating a malicious images (sic) and running it on the device is not possible because of image signing and verification." - Fortinet PSIRT
On the other hand, they promptly locked down access to firmware downloads, limiting each account to products with active licenses. As a trial user, you can now only download virtual machine images.
Disclosure timeline:
Initial report |
5/26/2023 |
Initial vendor response requesting clarification |
6/3/2023 |
Clarification provided |
6/12/2023 |
Final vendor response |
6/26/2023 |
Conclusion
Regardless of Fortinet’s stance on the matter, breaking encryption on the firmware images allowed our team to derive several benefits from the research effort:
- Detection: Fortinet products are not always easy to identify on the public internet. Analyzing a large number of firmware images allowed us to develop new techniques to find these devices when they are deployed with publicly accessible interfaces.
- Fingerprinting: Most Fortinet products do not advertise their running software versions or the hardware they are deployed on. Our analysis produced a new technique for precisely identifying hardware and software versions of FortiGate and FortiProxy appliances without authenticated access.
- Exploit development: With access to the entire library of Fortinet firmware images across multiple product lines, our team is now uniquely positioned to conduct ongoing research. When new vulnerabilities are discovered, such as CVE-2023-27997, we can develop scanners and exploits very quickly, and in time we may uncover 0-day vulnerabilities as well.
This is great for us, because at Bishop Fox we love doing this kind of research – but the real value goes to Cosmos customers, who benefit by having greater knowledge of their attack surface, more precise information about the risk of exposure, and an improved understanding of the impact of exploitation. All of this information helps our customers advocate for the upgrades and policy changes they need to stay safe and secure – which, at the end of the day, is why we do what we do.
Subscribe to Bishop Fox's Security Blog
Be first to learn about latest tools, advisories, and findings.
Thank You! You have been subscribed.
