On May 22, 2022, the Kernel Key Retention Service (KKRS) was released as part of Linux 5.18. This introduced an in-kernel key management and retention feature, which provided a secure method for storing secrets on Linux systems. With this new feature, it was now possible for processes to process cryptographic material without having direct access to the key material. Instead, the kernel would handle all cryptographic operations and key material access. By storing keys in the kernel, typical user-land processes can not retrieve most keys, and it can be challenging for even root to retrieve the most keys.

This is a huge pain for the security assessment team at ivision! When we assess an embedded system, one of our goals is to recover key material and investigate the downstream threats from an attacker obtaining that key material. The KKRS makes that much more challenging! Which is good, from a defender’s perspective. We recommend that everyone use the KKRS when building embedded systems. But from an attacker’s perspective, recovery of key material stored in the KKRS is difficult. The KKRS was designed to prevent recovery, so attackers must implement a custom key recovery feature. Furthermore, since the KKRS is a Linux kernel feature, the key recovery feature must be implemented as a Linux kernel module. On mainline X86_64 Linux kernels, this is already a challenge but throw in the need to recover key material from embedded systems, which typically lack kernel headers, and it becomes a more difficult challenge to navigate.

In this two part blog series, we will show how to recover keys from the KKRS. In this first part, we will focus on key recovery from a mainline kernel with access to Linux kernel headers on x86_64. In Part 2, we will cover how to recover keys from a mainline kernel without headers on AArch64.

Getting set up

If you want to follow along, we’ve put together a simple Linux image you can boot in QEMU. You can download that resource and all other mentioned resources here

It is using the KKRS to store the following keys:

Key Type Key Name Key Value Session Key ID
user user_key need pentesting? 1011652325
logon logon_key contact us at ivision.com 171149254
asymmetric x509 asym_x509 a public key 588852448
asymmetric pkcs #8 asym_pkcs8 a private key 1033559895

Once QEMU is running you can run keyctl show to get a list of keys stored in the KKRS.

Note: The Key ID values will change for each reboot. We’ll use the following throughout this blog post

# keyctl show
Session Keyring
 850642432 --alswrv      0 65534  keyring: _uid_ses.0
 389366852 --alswrv      0 65534   \_ keyring: _uid.0
 1011652325 --alswrv      0     0       \_ user: user_key
 1033559895 --als--v      0     0       \_ asymmetric: asym_pkcs8
  588852448 --als--v      0     0       \_ asymmetric: asym_x509
  171149254 --alsw-v      0     0       \_ logon: logon:logon_key

Using the Key Retention Service: Basics

For our purposes we only need to understand enough to retrieve stored key material. In general, KKRS is a keyring or a password manager for the kernel. That means it stores all key material in kernel memory. This is important because it means that key material never touches the disk! It only resides in kernel memory. It is up to the system owner to figure out how to get the key into memory securely. When user-space applications want to perform cryptographic operations, they have to ask the kernel, and it will decide how/if the keys can be used. In some cases, the KKRS may return the key material entirely to users, but more commonly, the KKRS will only allow user-space applications to encrypt/decrypt data using keys stored in the kernel. The exact actions allowed depend on the key type.

All keys stored within the KKRS are assigned a type. The KKRS is built to be extendable, so various types can exist, however, for our cases, we’ll focus on the following key types:

Key Type Description Notes
Keyring Special key type to hold more keys None
User General purpose key Contents can be read by user-land processes
Logon General purpose key Contents can not be read
asymmetric Supports public-key cryptography Contents can not be read but can be used for encryption/decryption processes

One of these key types, user, makes it simple to retrieve the key material. We can just use the following command keyctl print <keyid>, where the KeyID is a unique identifier for each key. This identifier can be retrieved using keyctl show. This will output the keys accessible to the current user in a tree-style hierarchy to help visualize which keyring holds which keys. The keyID is the left most numeric value

# keyctl show
Session Keyring
 850642432 --alswrv      0 65534  keyring: _uid_ses.0
 389366852 --alswrv      0 65534   \_ keyring: _uid.0
 1011652325 --alswrv      0     0       \_ user: user_key
 1033559895 --als--v      0     0       \_ asymmetric: asym_pkcs8
  588852448 --als--v      0     0       \_ asymmetric: asym_x509
  171149254 --alsw-v      0     0       \_ logon: logon:logon_key

# keyctl print 1011652325
need pentesting?

For our purposes that’s all we need to know. If you’re interested in a deeper dive into the KKRS, we suggest checking out the following resources:

Exploring the Key Retention Service Implementations

As far as KKRS basics go, that is the gist of it. Our goal is to retrieve key material stored in non-readable key types (anything that’s not user). How would we achieve that? The KKRS is designed such that user-land process are unable to retrieve key material, however, the kernel stores to the key material in memory. This is important - we’ll need to execute within the context of the kernel. Once in the kernel context, we can explore the kernel memory to retrieve the key material, bypassing all controls implemented by the KKRS. To obtain this level of access, we’ll build a custom kernel module which will do the following:

  1. Retrieve a pointer to the key struct
  2. Traverse the key memory structure
  3. Dump the key material for our usage

In order to achieve these goals however we will need to get intimate with the KKRS’ implementation. For that we will need to take a look at the Linux Kernel source code and understand the underlying implementation of the KKRS. The QEMU image provided alongside this blog uses uses version 6.11.0-rc6-g9d4c304001cf of the Linux Kernel, so we’ll be using that as well. You can view the source here.

Retrieve a pointer to the key struct

The first obstacle we need to overcome is finding the key material in memory. Ideally, we could pass a custom kernel module the KeyID of a key we want to recover and it will just work. To make that happen, we need to take a look at how the KKRS is implemented and see if there are any methods that will just give us a pointer to a key. The KKRS key implementation can be found at linux/security/key/key.c. If we look through the source, we will see there’s a key_lookup function, which does exactly what we want—it gives us a pointer to a key struct in memory. We just need to give it the key serial, but that’s easy enough to grab from user space using keyctl show. It is important to note that key_lookup returns a pointer to struct key. In the next section we will take a close look at the struct key type.

Traverse the key memory structure

We now have a method of gaining a pointer to a key struct in memory, but we’ll need to figure out where in the key struct the cryptographic key material is stored. Let’s start by taking a look at the struct key type. This is defined in include/linux/key.h. Reading the source is useful, and the comments make it clear that the union key_payload field holds the cryptographic key material. The KKRS documentation states the following about the payload field:

The simplest payload is just data stored in key->payload directly. In this case, there’s no need to indulge in RCU or locking when accessing the payload. More complex payload contents must be allocated and pointers to them set in the key->payload.data[] array.

This implies that the key material is always stored within a pointer in the key->payload.data array. However, since it is a void pointer, it can be of any type. We will need to figure out the pointer type for each of the different key types. To do so, we will need to review each key type implementation and figure out how each type stores key material. Covering each key type would take a while, so instead the following table shows where key material is stored for each type:

Key Type Key Material Type Key Material Location Underlying Memory Structure
User Entire Key Key->payload.data[0] Raw bytes
Logon Entire Key Key->payload.data[0] Raw bytes
Asymmetric - x509 Public Key Certificate Key->payload.data[0]->key public_key
Asymmetric - PKCS #8 Private Key Key->payload.data[0]->key public_key (with private key)

Note: For this blog, we’ll only focus on recovery of RSA keys.

For the curious, this table was created by:

  1. Identifying each type’s struct key_type implementation. This structure is implemented by all key types and is used to define a new type of supported key.
  2. Review key_type.preparse and key_type.instantiate for each type. The preparse function directly passes the user supplied data to the key type implementation and enables it to pre-process the key data. The instantiate function is responsible for actually creating the key struct and storing the key material.

Now we can actually dump the keys.

Dumping the Key Material

Now that we have a way to find our keys in kernel memory and know how each type stores its key material, it’s time to actually dump the keys. For this, we will need to build a Linux kernel module. Luckily, we wrote the code already and you can find it here. The implementation itself is pretty straightforward, so we will not review it line by line, but instead cover at a high level what it does:

  1. It obtains a reference to the key in memory via the key_lookup function
  2. It passes the key->payload value to an appropriate key type dump function based on the key type.
  3. The key type’s dump function contains type specific key material recovery logic
  4. The contents are dumped to the kernel log as a hex dump

Note: Remember, it’s a Linux kernel module, so if you want to use it on your own system, make sure you update the Makefile to point to your kernel’s headers.

We have already built the Linux kernel module and you can find it at /root/keydump.ko within the accompanying QEMU image. Speaking of the accompanying, QEMU image, let’s jump over to that image and talk about how to use the keydump.ko kernel module.

First we need to load the module. We’ll pass in two parameters: keytype and keyid.

# insmod /root/keydump.ko keytype="user" keyid="1011652325"

Once loaded, we can view the resulting key dump using dmesg:

# dmesg
[ 7004.792704] USER/LOGON - KEY:6e 65 65 64 20 70 65 6e 74 65 73 74 69 6e 67 3f                                                  need pentesting?

To view another key’s content, we need to unload the module first.

# rmmod keydump

Let’s talk about the output for each of the supported types.

User:

# insmod /root/keydump.ko keytype="user" keyid="1011652325"
# dmesg

[ 7004.792704] USER/LOGON - KEY:6e 65 65 64 20 70 65 6e 74 65 73 74 69 6e 67 3f                                                  need pentesting?

Logon:

# insmod /root/keydump.ko keytype="logon" keyid="171149254"
# dmesg
[ 7091.147014] USER/LOGON - KEY:72 65 61 63 68 5f 6f 75 74 5f 74 6f 5f 69 76 69 73 69 6f 6e                                      reach_out_to_ivision

Asymmetric - X.509:

X.509 certificates only store the public key, not the private key. The hex dump is an ASN.1 encoded subjectpublickeyinfo object.

# insmod /root/keydump.ko keytype="asymmetric" keyid="588852448"
# dmesg
[ 7241.728426] Public - ASYM KEY:30 82 01 0a 02 82 01 01 00 b9 b2 e1 11 ab 58 30 b6 92 2a 27 56 5e 06 2c eb fe 81 72 23 1c e9 69  0.............X0..*'V^.,...r#..i
[ 7241.728956] Public - ASYM KEY:c4 06 8e d7 77 59 f3 0e 0e 81 51 6c 2f 13 10 60 39 e5 40 0a 06 d8 24 b5 e8 71 18 be dc 95 61 47  ....wY....Ql/..`9.@...$..q....aG
[ 7241.729334] Public - ASYM KEY:df ac a5 e1 9f 65 95 5d bf 23 5e d1 b4 5b 01 d0 52 a8 10 49 b7 25 60 d9 d0 cd 75 07 73 93 64 cb  .....e.].#^..[..R..I.%`...u.s.d.
[ 7241.729656] Public - ASYM KEY:59 07 94 40 e7 e9 6d 38 8d c1 39 7b 67 56 17 07 f6 11 22 88 da 69 c7 ec cf c2 4e 97 9b a9 3b 63  Y..@..m8..9{gV...."..i....N...;c
[ 7241.730113] Public - ASYM KEY:78 67 3c d0 e7 fc 97 e5 79 fd ce 9f 27 45 c8 dc b6 17 4e bb 61 45 5f ba 02 ad 1a b2 50 6a a0 6e  xg<.....y...'E....N.aE_.....Pj.n
[ 7241.730432] Public - ASYM KEY:1e 7f e3 17 47 9c 89 97 8d fd 4e 87 f8 b7 76 54 da 8a fd 7e 49 96 e0 dd f0 5b 70 2f 5c 8d 67 af  ....G.....N...vT...~I....[p/\.g.
[ 7241.730749] Public - ASYM KEY:22 0d a9 8a d1 29 81 37 76 08 99 70 04 ad 40 e9 5f 8c 11 3c 06 8e 6f b4 fe 09 68 69 9c ea 9e 41  "....).7v..p..@._..<..o...hi...A
[ 7241.731262] Public - ASYM KEY:a4 fc ed 60 71 88 34 5e 19 ff d4 23 d4 b2 2a bc 46 e3 a8 68 e0 55 8b c9 75 3b df 9a 36 67 ca a4  ...`q.4^...#..*.F..h.U..u;..6g..
[ 7241.731603] Public - ASYM KEY:13 b9 11 ca 78 a9 c7 2f 87 02 03 01 00 01                                                        ....x../......

We need to convert it back into a usable PEM format. We can achieve this with:

echo "30 82 01 ...KEY HEX DUMP" | xxd -r -ps | openssl rsa -pubin

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAubLhEatYMLaSKidWXgYs
6/6BciMc6WnEBo7Xd1nzDg6BUWwvExBgOeVACgbYJLXocRi+3JVhR9+speGfZZVd
vyNe0bRbAdBSqBBJtyVg2dDNdQdzk2TLWQeUQOfpbTiNwTl7Z1YXB/YRIojaacfs
z8JOl5upO2N4ZzzQ5/yX5Xn9zp8nRcjcthdOu2FFX7oCrRqyUGqgbh5/4xdHnImX
jf1Oh/i3dlTaiv1+SZbg3fBbcC9cjWevIg2pitEpgTd2CJlwBK1A6V+METwGjm+0
/gloaZzqnkGk/O1gcYg0Xhn/1CPUsiq8RuOoaOBVi8l1O9+aNmfKpBO5Ecp4qccv
hwIDAQAB
-----END PUBLIC KEY-----

Asymmetric - PKCS #8:

The PKCS #8 contains the private key (which be used to derive the public key), and is an ASN.1 encoded RSAPrivateKey object.

# insmod /root/keydump.ko keytype="asymmetric" keyid="1033559895"
# dmesg
[ 4868.115468] Private - ASYM KEY:30 82 04 a4 02 01 00 02 82 01 01 00 b9 b2 e1 11 ab 58 30 b6 92 2a 27 56 5e 06 2c eb fe 81 72 23  0................X0..*'V^.,...r#
[ 4868.116046] Private - ASYM KEY:1c e9 69 c4 06 8e d7 77 59 f3 0e 0e 81 51 6c 2f 13 10 60 39 e5 40 0a 06 d8 24 b5 e8 71 18 be dc  ..i....wY....Ql/..`9.@...$..q...
(...30 Lines Skipped)
[ 4868.129325] Private - ASYM KEY:43 6a ff 4b 71 02 81 80 06 71 05 dc 21 85 f8 81 a6 c0 33 cc 53 f3 ce 20 70 48 df f2 7f c9 5d 55  Cj.Kq....q..!.....3.S.. pH....]U
[ 4868.129643] Private - ASYM KEY:7c bd cf 96 df 87 51 5e aa 38 fb 09 e1 20 d1 74 44 ac 80 0e 66 12 5f b1 47 5a 4f fd ea 3c ec 98  |.....Q^.8... .tD...f._.GZO..<..
[ 4868.130121] Private - ASYM KEY:b2 00 6a b0 93 99 3d ee 62 02 02 94 31 13 38 f5 cb ff c6 82 bd 83 1f 57 47 53 ae d5 ca 88 e3 3d  ..j...=.b...1.8........WGS.....=
[ 4868.130444] Private - ASYM KEY:5c aa 95 44 f1 8c 69 01 0a 6e 2e 73 b5 13 bf 54 1d 67 22 e4 df 2f 16 13 8c 7c a6 27 70 a4 93 c1  \..D..i..n.s...T.g"../...|.'p...
[ 4868.130764] Private - ASYM KEY:6a 36 87 12 86 43 98 16                                                                          j6...C..

We need to convert it to back to into a usable PEM format. We can achieve this with:

echo "30 82 01 ...KEY HEX DUMP" | xxd -r -ps | openssl rsa

-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC5suERq1gwtpIq
J1ZeBizr/oFyIxzpacQGjtd3WfMODoFRbC8TEGA55UAKBtgktehxGL7clWFH36yl
4Z9llV2/I17RtFsB0FKoEEm3JWDZ0M11B3OTZMtZB5RA5+ltOI3BOXtnVhcH9hEi
iNppx+zPwk6Xm6k7Y3hnPNDn/Jflef3OnydFyNy2F067YUVfugKtGrJQaqBuHn/j
F0eciZeN/U6H+Ld2VNqK/X5JluDd8FtwL1yNZ68iDamK0SmBN3YImXAErUDpX4wR
PAaOb7T+CWhpnOqeQaT87WBxiDReGf/UI9SyKrxG46ho4FWLyXU735o2Z8qkE7kR
ynipxy+HAgMBAAECggEAXE9BdokYY869xdqcIk0rk3yGWHh/7L+4xBHJpfi+y+QG
ALWoiO+BBgah2NoiI82RaHcGmZxgKVxU9Hi9xb+ERHmOruvauYIXNJg0LKiWBnC3
UU+M8l4Of4k29zTRYovUW5L5dxrihACb7/Dbse3CGFzcuJyWeYEu3xLqPgfgch9i
GrY+vZuIzMo7Jb8MR/L8z4MPsEPtH8qJRMzGOMvdl3KWLkVwHnd125XGPsnOuzDp
bzUQ+V0Mne3gBhnqaCsS2DfKCKo+zssYuoRflodevIvVmjZjkdpxOYRHXBtAy0FF
68FlqQKANwAhUWwDmOg4P6WLXAiDs5TP5w3rKU4ZwQKBgQDz7RxBRpPlZQq1XYgp
jyjMQuOzE8vVnwMx96A/vhJ7P0eRLrSBiGrdEFsJ0FCzpzyWof3pjGL5XxdXJZkv
pfhAyTFiZYup5sURoHr1qqtg3zoNkcENB1e2W/CbleGQ4eGIjAKvyQitD9BKrjK0
GetIAdVMmm0lkLgSH45FaeY9VwKBgQDC4/HPunJZnlE7Xl18QbqmZ3MmH7CefeNs
2rYTaHPCjtybybvq1qRJXS96JJBUPqREi5GNSy7ANIHhpegeTS1cu27jQ1jKADcf
6yiOTB1NqWPrEdn3NHPBDWq2TUYSV8I1ZERGOcVvRqnQac7s0+wXkimF8HBEM14j
q6PEjR0RUQKBgQDY5bchKAPsj7tE6CViZwTtFHFqVCkFC/8IWFtrMrU2WF9n7nLd
V5NGfPumr1CDXyrV+rdQC+AaQ/76I9mh79/eo8hA6XCWJDiJ4vxTJG59liEF3dwe
nK/W3MwHXPAVJ3WnOHBPJOCNGqmpLlqxoPaAsfibxBlXtOaoXtzgRjlPwwKBgQC+
BPgxfO2feaBkTVbDH2c4dicspbtUXRQQf5MFi1NLAHYSo3hIua0HJwdyhRIAQe9y
Mc7hv8s+djOo6lVOrhsrjkjI51I76kfVJDivvYDYxu74NLibshWmxkkkpGVM3yop
WrDC8/MP0wshfPjXqq1IMiewQ0WDCb6g8upDav9LcQKBgAZxBdwhhfiBpsAzzFPz
ziBwSN/yf8ldVXy9z5bfh1Feqjj7CeEg0XRErIAOZhJfsUdaT/3qPOyYsgBqsJOZ
Pe5iAgKUMRM49cv/xoK9gx9XR1Ou1cqI4z1cqpVE8YxpAQpuLnO1E79UHWci5N8v
FhOMfKYncKSTwWo2hxKGQ5gW
-----END PRIVATE KEY-----

Making it Portable

This blog post started with a rant and two problem statements. We’ve reviewed a solution to one of those problems: recovering key material. The process here, however, relies on importing kernel headers, which on production embedded systems are likely to be missing. In the next blog post, we’ll talk about how to cross-compile kernel modules for AARCH64 without needing to depend on Linux kernel headers being preset.