In the last entry of this series, we created a Linux kernel module which was able to dump keys from the Linux Kernel Key Retention Service. The kernel module was limited to only x86_64 machines with a mainline Linux kernel. It also required having the appropriate Linux kernel headers. Access to Linux kernel headers is typically not available for embedded systems, making our Linux kernel module useless on most embedded systems. In this blog post, we will fix that by creating a kernel module which will not require any Linux kernel headers. We will also cross-compile the kernel module so that it can run on AArch64 based systems, which tend to be a common architecture for embedded systems.

Getting set up

If you want to follow along, please download this blog’s resource here. Included in the download are the following two QEMU images:

  • QEMU x86_64: Based on Linux Kernel 6.6.32
  • QEMU AArch64: Based on Linux kernel 6.6.32

These two images contain all the tools and resources needed for the blog post. Within each image are the following keys:

Key Type Key Name Key Value
user user_key need pentesting?
logon logon_key contact us at ivision.com
asymmetric x509 asym_x509 a public certificate
asymmetric pkcs #8 asym_pkcs8 a private key

Additionally, the source code for a headerless Linux kernel module is included.

What makes a Linux Kernel Module valid

Our goal is to create a loadable kernel module without using any kernel headers. To achieve that, we will need to convince the Linux kernel that the module supplied is in fact a valid Linux kernel module. This begs the question: “What steps does the Linux Kernel take to validate a Linux kernel module”? Perhaps, if we are able to understand each of these validity checks we will be able to build our own file that that satisfies these checks. These checks are all implemented in the elf_validity_cache_copy function. A review of the function indicates that it initially performs the following checks on all to-be-loaded modules:

  1. Validation of ELF File Type
  2. Validation of ELF Section Headers

Note: These are not all the checks done by Linux Kernel, however, satisfaction of these two checks is typically enough to let us load our own kernel module. Curious about what all the checks are? Take a look at the load_module function

Validation of ELF File Type

The first step in validating a kernel module is validating the kernel module’s file type. A review of the checks performed by the elf_validity_cache_copy method indicates that a Linux kernel module must be an ELF type. Let’s take a closer look at that requirement. The elf_validity_cache_copy function implements the following check:

if (info->hdr->e_type != ET_REL) {
		pr_err("Invalid ELF header type: %u != %u\n",
		       info->hdr->e_type, ET_REL);
		goto no_exec;
	}

This indicates that the ELF type must be of type EL_REL (Relocatable) per the elf man page. We can generate ELF relocatable files by using gcc -c. This will create a file, which we can confirm is relocatable via the file command. Actually, let’s make a simple kernel module ivision.c to learn how to meet various Linux kernel module checks.

Note: The completed ivision.c file can be found in this blog’s this blog’s resource here

int main(){
  return 0;
}

Let’s compile ivision.c using gcc -c ivision.c -o ivision.ko. Once created, we can use file to confirm that the resulting file ivision.o is of type ELF ... relocatable.

$ file ivision.o
ivision.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

For now step one is complete. We have met the file type condition. Up next, we need to modify the ELF so that it fits the Linux Kernel’s ELF section header expectations.

Validation of ELF Section Headers

In addition to validating the kernel module is of the correct ELF type, the elf_validity_cache_copy method checks to see if the kernel module contains specific ELF headers. Specifically, it checks for the following section headers:

  • .modinfo
  • .gnu.linkonce.this_module
  • Relocation table for startup and deconstruction functions
  • For AArch64: .plt and .init.plt

All ELF files have sections. Some sections are pretty standard (.text for code, .data for data, etc.) but there can be unique sections as well (such as the above). These unique sections are referenced and required by the Linux kernel. So let’s take a look at each of these sections:

ELF Section — .modinfo

.modinfo is an ELF section containing null-byte separated key=value pairs with module information. Most of the information in this section is not needed, however the following keys/value pairs are important:

  • name: The name of the module. This is what will be used when log messages are printed in any kernel log entries.
  • vermagic: A magic value unique per Linux kernel build which is used by the Linux kernel to validate that the module was built against our specific kernel version.

If we want to build a kernel module, we will need to appropriately populate the .modinfo section. To do that, we need to:

  1. Identify the correct vermagic.
  2. Create a new elf section called .modinfo.
  3. Populate .modinfo with the appropriate keys.

Let’s start by talking about how to find the vermagic. It is used by the Linux kernel to validate the compatibility of a Linux kernel module, so all Linux kernel modules must have the string in their .modinfo section. We can extract the value from an existing Linux kernel module on our test system. Let’s start by finding a suitable module inside our X86_64 QEMU VM:

$ find / | grep ko
/lib/modules/6.6.32/kernel/drivers/char/hw_random/via-rng.ko
/lib/modules/6.6.32/kernel/drivers/char/hw_random/rng-core.ko
/lib/modules/6.6.32/kernel/drivers/char/hw_random/intel-rng.ko
/lib/modules/6.6.32/kernel/drivers/char/hw_random/virtio-rng.ko
/lib/modules/6.6.32/kernel/drivers/char/hw_random/amd-rng.ko

Using amd-rng.ko, we can extract the vermagic by copying amd-rng.ko locally and dumping the contents using readelf.

$ readelf amd-rng.ko -j .modinfo   

Hex dump of section '.modinfo':
  0x00000000 6c696365 6e73653d 47504c00 64657363 license=GPL.desc
  0x00000010 72697074 696f6e3d 482f5720 524e4720 ription=H/W RNG 
  0x00000020 64726976 65722066 6f722041 4d442063 driver for AMD c
  0x00000030 68697073 65747300 61757468 6f723d54 hipsets.author=T
  0x00000040 6865204c 696e7578 204b6572 6e656c20 he Linux Kernel 
  0x00000050 7465616d 00616c69 61733d70 63693a76 team.alias=pci:v
  0x00000060 30303030 31303232 64303030 30373436 00001022d0000746
  0x00000070 4273762a 73642a62 632a7363 2a692a00 Bsv*sd*bc*sc*i*.
  0x00000080 616c6961 733d7063 693a7630 30303031 alias=pci:v00001
  0x00000090 30323264 30303030 37343433 73762a73 022d00007443sv*s
  0x000000a0 642a6263 2a73632a 692a0064 6570656e d*bc*sc*i*.depen
  0x000000b0 64733d72 6e672d63 6f726500 72657470 ds=rng-core.retp
  0x000000c0 6f6c696e 653d5900 696e7472 65653d59 oline=Y.intree=Y
  0x000000d0 006e616d 653d616d 645f726e 67007665 .name=amd_rng.ve
  0x000000e0 726d6167 69633d36 2e362e33 3220534d rmagic=6.6.32 SM
  0x000000f0 50207072 65656d70 74206d6f 645f756e P preempt mod_un
  0x00000100 6c6f6164 2000                       load .

We could have also used the following:

$ strings amd-rng.ko | grep vermagic
vermagic=6.6.32 SMP preempt mod_unload

But make sure to compare the output of both commands. Kernel module vermagics can have a trailing space. With the value obtained, let’s talk about how to create the new section header. We can achieve that by using gcc’s compiler attributes. Specifically, we can use the __attribute__((section("section_name"))) feature to define where certain variables should be located in the ELF header. Let’s update ivision.c to include a .modinfo section which contains a vermagic and name key.

#define NAME "ivision"
#define VERMAGIC "6.6.32 SMP preempt mod_unload "

char modinfo[]  __attribute__((section(".modinfo"))) = "name=" NAME "\x00vermagic=" VERMAGIC;

int main(){
  return 0;
}

Note: Once again, we can compile our snippet using gcc -c ivision.c -o ivision.ko and the use readelf ivision.ko -j .modinfo to make sure the vermagic and .modinfo section were created.

$ readelf amd-rng.ko -j .modinfo   

Hex dump of section '.modinfo':
  0x00000000 6e616d65 3d697669 73696f6e 00766572  name=ivision.ver
  0x00000010 6d616769 633d362e 362e3332 20534d50  magic=6.6.32 SMP
  0x00000020 20707265 656d7074 206d6f64 5f756e6c   preempt mod_unl
  0x00000030 6f616420 00000000 00000000 00000000  oad ............
  0x00000040 00000000 00000000 00000000 00000000  ................
  0x00000050 00000000 00000000 00000000 00000000  ................
  0x00000060 00000000                             ....

ELF Section — .gnu.linkonce.this_module

.gnu.linkonce.this_module is an ELF section with a pre-defined size (unique for each kernel) that is used to store the initialized struct module. This section must be exactly the size of a struct module, which varies depending on the Linux kernel configuration. Except for the module name, this section can typically be filled with zeros and left for the kernel to initialize. Similar to .modinfo, we will need to collect kernel specific information from another Linux kernel module to populate .gnu.linkone.this_module. Specifically, we need to grab:

  1. .gnu.linkonce.this_module ELF section size.
  2. The offset where the module name is stored.
  3. The offsets for the startup and deconstruction functions.

Let’s start by grabbing the .gnu.linkonce.this_module section size. We can once again use readelf to grab the information. Specifically we can use:


$ readelf -S amd-rng.ko -W | grep linkonce_this

  [Nr] Name                      Type            Address          Off    Size   ES Flg Lk Inf Al
  ...
  [26] .gnu.linkonce.this_module PROGBITS        0000000000000000 000900 000440 00  WA  0   0 64

We’re interested in the hex value (0x440) in the Size column.

Next we will want to identify, where the module name offset within the .gnu.linkonce.this_module section. We can also use readelf to get that value, except this time we will manually count the offset:

readelf -j .gnu.linkonce.this_module amd-rng.ko

Hex dump of section '.gnu.linkonce.this_module':
 NOTE: This section has relocations against it, but these have NOT been applied to this dump.
  0x00000000 00000000 00000000 00000000 00000000 ................
  0x00000010 00000000 00000000 616d645f 726e6700 ........amd_rng.

Notice that in the above result, the module name (amd_rng) starts at 0x18 and is null terminated.

Note: You might remember that .modinfo also contained the module name. However, it is important to note that these two names are used for different operations and can be different. The .modinfo name is used as a tag for log messages and the .gnu.linkonce.this_module name is the actual name of the module and used by utilities like lsmod.

The final piece of information we need is the startup/deconstruction method offsets. Now these values are not hard-coded in the section header like the name, instead these values are stored as relocation entries. We can once again do that using readelf:

$ readelf  amd-rng.ko -r 

Relocation section '.rela.text' at offset 0x16f0 contains 15 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend

....

Relocation section '.rela.gnu.linkonce.this_module' at offset 0x2068 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000138  002c00000001 R_X86_64_64       0000000000000010 init_module + 0
0000000003f8  002700000001 R_X86_64_64       0000000000000010 cleanup_module + 0

Notice that there are two offsets (0x138, 0x3f8) one for the init method (init_module) and one for the cleanup method (cleanup_module).

So far we’ve collected the following information:

  1. .gnu.linkonce.this_module ELF section size: 0x440
  2. The offset where the module name is stored: 0x18
  3. The offsets for the startup and deconstruction functions: 0x138, 0x3f8

Let’s update our ivision module to integrate this new information:

#define NAME "ivision"
#define VERMAGIC "6.6.32 SMP preempt mod_unload "
#define GNU_LINK_SIZE 0x440
#define GNU_LINK_NAME_OFFSET 0x18
#define INIT_LOCATION  0x138
#define CLEANUP_LOCATION 0x3f8

char modinfo[100]  __attribute__((section(".modinfo"))) = "name=" NAME "\x00vermagic=" VERMAGIC;

int init(void){
    return 0;
}

void cleanup(void){
}

int main(){
  return 0;
}


struct module { 
    char __padding[GNU_LINK_NAME_OFFSET];
    char name [sizeof(NAME)];
    char __padding1[INIT_LOCATION-GNU_LINK_NAME_OFFSET-sizeof(NAME)];
    void *init;
    char __padding2[CLEANUP_LOCATION-INIT_LOCATION-sizeof(void*)];
    void *cleanup;
    char __padding3[GNU_LINK_SIZE-CLEANUP_LOCATION-sizeof(void*)];
}__attribute__((packed));

struct module tmp __attribute__ ((section (".gnu.linkonce.this_module"))) =
{
  .name = NAME,
  .init = init,
  .cleanup = cleanup,
};

Note: The init function must return a 0 to indicate a successful init. The cleanup function should not have any return.

Let’s talk about the changes. We’ve declared a new struct module which contains all the .gnu.linkonce.this_module section information which we want to define. It also uses our offsets to make sure our data is located in the correct locations. We also made use of the __attribute__((packed)) feature to make sure that we have full control over the padding and size of the strict. This attribute disables the alignment that the compiler would otherwise perform.

Additionally, we’ve created two new functions: init and cleanup. These will ultimately be called by our module, but for now they are empty.

Finally, we define the tmp variable as a new struct module and set up all the necessary values:

  • Module Name
  • Init method
  • Clean up method

We also use the section attribute to store the resulting variable in .gnu.linkonce.this_module. With this we’ve almost completed creating our own Linux kernel module from scratch. It would load now on an x86_64 machine, but it would not do anything.

ELF Section — .plt and .init.plt

One of our goals was make this runnable on AArch64 devices. In order to achieve that, we need to include the .plt and .init.plt ELF section headers. Again, we’ll follow the process of extracting a Linux kernel module from the image, running readelf and copying the results. In this instance, we will use the /lib/modules/6.6.32/kernel/net/802/p8022.ko kernel module from the AArch64 QEMU image. First we will grab the size of the .plt and .init.plt sections:

$ readelf -S  p8022.ko -W 
There are 25 section headers, starting at offset 0x1078:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  ...
  [ 7] .plt              PROGBITS        0000000000000000 0001d4 000001 00  AX  0   0  1
  [ 8] .init.plt         PROGBITS        0000000000000000 0001d5 000001 00   A  0   0  1

Based on the results of the above, both section have a size of 0x1. Next, let’s check if that section contains any hard-coded values:

$ readelf -j .plt -j .init.plt  p8022.ko -W 

Hex dump of section '.plt':
  0x00000000 00                                  .


Hex dump of section '.init.plt':
  0x00000000 00

Based on the results, it appears that the sections are empty. With this information we can add the following lines to ivision.c to enable it to run on AArch64.

char plt[0x000001] __attribute__ ((section (".plt"))) __attribute__ ((packed)) = "\x00";
char init_plt[0x000001] __attribute__ ((section (".init.plt"))) __attribute__ ((packed)) = "\x00";

Now our module, once compiled, could run on both X86_64 and AArch64 architectures. There’s just one problem, it does not do anything.

Calling Kernel Functions

So far, we have a kernel module, but it does not do anything. Let’s modify init to print ivision can help with your pentesting needs and cleanup to print contact us at ivision.com. Typically, we would import printk.h and then call printk to achieve this goal. However, we do not have access to any headers, so we have to take a different approach. We will use function pointers to call functions in the kernel. In order to do that, we’ll need to:

  1. Identify the functions available to us.
  2. Identify their memory address.
  3. Create a function pointer to call the function.

We can solve problems 1 and 2 by taking a look at /proc/kallsyms. This file contains all kernel symbols, and it often contains a list of the exported functions and their memory addresses. We are primarily interested in printk, so let’s use look for that symbol:

# grep printk /proc/kallsyms
ffffffff810c1da0 T _printk

Note: The above only functions when CONFIG_KALLSYMS_ALL is set to y and when kptr_restrict is set to 0. On real-world devices, you will likely want to run the above as root and hope that CONFIG_KALLSYMS_ALL is enabled. If not, you’ll likely need to find a different way to leak kernel function addresses.

You will notice that there is not a printk function, but there is an _printk. This acts just like printk, so we will use it instead. With the symbol identified, we can use C’s function pointer syntax to call functions at a specific memory address. For _printk, we can create a function pointer using the following

int (*_printk)(const char *) = (void*) 0xffffffff810c1da0;

For reference, the basic format of a function pointer is:

return_type (*function_name)(param_type_1, param_type_2, ...) = (void *) address;

Let’s update our ivision.c’s init and cleanup to call this _printk function:

int init(void){
  (*_printk)("[INIT] ivision can help with your pentesting needs\n");
   return 0;
}

void cleanup(void){
  (*_printk)("[CLEANUP] contact us at ivision.com\n");
}

We have now written a Linux kernel module. Let’s talk briefly about compiling. We can compile our code into a kernel module by using gcc -c ivision.c -o ivision.ko. Once compiled, we can load and unload our module using insmod and rmmod. Finally, if we want to build our module for AArch64, we’ll need a bare-metal tool chain. For AArch64, we can get one from ARM here and we will need to adjust the .modinfo and .gnu.module.this_module fields accordingly, but once adjusted we can build it using the toolchain’s prebuilt aarch64-none-elf-gcc binary. This acts just like gcc, but builds for AArch64, thus we only need to run aarch64-none-elf-gcc -c ivision.c -o ivision.ko to get an AArch64 kernel module.

Now we’ve got a completed kernel module and can load and unload our module:

$ gcc -c ivision.c -o ivision.ko
$ scp -O -P 2222 ivision.ko root@127.0.0.1:/tmp
$ ssh root@127.0.0.1 -p 2222
# insmod /tmp/ivision.ko
# rmmod /tmp/ivision.ko
# dmesg | grep ivision
ivision: loading module not compiled with retpoline compiler.
ivision: module license 'unspecified' taints kernel.
ivision: module license taints kernel.
[INIT] ivision can help with your pentesting needs
[CLEANUP] contact us at ivision.com

Making the keydump module portable

We are almost done here. The final task is to convert our keydump.ko module from the previous blog into something that is usable without Linux kernel headers. That program can be found in this blog’s resources. We will not cover all the code, but let’s talk about some of the highlights:

  • We had to define struct key. We did not have existing Linux kernel headers to link against, so we had to define our own structure. We didn’t need access to all the members of the struct, so we took a shortcut and ignored any unused members. We took a similar approach for all memory structures, which would typically be included as a header. For the example, the following struct key was created by computing the offsets based on the memory structure documented in Linux Kernel’s source code:
struct key {
  unsigned long usage;
  unsigned long serial;
  char __padding[152];
  char* description;
  void* payload;
};
  • The kernel module no longer uses kernel parameters, it now has to be recompiled for every key dump attempt.

Once the #defines and function pointers are updated, we can compile it using gcc -c keydump.c -o keydump.ko or aarch64-none-elf-gcc -c ivision.c -o ivision.ko for AArch64, run it using insmod keydump.ko, and remove the module using rmmod keydump.ko. Like before, we will see the results in dmesg:

# dmesg
USER/LOGON - KEY:63 6f 6e 74 61 63 74 20 75 73 20 61 74 20 69 76 69 73 69 6f 6e 2e 63 6f 6d                       contact us at ivision.com

Running on real hardware

This Linux kernel module we wrote works well on QEMU with a mainline kernel. In practice, you might find that its difficult to just compile keydump.c without headers and have it run as intended. Why? Well in the real world, devices use custom kernels with custom configurations. These custom configurations probably will not stop the Linux kernel module from loading, but it will likely crash when attempting to interact with kernel memory. This is on account of the custom configurations potentially altering the data structure used by the Linux Kernel Key Retention Service. We have seen this in real world applications. Custom configurations result in changes to the underlying key structure resulting in kernel panics when invalid memory regions are read. It is a difficult problem to solve, especially when kernel headers are not present. The solution? Staring at the memory dump. The approach we have found to work best has been to:

  1. Determine the Linux kernel version being used and clone its source
  2. Inject your own known key into the kernel keyring (ideally it should match the type of the key you want to extract)
  3. Figure out how to get a reference to a key structure in kernel memory. We have been lucky and the key_lookup function was always present
  4. Dump the memory at the given reference
  5. Compare the Linux kernel source’s key struct to what is in the memory dump
  6. Reconstruct the memory structure enough to recover your known key
  7. Repeat but with your target key

Hopefully, this helps if you are looking to dump keys from the Linux Kernel Key Retention service or if you’re trying to figure out how to build stand-alone Linux kernel modules. If you have any questions, feel free to contact me directly at jsotoventura at company name dot com.