A Phantom Zone for AI

The problem

A customer with serious compliance requirements wants a model running on their data, on a brand new bare-metal GPU server. The server is air-gapped. You provision it out of band over Redfish, and it boots over PXE. Before any model runtime or customer data touches memory, three things have to be verified and active: the TPM 2.0 measurements, your custom eBPF network hooks and the GPUs' VFIO bindings.

That sounds narrow. It isn't. Swap the GPU server for an edge box on a factory floor, a rack in a sovereign cloud or an inference appliance in a hospital, and it's the same problem. A machine has to prove what it is before it gets anything worth stealing.

Each piece is well documented on its own. The hard part is the order, and proving the order happened. What follows is a design that holds up, piece by piece, and the places it breaks.

Phantoms can watch. They can't touch.

In the Superman comics, the Phantom Zone is where Krypton sent its worst criminals. Prisoners like General Zod could see everything going on outside. They just couldn't touch any of it.

That's the state I want a fresh node in. It can power on, get an address, pull its boot image and talk to the one server that checks it. It can't touch anything that matters, because everything that matters is encrypted and it doesn't have the key. It stays a phantom until it proves it shouldn't be one.

Once it does, the zone flips around. The model runs inside a zone of its own: GPUs fenced off, network shut except for what you allowed. It can see its inputs. It can't reach out.

The obvious approach is all about ordering. Load this, then that, then start the runtime. Order matters, but it isn't security. Anything running as root can start whatever it wants, in whatever order it likes. What it can't do is decrypt data it doesn't have a key for.

So the whole design hangs on one rule. The key to the model and the customer's data only comes out after the node proves it booted exactly what you expected, with the network hooks and the GPU bindings live.

Boot order keeps honest software in line. Key release keeps everything else out.

What the TPM actually does

The TPM gets oversold, so it's worth a minute.

It doesn't stop anything from running. Secure Boot is the bouncer: firmware refuses to run code that isn't signed by a key you trust. The TPM is the camera over the door. Each stage of the boot hashes the next stage before handing it control, and adds that hash to a register called a PCR. You can't write a PCR. You can only extend it: the new value is a hash of the old value plus the new measurement. Change anything in the chain and that register comes out different, and nothing later in the boot can put it back. About the only thing a TPM will flat out refuse is handing back a secret sealed to PCR values that no longer match.

The firmware and the OS also keep an event log of everything they measured. When asked, the TPM signs the current PCR values, plus a fresh nonce so the answer can't be replayed, with a key that never leaves the chip. That signed bundle is a quote. A verifier replays the log and checks that it adds up to the quoted values. If it does, nobody edited the log. Your policy decides whether it's good news.

You want both. Secure Boot stops unsigned code. It does nothing about an old kernel you signed last year with a known hole in it, unless you remembered to revoke it. The measurements catch it either way.

The boot, step by step

The whole chain is below. Pick something to tamper with and watch where it gets caught.

1. Prep it before it powers on. The BMC lives on its own management network, never the data network. Over Redfish you turn on the TPM, Secure Boot, the IOMMU and pre-boot DMA protection, enroll your own Secure Boot keys where the BMC supports it, and set a one-time network boot. When the box arrives, record its TPM's endorsement key certificate and enroll it with your attestation server. That certificate is how you'll know later that you're talking to this chip, and not something pretending to be it.
2. Let the firmware measure itself. Firmware measures its own code into PCR 0, its configuration into PCR 1, and any option ROMs it runs, like the network card's PXE ROM, into PCR 2. Secure Boot's state and keys go into PCR 7. Something measuring itself could just lie, which is why you want Intel Boot Guard or AMD Platform Secure Boot. With either, the processor checks the first block of firmware before running it, so a reflashed image doesn't get to vouch for itself.
3. Boot one signed file. PXE and TFTP don't authenticate anything, so don't trust the wire. Trust the signature. Boot a unified kernel image: kernel, initramfs and command line in one file, signed with your key. The firmware measures the whole file into PCR 4, and the stub inside it measures each piece into PCR 11. With Secure Boot on, the stub ignores any command line handed to it by whatever loaded it, so nobody slips iommu=off in over the network. Anything it does add, like a signed add-on, lands in PCR 12. The baked-in command line turns the IOMMU on with passthrough off, keeps the host's GPU drivers from loading, and turns on kernel lockdown, IMA and the BPF LSM.
4. Make the initramfs the gate. It's small and it has one job: put the box in a safe shape and prove it. Load nftables with a default drop, open only to the attestation server. Bind every GPU to vfio-pci before any host driver can grab it. Load the eBPF programs, attach them and pin them, so they stay attached after the loader exits. Then read back what the kernel says is live, hash it, and extend that into a PCR nothing else on this boot path uses and nothing can reset, so not 16 or 23. The unit that asks for the key Requires= the gate and runs After= it. If a check fails, the gate fails. It has to fail, not skip: a unit skipped on a condition doesn't stop the units that require it.
5. Prove it. The attestation server sends a fresh nonce, and the node answers with a quote over PCRs 0 through 7, 11 and 12, plus the gate's register. It's signed by an attestation key the server has already tied to that endorsement certificate. The server replays the event logs against known-good values. Keylime does all of this, it's open source, and it runs inside your own walls.
6. Release the key, and nothing else. Pass, and the node gets the key to the encrypted volumes that hold the model weights and the customer's data. Fail, and it gets nothing. It's up, it can reach one server, and it's holding a disk full of noise. A phantom.
7. Run the model in its own zone. The runtime starts last, off the volume the key unlocked, so without the key there's nothing to start. It runs in a VM with the GPUs passed through over VFIO, so tenant code never touches a host GPU driver. Egress is denied by default in eBPF and nftables, so the model, or an agent built on it, has nowhere to send anything you didn't allow. If a NIC or InfiniBand card goes through to the VM for GPU traffic, the host never sees that traffic, so fence it at the switch.
8. Keep proving it. IMA measures every binary and kernel module into PCR 10 before it runs, and with appraisal on, a file without a valid signature doesn't run at all. The verifier keeps asking for fresh quotes. When something shows up that isn't on the list, it revokes: the key server stops answering and the rest of the fleet stops talking to that node. Give certificates and tokens short lives, so a node that can't keep proving itself loses access on its own. A disk that's already unlocked stays unlocked, though, so when you stop trusting a node, cut its power through the BMC.

The gate, up close

Verified and active are different things, and plenty of designs only check the first one.

An eBPF object file on disk is present. A program attached to an interface is active. So the gate doesn't hash files and call it a day. It asks the kernel what's attached right now (bpftool lists the programs on every interface and cgroup), hashes the instructions the kernel hands back, and measures that. Zero out the map IDs first. They change every boot, and the hash shouldn't.

Same for the GPUs. The gate reads sysfs to see which driver owns each device and what else sits in its IOMMU group. The group is the real unit of isolation, and VFIO won't hand one over until every device in it is bound to a VFIO driver or has no driver at all. So check the whole group, not just the GPU.

Then it locks the door behind itself, as far as it can. A small BPF LSM program can refuse new bpf() calls once the gate is done. Give it an allow-list, because systemd and the VM manager load small BPF programs of their own every time they start a unit or a VM. It won't stop root from rewriting nftables, though. Root on the host is its own problem, and I'll get to it.

Two things bite people here. iommu=pt shows up in half the GPU passthrough guides online. It lets the devices the host keeps for itself skip the IOMMU. Fine for a gaming rig, wrong for this one. Use iommu.passthrough=0 and iommu.strict=1, so a device loses access to a buffer the moment the driver lets go of it, not a little later. The other is the gap between firmware handing off and Linux setting the IOMMU back up, where a hostile device can still DMA into memory. Pre-boot DMA protection in firmware, plus efi=disable_early_pci_dma, closes it. Test that last one first. Some hardware doesn't like it.

Where it breaks

The crypto is the easy part. The trouble is everything around it.

• Updates. PCRs change every time the firmware or the kernel does. Bind keys to literal PCR values and a routine update bricks the fleet. On the verifier, judge firmware with a policy over the event log instead of one magic number. For anything sealed on the node itself, bind to PCR 7 plus a signed policy for PCR 11, so any release you signed keeps unlocking. Watch the flip side: an old release still carries a valid old signature. Keep a version check on the verifier, or look at systemd-pcrlock, which tells the TPM exactly which versions are allowed.
• Your own keys. Enroll only your keys and PXE stops working, because the network card's boot ROM is usually signed by Microsoft's third-party UEFI CA. Keep that CA and you trust everything else it ever signed. Or enroll the hash of each option ROM you actually run. Either way, keep the revocation list current.
• The air gap itself. Everything that normally phones home can't. There's no public NTP, so bring a time source you trust, like a GPS clock, or certificate checks drift into nonsense. The TPM makers' CA certificates have to be inside before the first box shows up. Some firmware TPMs keep their endorsement certificate on the maker's server instead of the chip, so fetch those before the box goes in. NVIDIA's GPU verifier and the Intel and AMD services behind confidential VMs expect the internet for reference values and revocation. Mirror what they need inside.
• The BMC. It can reflash firmware, mount virtual media and power cycle the box, and nothing in this chain measures the BMC itself. It's the softest thing in the rack. The measurements will catch a reflash or a changed setting on the host. They won't catch the BMC being owned, so keep it on its own network and keep its firmware current.
• Process. An attested box run by a sloppy process is still a sloppy box. The golden values, the signing keys and the update media need custody rules as strict as the box itself.

In Superman II, the Phantom Zone doesn't fail because Zod outsmarts it. It fails because Superman hurls a hydrogen bomb into space and the blast happens to crack it open. Nobody modeled the bomb. Model the bomb.

If you don't trust the host either

Everything so far keeps the node honest. It doesn't protect the customer from whoever runs the host, and revocation is damage control, not time travel. If someone had root before you noticed, anything in memory was theirs to read.

If the operator is in your threat model, and for a regulated customer it often is, go one step further. Run the model in a confidential VM, on Intel TDX or AMD SEV-SNP, with the GPU in confidential computing mode, which NVIDIA's Hopper and Blackwell parts support. The VM's memory is encrypted with keys the host never sees, and the GPU produces its own attestation evidence. Now the key release checks two pieces of evidence, the CPU's and the GPU's, and the key goes into the confidential VM, never to the host. Done right, not even root on the host can read the weights.

The short version

All of it in one breath: Secure Boot with your own keys, a signed unified kernel image, and every stage measured into the TPM. A tiny initramfs that drops every packet except attestation, binds the GPUs to vfio-pci, attaches the eBPF programs, then measures what's actually live. A quote to an attestation server inside the air gap. And a key that only comes out if everything matches.

No match, no key. The node stays a phantom.

Every system I've built has had to live somewhere I didn't control: lockers in other people's buildings, cameras on warehouse floors, someone else's power and someone else's network. This is the cleanest version of that problem I've seen. A box that has to earn trust before it touches anything that matters.

Sources

1. UAPI Group, Linux TPM PCR Registry
2. UAPI Group, Unified Kernel Image specification
3. systemd-stub(7): the command line under Secure Boot, PCR 11 and PCR 12
4. systemd-cryptenroll(1): signed PCR policies and which PCRs to bind
5. systemd-pcrlock(8): a TPM policy that names the versions allowed to unlock
6. systemd.unit(5): a failed condition doesn't fail the units that require it
7. Keylime, design overview
8. Linux kernel documentation, VFIO
9. Linux kernel documentation, LSM BPF programs
10. Linux kernel parameters, including iommu.passthrough and iommu.strict
11. The patch that added efi=disable_early_pci_dma
12. DMTF, Redfish release 2020.1 overview (Secure Boot databases)
13. NVIDIA, Hopper single GPU attestation example
14. The Phantom Zone
15. Superman II (1980)