The nastiest bugs almost never come from the shiny new fleet. They come from the one aging system a customer has leaned on for years and cannot simply retire. This one was a customer's legacy box, a Supermicro H11DSi with dual EPYC and a MegaRAID controller, and after a routine rebuild it started eating its own filesystem on the first boot.
ext4 checksum failures everywhere, bad extra_isize, Directory block failed checksum, random segfaults on both sockets, services refusing to start. fsck would repair a pile of errors, then they were back after a little I/O. Every instinct said dying hardware, which is exactly the trap with old kit: you assume the tired old box is finally giving up.
We swapped the controller, reseated cards, ran memtester, checked SMART and EDAC. All clean. The hardware was fine. The culprit was AMD's IOMMU, an old kernel, and the MegaRAID driver quietly writing DMA into the wrong physical pages.
If you run AMD EPYC on Supermicro H11DSi boards with a Broadcom LSI SAS3408 tri-mode controller (the 9440-8i in RAID mode, or the 9400-8i as an HBA) on an enterprise Linux 9 kernel, this can bite you. Here is how to spot it, prove it, and fix it.
The Symptom That Screams Hardware
A brand-new install, corrupt on the first boot. ext4 directory-block and inode checksum failures across the disk. ld-linux segfaulting at random on both sockets. Services that will not start. You run fsck, it repairs everything, and the damage is back after a bit of I/O. It looks exactly like a failing controller or bad DIMMs. It usually is not.
The Fastest Way to Prove It Is Not the Hardware
Boot a live ISO with a modern 6.x kernel. We reach for grml. Then hammer the disk: raw dd, O_DIRECT churn, a full mkfs.ext4 with thousands of files and an e2fsck, plus memtester, SMART and EDAC. If the box is rock solid under the 6.x kernel but the enterprise Linux 9 install corrupts on the exact same hardware, you are not looking at a hardware fault. You are looking at a kernel bug. That contrast is the whole diagnosis, and it takes twenty minutes.
Root Cause: IOMMU, an Old Kernel, and MegaRAID DMA
The boards had AMD IOMMU enabled in BIOS. On the older enterprise Linux 9 kernel (5.14), the megaraid_sas driver (9440) and the mpt3sas driver (9400) mishandle DMA under IOMMU translated and lazy-TLB mode. DMA lands in the wrong physical pages, and the filesystem is silently corrupted. This is a known upstream kernel bug class, fixed in newer kernels, which is precisely why a 6.x live kernel cannot reproduce it. The tell is baked into the mechanism.
The Fix
If the machine never virtualizes, it never needs IOMMU. Turn it off at the OS level with a single kernel cmdline argument:
Do not reach for iommu=pt here. It is unreliable while a MegaRAID virtual disk is doing background init or a consistency check. Use amd_iommu=off. Bake it into your PXE or kickstart bootloader line so every fresh install carries it automatically.
For a box that is already running and about to receive a BIOS update that will switch IOMMU on, set it on the live system first, before the update, so it does not start corrupting the moment IOMMU appears:
The Virtualization Exception
Hosts that genuinely need IOMMU for PCI passthrough are different. Do not disable it there. Keep IOMMU on, run a 6.x kernel that carries the fixes, and use iommu=pt. The amd_iommu=off rule is for bare-metal nodes that never pass a device through.
After enabling, audit for the fingerprint:
One consolation for ZFS users: end-to-end checksums detect this corruption, showing up as CKSUM errors and self-healing on redundant pools, instead of letting it silently destroy the filesystem. An ext4 boot partition behind the same controller has no such safety net, which is why the boot disk is usually where you see it first.
We Run the Metal so You Do Not Chase Ghosts
Two afternoons went into that corruption before a 6.x kernel gave it away, and that is the tax legacy hardware quietly charges. The one tired box a customer cannot retire is always where the pain hides. We run bare metal and storage in our own European datacenters (AS215197), including the awkward legacy others would rather not touch. The gremlins are our problem, not yours.