After a routine apt upgrade that included a kernel jump from 6.12 to 6.18 and a new raspi-firmware package, NAS08 lost all four SATA data drives connected via the Radxa Penta SATA HAT. ZFS reported no pools available and OMV could not see the drives.
The upgrade included linux-image-6.18.29+rpt-rpi-2712, raspi-firmware 1:1.20260513-1, and openmediavault 8.2.13-1 among 20+ packages. The Radxa Penta SATA HAT uses a JMicron JMB585 PCIe SATA controller. An upstream kernel commit (ee95f3c) affecting the JMB585 chipset means kernel 6.18 now requires the pcie-32bit-dma-pi5 device tree overlay for the controller to allocate DMA memory successfully. This overlay was present in config.txt but placed under the [cm5] conditional section, which is ignored on a Raspberry Pi 5 Model B. As a result, the overlay never loaded and the JMB585 had no DMA32 address space.
lsmod | grep zfs
Rule out ZFS module as the cause — if it's not loaded, drives won't be visible to any pool.
ZFS module loaded correctly — zfs and spl both present. Not the cause.
uname -r dpkg -l | grep linux-image
Confirm which kernel booted and whether a version mismatch caused the issue.
Booted into 6.18.29 — confirmed the kernel jump from 6.12 happened. Both versions still installed.
lsblk ls /dev/disk/by-id/
Determine if the problem is at the ZFS layer or the OS/hardware layer.
Only boot SSD visible. Data drives completely absent — hardware/driver level problem confirmed.
dmesg | grep -iE "usb|sd|storage|attach|disconnect|reset" | grep -v "loop\|zram\|firmware" | tail -60
Identify whether drives were seen at boot and then lost, or never detected at all.
Only one USB device detected at boot (boot SSD). No SATA drives enumerated. Pointed to PCIe HAT issue.
lspci
Confirm whether the JMicron JMB585 SATA controller on the Radxa HAT is being detected by the PCIe bus.
JMicron JMB585 visible at 0001:01:00.0 — controller detected but not functional.
dmesg | grep -iE "pcie|pci|ahci|sata|asm|jmb|asmedia" | head -40
Identify why the detected controller isn't working — look for resource allocation failures.
BAR 0–4 all failed to assign I/O space. PCI bridge showed zero-length address range. AHCI errno=-12 (ENOMEM).
cat /proc/meminfo | grep -i cma dmesg | grep -iE "cma|dma|coherent" | head -20
Determine why AHCI port start was failing with ENOMEM — check DMA zone availability.
DMA32 zone empty. CmaFree: 0 kB. JMB585 forced into 32-bit DMA mode with no available DMA32 address space.
cat /proc/device-tree/model
Confirm exact hardware model — critical for applying the correct config.txt overlay section.
Raspberry Pi 5 Model B Rev 1.1 — confirmed Pi 5, not CM5. This meant the [cm5] section in config.txt was being ignored.
cat /boot/firmware/config.txt
Identify whether the pcie-32bit-dma-pi5 overlay was present and in the correct conditional section.
Overlay was present but under [cm5] section — ignored on Pi 5. Moving it to [all] section was the fix.
sudo rpi-eeprom-update sudo rpi-eeprom-config
Determine if pci=pcie_bus_safe was coming from the EEPROM bootloader config.
EEPROM was 5 months out of date. Updated during recovery session.
An upstream Linux kernel commit (ee95f3c) changed how the JMB585 PCIe SATA controller allocates DMA memory on kernel 6.18. The JMB585 now requires the pcie-32bit-dma-pi5 device tree overlay to function correctly. This overlay was present in /boot/firmware/config.txt but placed under the [cm5] conditional section, which is silently ignored on a Raspberry Pi 5 Model B. As a result the overlay never loaded, the JMB585 had no DMA32 address space, and AHCI failed with ENOMEM on every port start attempt.
The config.txt overlay section mismatch was the root cause. The overlay fix was correct — it was simply applied to the wrong conditional block during the initial recovery attempt.
# Wrong — ignored on Pi 5 Model B: [cm5] dtoverlay=pcie-32bit-dma-pi5 # Correct — applies to all Pi hardware: [all] dtoverlay=pcie-32bit-dma-pi5
The kernel 6.18 requirement for pcie-32bit-dma-pi5 on JMB585 hardware is documented upstream. Any Pi 5 running kernel 6.18 with this HAT requires this overlay in the [all] or [pi5] section of config.txt.
Resolution: Moved dtoverlay=pcie-32bit-dma-pi5 from the [cm5] section to the [all] section in /boot/firmware/config.txt, then copied the 6.18.29 kernel binary back to /boot/firmware/kernel_2712.img and rebooted. All four SATA drives appeared immediately on kernel 6.18.
sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8 sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-2712 /boot/firmware/kernel_2712.img sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-v8 /boot/firmware/kernel8.img sudo reboot
Immediately restored drive visibility on 6.12 while the root cause was investigated. Holds prevented the firmware hook from re-upgrading kernel_2712.img on reboot.
All four drives (sdb/sdc/sdd/sde) returned on 6.12.
sudo nano /boot/firmware/config.txt # Remove dtoverlay=pcie-32bit-dma-pi5 from the [cm5] section # Add it to the [all] section instead: [cm5] dtoverlay=dwc2,dr_mode=host [all] dtoverlay=pcie-32bit-dma-pi5
The overlay was present but in the wrong conditional section. Moving it to [all] ensures it applies to the Pi 5 Model B hardware.
sudo apt-mark unhold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8 sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-2712 /boot/firmware/kernel_2712.img sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-v8 /boot/firmware/kernel8.img sudo reboot
6.18 was already installed — only the kernel_2712.img binary needed updating. The overlay fix was now in place before boot.
uname -r # should show 6.18.29+rpt-rpi-2712 lsblk # should show sdb sdc sdd sde sudo zpool status sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8 sudo apt-mark showhold
All four drives present on 6.18.29. ZFS pool ONLINE. Holds re-applied to prevent uncontrolled kernel upgrades.
If a future upgrade causes drives to disappear again, this is the fast rollback procedure. Takes less than 5 minutes.
sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8
The z50-raspi-firmware hook overwrites kernel_2712.img on every apt operation. Hold first to stop this.
sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-2712 /boot/firmware/kernel_2712.img sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-v8 /boot/firmware/kernel8.img # Verify sizes — should be 9698043 and 9695883 ls -la /boot/firmware/kernel_2712.img /boot/firmware/kernel8.img
raspi-firmware boots whichever kernel is in kernel_2712.img directly. Replacing it with 6.12 forces the rollback.
sudo reboot # After reboot: uname -r # should show 6.12.75+rpt-rpi-2712 lsblk # should show sdb sdc sdd sde sudo zpool status
Drives and ZFS pools return immediately on 6.12.
zfs_report_nas08.sh
zfs_report_nas16.sh
zpool status -v <pool> zpool list -H -o name,size,alloc,free,frag,cap,health <pool> zfs list -r -H -o name,used,avail,refer,mountpoint <pool>
chmod +x zfs_report_nas08.sh sudo mv zfs_report_nas08.sh /opt/scripts/ # Add to crontab — runs daily at 7:00 AM: 0 7 * * * /opt/scripts/zfs_report_nas08.sh >> /var/log/zfs_report.log 2>&1
rufus.king@kingdezigns.com
NAS16 runs the identical Radxa Penta SATA HAT on a Raspberry Pi 5. It will hit the exact same kernel 6.18 / JMB585 DMA issue if updated without the overlay fix in place. SSH into NAS16 and follow these steps.
sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 sudo apt-mark showhold
Prevents kernel 6.18 and new raspi-firmware from installing automatically while the config.txt fix is applied.
Expected: all packages listed as held.
cat /proc/device-tree/model
Verify NAS16 is also a Pi 5 Model B so the [all] section placement is correct.
sudo nano /boot/firmware/config.txt # Ensure pcie-32bit-dma-pi5 is in the [all] section: [all] dtoverlay=pcie-32bit-dma-pi5
Same fix as NAS08 — overlay must be in [all], not [cm5], to apply on Pi 5 hardware.
sudo apt-mark unhold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 sudo apt upgrade sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-2712 /boot/firmware/kernel_2712.img sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-v8 /boot/firmware/kernel8.img sudo reboot
uname -r lsblk sudo zpool status sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8
Expected: sdb sdc sdd sde all present, uname shows 6.18.x, ZFS ONLINE.
| Lesson | Detail |
|---|---|
| Always identify exact hardware before troubleshooting | Run cat /proc/device-tree/model at the start of every session. The initial recovery misidentified NAS08 as a CM5, which placed the overlay fix under [cm5] where it was silently ignored on the Pi 5 Model B. |
| Never update both NAS systems at the same time | Update NAS08 first, confirm stable for 48hrs, then update NAS16. This approach would have prevented both systems from failing simultaneously. |
| config.txt conditional sections are silently ignored | Overlays placed under [cm4] or [cm5] are completely ignored on Pi 5 Model B hardware. There is no warning or error. Always verify placement with cat /proc/device-tree/model first. |
| Kernel 6.18 requires pcie-32bit-dma-pi5 for JMB585 | Upstream commit ee95f3c changed DMA behaviour for the JMB585 chipset. Any Pi 5 running kernel 6.18 with the Radxa Penta SATA HAT must have dtoverlay=pcie-32bit-dma-pi5 in the [all] section of config.txt or drives will not appear. |
| Hold kernel packages on all Pi-based NAS systems | Kernel updates can silently break HAT compatibility. Always hold and update deliberately, never via unattended-upgrades. |
| raspi-firmware manages kernel_2712.img directly | The post-install hook z50-raspi-firmware copies the kernel directly to /boot/firmware/kernel_2712.img. Copying an older kernel binary over this file is the correct rollback mechanism. Hold raspi-firmware before doing so or it will be overwritten on next boot. |
| Always simulate before upgrading | Run sudo apt upgrade --simulate 2>&1 | grep -E "raspi-firmware|linux-image|linux-headers|Inst|Remv" before every upgrade. If any kernel or firmware packages appear as Inst lines, investigate before proceeding. |
This is the known-good configuration for NAS08 on kernel 6.18.29 with all four SATA drives visible and ZFS healthy.
# Enable PCIe dtparam=pciex1 dtparam=pciex1_gen=3 [cm4] otg_mode=1 [cm5] dtoverlay=dwc2,dr_mode=host [all] dtoverlay=pcie-32bit-dma-pi5
console=serial0,115200 console=tty1 root=PARTUUID=57cb028e-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US
6.18.29+rpt-rpi-2712
raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8
| Item | Old value | New value |
|---|---|---|
| NAS08 kernel version | 6.12.75+rpt-rpi-2712 |
6.18.29+rpt-rpi-2712 (held) |
| NAS08 config.txt overlay | pcie-32bit-dma-pi5 under [cm5] — ignored |
pcie-32bit-dma-pi5 under [all] — active |
| NAS08 raspi-firmware | auto-update enabled |
held at 1:1.20260513-1 |
| NAS16 kernel + overlay fix | 6.12.75 — pending upgrade |
apply config.txt fix first, then upgrade to 6.18.29 |
| ZFS report scripts | not deployed |
zfs_report_nas08.sh + zfs_report_nas16.sh — deploy to /opt/scripts/ |