KingDezigns — NAS08 Kernel Recovery Summary

The problem

After a routine apt upgrade that included a kernel jump from 6.12 to 6.18 and a new raspi-firmware package, NAS08 lost all four SATA data drives connected via the Radxa Penta SATA HAT. ZFS reported no pools available and OMV could not see the drives.

zpool status: no pools available ahci: failed to start port 0 (errno=-12) lsblk: sdb/sdc/sdd/sde missing BAR 0 [io size 0x0080]: can't assign; no space DMA32: empty

The upgrade included linux-image-6.18.29+rpt-rpi-2712, raspi-firmware 1:1.20260513-1, and openmediavault 8.2.13-1 among 20+ packages. The Radxa Penta SATA HAT uses a JMicron JMB585 PCIe SATA controller. An upstream kernel commit (ee95f3c) affecting the JMB585 chipset means kernel 6.18 now requires the pcie-32bit-dma-pi5 device tree overlay for the controller to allocate DMA memory successfully. This overlay was present in config.txt but placed under the [cm5] conditional section, which is ignored on a Raspberry Pi 5 Model B. As a result, the overlay never loaded and the JMB585 had no DMA32 address space.

Diagnostic commands run & why

1 Confirm ZFS module is loaded

lsmod | grep zfs

Rule out ZFS module as the cause — if it's not loaded, drives won't be visible to any pool.

ZFS module loaded correctly — zfs and spl both present. Not the cause.

2 Check running kernel vs installed kernels

uname -r
dpkg -l | grep linux-image

Confirm which kernel booted and whether a version mismatch caused the issue.

Booted into 6.18.29 — confirmed the kernel jump from 6.12 happened. Both versions still installed.

3 Check if OS sees the drives at all

lsblk
ls /dev/disk/by-id/

Determine if the problem is at the ZFS layer or the OS/hardware layer.

Only boot SSD visible. Data drives completely absent — hardware/driver level problem confirmed.

4 Check dmesg for USB/SATA/PCIe errors

dmesg | grep -iE "usb|sd|storage|attach|disconnect|reset" | grep -v "loop\|zram\|firmware" | tail -60

Identify whether drives were seen at boot and then lost, or never detected at all.

Only one USB device detected at boot (boot SSD). No SATA drives enumerated. Pointed to PCIe HAT issue.

5 Check PCIe controller visibility

lspci

Confirm whether the JMicron JMB585 SATA controller on the Radxa HAT is being detected by the PCIe bus.

JMicron JMB585 visible at 0001:01:00.0 — controller detected but not functional.

6 Check PCIe BAR assignment errors

dmesg | grep -iE "pcie|pci|ahci|sata|asm|jmb|asmedia" | head -40

Identify why the detected controller isn't working — look for resource allocation failures.

BAR 0–4 all failed to assign I/O space. PCI bridge showed zero-length address range. AHCI errno=-12 (ENOMEM).

7 Check DMA memory layout

cat /proc/meminfo | grep -i cma
dmesg | grep -iE "cma|dma|coherent" | head -20

Determine why AHCI port start was failing with ENOMEM — check DMA zone availability.

DMA32 zone empty. CmaFree: 0 kB. JMB585 forced into 32-bit DMA mode with no available DMA32 address space.

8 Check hardware identity

cat /proc/device-tree/model

Confirm exact hardware model — critical for applying the correct config.txt overlay section.

Raspberry Pi 5 Model B Rev 1.1 — confirmed Pi 5, not CM5. This meant the [cm5] section in config.txt was being ignored.

9 Check config.txt overlay placement

cat /boot/firmware/config.txt

Identify whether the pcie-32bit-dma-pi5 overlay was present and in the correct conditional section.

Overlay was present but under [cm5] section — ignored on Pi 5. Moving it to [all] section was the fix.

10 Check EEPROM version and config

sudo rpi-eeprom-update
sudo rpi-eeprom-config

Determine if pci=pcie_bus_safe was coming from the EEPROM bootloader config.

EEPROM was 5 months out of date. Updated during recovery session.

Root cause

An upstream Linux kernel commit (ee95f3c) changed how the JMB585 PCIe SATA controller allocates DMA memory on kernel 6.18. The JMB585 now requires the pcie-32bit-dma-pi5 device tree overlay to function correctly. This overlay was present in /boot/firmware/config.txt but placed under the [cm5] conditional section, which is silently ignored on a Raspberry Pi 5 Model B. As a result the overlay never loaded, the JMB585 had no DMA32 address space, and AHCI failed with ENOMEM on every port start attempt.

⚠ Key clue

The config.txt overlay section mismatch was the root cause. The overlay fix was correct — it was simply applied to the wrong conditional block during the initial recovery attempt.

# Wrong — ignored on Pi 5 Model B:
[cm5]
dtoverlay=pcie-32bit-dma-pi5

# Correct — applies to all Pi hardware:
[all]
dtoverlay=pcie-32bit-dma-pi5

The kernel 6.18 requirement for pcie-32bit-dma-pi5 on JMB585 hardware is documented upstream. Any Pi 5 running kernel 6.18 with this HAT requires this overlay in the [all] or [pi5] section of config.txt.

Resolution: Moved dtoverlay=pcie-32bit-dma-pi5 from the [cm5] section to the [all] section in /boot/firmware/config.txt, then copied the 6.18.29 kernel binary back to /boot/firmware/kernel_2712.img and rebooted. All four SATA drives appeared immediately on kernel 6.18.

Recovery procedure applied

1 Roll back to kernel 6.12 (initial emergency recovery)

sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8
sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-2712 /boot/firmware/kernel_2712.img
sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-v8 /boot/firmware/kernel8.img
sudo reboot

Immediately restored drive visibility on 6.12 while the root cause was investigated. Holds prevented the firmware hook from re-upgrading kernel_2712.img on reboot.

All four drives (sdb/sdc/sdd/sde) returned on 6.12.

2 Fix the overlay placement in config.txt

sudo nano /boot/firmware/config.txt

# Remove dtoverlay=pcie-32bit-dma-pi5 from the [cm5] section
# Add it to the [all] section instead:

[cm5]
dtoverlay=dwc2,dr_mode=host

[all]
dtoverlay=pcie-32bit-dma-pi5

The overlay was present but in the wrong conditional section. Moving it to [all] ensures it applies to the Pi 5 Model B hardware.

3 Unhold packages and copy 6.18 kernel back

sudo apt-mark unhold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8
sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-2712 /boot/firmware/kernel_2712.img
sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-v8 /boot/firmware/kernel8.img
sudo reboot

6.18 was already installed — only the kernel_2712.img binary needed updating. The overlay fix was now in place before boot.

4 Verify and re-apply holds

uname -r        # should show 6.18.29+rpt-rpi-2712
lsblk           # should show sdb sdc sdd sde
sudo zpool status

sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8
sudo apt-mark showhold

All four drives present on 6.18.29. ZFS pool ONLINE. Holds re-applied to prevent uncontrolled kernel upgrades.

Emergency rollback — if a future upgrade breaks drives again

If a future upgrade causes drives to disappear again, this is the fast rollback procedure. Takes less than 5 minutes.

1 Hold packages immediately

sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8

The z50-raspi-firmware hook overwrites kernel_2712.img on every apt operation. Hold first to stop this.

2 Copy 6.12 kernel directly over the firmware kernel image

sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-2712 /boot/firmware/kernel_2712.img
sudo cp /boot/vmlinuz-6.12.75+rpt-rpi-v8 /boot/firmware/kernel8.img

# Verify sizes — should be 9698043 and 9695883
ls -la /boot/firmware/kernel_2712.img /boot/firmware/kernel8.img

raspi-firmware boots whichever kernel is in kernel_2712.img directly. Replacing it with 6.12 forces the rollback.

3 Reboot and confirm

sudo reboot

# After reboot:
uname -r        # should show 6.12.75+rpt-rpi-2712
lsblk           # should show sdb sdc sdd sde
sudo zpool status

Drives and ZFS pools return immediately on 6.12.

Scripts created this session

NAS08 ZFS Health Report

zfs_report_nas08.sh

NAS16 ZFS Health Report

zfs_report_nas16.sh

Commands executed per run

zpool status -v <pool>
zpool list -H -o name,size,alloc,free,frag,cap,health <pool>
zfs list -r -H -o name,used,avail,refer,mountpoint <pool>

Deploy & schedule (cron)

chmod +x zfs_report_nas08.sh
sudo mv zfs_report_nas08.sh /opt/scripts/
# Add to crontab — runs daily at 7:00 AM:
0 7 * * * /opt/scripts/zfs_report_nas08.sh >> /var/log/zfs_report.log 2>&1

Email recipient

rufus.king@kingdezigns.com

NAS16 — apply same fix

NAS16 runs the identical Radxa Penta SATA HAT on a Raspberry Pi 5. It will hit the exact same kernel 6.18 / JMB585 DMA issue if updated without the overlay fix in place. SSH into NAS16 and follow these steps.

1 Hold kernel and firmware packages on NAS16

sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8
sudo apt-mark showhold

Prevents kernel 6.18 and new raspi-firmware from installing automatically while the config.txt fix is applied.

Expected: all packages listed as held.

2 Confirm hardware identity

cat /proc/device-tree/model

Verify NAS16 is also a Pi 5 Model B so the [all] section placement is correct.

3 Fix the overlay placement in config.txt

sudo nano /boot/firmware/config.txt

# Ensure pcie-32bit-dma-pi5 is in the [all] section:
[all]
dtoverlay=pcie-32bit-dma-pi5

Same fix as NAS08 — overlay must be in [all], not [cm5], to apply on Pi 5 hardware.

4 Unhold, upgrade, and reboot

sudo apt-mark unhold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8
sudo apt upgrade
sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-2712 /boot/firmware/kernel_2712.img
sudo cp /boot/vmlinuz-6.18.29+rpt-rpi-v8 /boot/firmware/kernel8.img
sudo reboot

5 Verify and re-hold

uname -r
lsblk
sudo zpool status
sudo apt-mark hold raspi-firmware linux-image-rpi-2712 linux-image-rpi-v8 linux-image-6.18.29+rpt-rpi-2712 linux-image-6.18.29+rpt-rpi-v8

Expected: sdb sdc sdd sde all present, uname shows 6.18.x, ZFS ONLINE.

Lessons learned

Lesson	Detail
Always identify exact hardware before troubleshooting	Run `cat /proc/device-tree/model` at the start of every session. The initial recovery misidentified NAS08 as a CM5, which placed the overlay fix under [cm5] where it was silently ignored on the Pi 5 Model B.
Never update both NAS systems at the same time	Update NAS08 first, confirm stable for 48hrs, then update NAS16. This approach would have prevented both systems from failing simultaneously.
config.txt conditional sections are silently ignored	Overlays placed under [cm4] or [cm5] are completely ignored on Pi 5 Model B hardware. There is no warning or error. Always verify placement with `cat /proc/device-tree/model` first.
Kernel 6.18 requires pcie-32bit-dma-pi5 for JMB585	Upstream commit ee95f3c changed DMA behaviour for the JMB585 chipset. Any Pi 5 running kernel 6.18 with the Radxa Penta SATA HAT must have `dtoverlay=pcie-32bit-dma-pi5` in the `[all]` section of config.txt or drives will not appear.
Hold kernel packages on all Pi-based NAS systems	Kernel updates can silently break HAT compatibility. Always hold and update deliberately, never via unattended-upgrades.
raspi-firmware manages kernel_2712.img directly	The post-install hook z50-raspi-firmware copies the kernel directly to /boot/firmware/kernel_2712.img. Copying an older kernel binary over this file is the correct rollback mechanism. Hold raspi-firmware before doing so or it will be overwritten on next boot.
Always simulate before upgrading	Run `sudo apt upgrade --simulate 2>&1 \| grep -E "raspi-firmware\|linux-image\|linux-headers\|Inst\|Remv"` before every upgrade. If any kernel or firmware packages appear as Inst lines, investigate before proceeding.

Current working config — NAS08

This is the known-good configuration for NAS08 on kernel 6.18.29 with all four SATA drives visible and ZFS healthy.

/boot/firmware/config.txt — bottom section

# Enable PCIe
dtparam=pciex1
dtparam=pciex1_gen=3

[cm4]
otg_mode=1

[cm5]
dtoverlay=dwc2,dr_mode=host

[all]
dtoverlay=pcie-32bit-dma-pi5

/boot/firmware/cmdline.txt

console=serial0,115200 console=tty1 root=PARTUUID=57cb028e-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US

Active kernel

6.18.29+rpt-rpi-2712

Held packages

raspi-firmware
linux-image-rpi-2712
linux-image-rpi-v8
linux-image-6.18.29+rpt-rpi-2712
linux-image-6.18.29+rpt-rpi-v8

Network map — items to update

Item	Old value	New value
NAS08 kernel version	`6.12.75+rpt-rpi-2712`	`6.18.29+rpt-rpi-2712 (held)`
NAS08 config.txt overlay	`pcie-32bit-dma-pi5 under [cm5] — ignored`	`pcie-32bit-dma-pi5 under [all] — active`
NAS08 raspi-firmware	`auto-update enabled`	`held at 1:1.20260513-1`
NAS16 kernel + overlay fix	`6.12.75 — pending upgrade`	`apply config.txt fix first, then upgrade to 6.18.29`
ZFS report scripts	`not deployed`	`zfs_report_nas08.sh + zfs_report_nas16.sh — deploy to /opt/scripts/`