ZFS RAIDZ2 with ZFSBootMenu: A Fault-Tolerant 4-Disk NAS

2026-09-15 5416 words 26 minutes

Contents

In my previous article, I described how to install Ubuntu on a ZFS mirror and boot it using ZFSBootMenu.

This time, the goal is more ambitious: build a proper four-disk NAS where both the operating system and user data live on a single ZFS RAIDZ2 pool, while still being able to boot the machine after losing one or even two physical disks.

I also wanted to avoid a dedicated boot SSD.

The final design looks like this:

        
        
        
    
HDD1
├── 1 GiB ESP
└── ZFS ─┐

HDD2     │
├── ESP  │
└── ZFS ─┤
         │
HDD3     ├── rpool / RAIDZ2
├── ESP  │
└── ZFS ─┤
         │
HDD4     │
├── ESP  │
└── ZFS ─┘

Each disk has its own EFI System Partition, containing a copy of ZFSBootMenu.

The ZFS pool itself contains Ubuntu and NAS data.

The system was also tested by physically removing disks, including a two-disk failure scenario.

Hardware and software

My setup is:

Component	Configuration
NAS	TerraMaster F4-425
HDDs	4 × WD Red WD60EFRX, 6 TB
RAM	16 GB
OS	Ubuntu Server 26.04 LTS
Kernel	Linux 7.0.x
OpenZFS	2.4.x
Pool layout	RAIDZ2
Bootloader	ZFSBootMenu
ESPs	One 1 GiB ESP per HDD

The internal TerraMaster USB DOM is left untouched.

The important design decision is that every HDD is independently bootable.

A disk therefore looks like this:

        
Disk
├── GPT partition 1: EFI System Partition
└── GPT partition 2: ZFS

And each ESP contains:

EFI/ZBM/VMLINUZ.EFI
EFI/BOOT/BOOTX64.EFI

The second file is the standard UEFI fallback path.

Installing Ubuntu on RAIDZ2

Preparing the live environment

Boot Ubuntu 26.04 installation media in x86_64 UEFI mode. The commands through the first reboot run in a root Bash shell. Secure Boot must be disabled for the unsigned upstream EFI image used below; signing and enrolling your own image is a separate setup.

        
sudo -i
test -d /sys/firmware/efi/efivars
apt update
apt install -y debootstrap gdisk dosfstools parted zfsutils-linux curl ca-certificates
test -e /usr/share/debootstrap/scripts/resolute

Create the host ID before creating the pool, then copy it into the new system later. Keep this value stable across boots:

zgenhostid
hostid

Preparing the disks

First, identify the drives.

Do not rely on /dev/sda, /dev/sdb, and similar names when creating the ZFS pool.

Use stable /dev/disk/by-id paths instead.

For example:

ls -l /dev/disk/by-id/
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE

Define the disks:

        
        
        
    
D1=/dev/disk/by-id/ata-DISK1
D2=/dev/disk/by-id/ata-DISK2
D3=/dev/disk/by-id/ata-DISK3
D4=/dev/disk/by-id/ata-DISK4

DISKS=(
  "$D1"
  "$D2"
  "$D3"
  "$D4"
)
export D1 D2 D3 D4

The following commands destroy the existing partition tables, so double-check the selected disks before continuing.

Create GPT tables and two partitions per disk:

        
        
        
    
for d in "${DISKS[@]}"; do
    wipefs -a "$d"
    sgdisk --zap-all "$d"

    sgdisk \
      -n 1:1M:+1G \
      -t 1:EF00 \
      -c 1:EFI \
      "$d"

    sgdisk \
      -n 2:0:0 \
      -t 2:6A85CF4D-1DD2-11B2-99A6-080020736631 \
      -c 2:ZFS \
      "$d"
done

partprobe
udevadm settle

Verify:

lsblk -o NAME,SIZE,PARTTYPE,FSTYPE

Creating the four EFI System Partitions

Format partition 1 on every disk:

        
mkfs.vfat -F 32 -n ZBM1 "${D1}-part1"
mkfs.vfat -F 32 -n ZBM2 "${D2}-part1"
mkfs.vfat -F 32 -n ZBM3 "${D3}-part1"
mkfs.vfat -F 32 -n ZBM4 "${D4}-part1"

Then get their UUIDs:

lsblk -o PATH,FSTYPE,LABEL,UUID,PARTTYPE

These UUIDs will later be used by /etc/fstab and zbm-esp-sync.

Creating the RAIDZ2 pool

The compatibility profile limits enabled pool features to ones supported by the selected ZFSBootMenu image. Do not run zpool upgrade without checking bootloader support in the upstream release notes.

Create the pool:

        
        
        
    
zpool create -f \
    -o ashift=12 \
    -o compatibility=openzfs-2.3-linux \
    -O compression=lz4 \
    -O acltype=posixacl \
    -O xattr=sa \
    -O relatime=on \
    -O mountpoint=none \
    -R /mnt \
    rpool \
    raidz2 \
    "${D1}-part2" \
    "${D2}-part2" \
    "${D3}-part2" \
    "${D4}-part2"

I use:

ashift=12

for 4 KiB sectors.

Compression is enabled with:

compression=lz4

For Linux systems using POSIX ACLs, I use:

acltype=posixacl
xattr=sa

Verify the pool:

zpool status
zpool get ashift,compatibility rpool

Creating the root datasets

Create a container for boot environments:

        
zfs create \
    -o canmount=off \
    -o mountpoint=none \
    rpool/ROOT

Create the Ubuntu root filesystem:

        
zfs create \
    -o canmount=noauto \
    -o mountpoint=/ \
    rpool/ROOT/ubuntu

Mount it:

zfs mount rpool/ROOT/ubuntu

Because the pool was imported with:

-R /mnt

the root filesystem appears under /mnt.

Create a separate /home dataset:

        
zfs create \
    -o mountpoint=/home \
    rpool/home

Set the boot filesystem:

        
zpool set bootfs=rpool/ROOT/ubuntu rpool

Set the kernel command line inherited by boot environments:

        
zfs set \
    org.zfsbootmenu:commandline="quiet" \
    rpool/ROOT

There is an important distinction here.

This property:

org.zfsbootmenu:commandline

controls the Linux kernel launched by ZFSBootMenu.

Parameters such as:

zbm.import_policy=force

control ZFSBootMenu itself and are stored in the command line embedded inside the ZFSBootMenu EFI executable.

Installing Ubuntu 26.04 with debootstrap

Bootstrap the system:

debootstrap --arch=amd64 resolute /mnt http://archive.ubuntu.com/ubuntu/

Bind the required virtual filesystems:

        
mount --rbind /dev  /mnt/dev
mount --make-rslave /mnt/dev

mount --rbind /proc /mnt/proc
mount --make-rslave /mnt/proc

mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys

mount --rbind /run /mnt/run
mount --make-rslave /mnt/run

Copy DNS configuration:

cp -L /etc/resolv.conf /mnt/etc/resolv.conf
cp /etc/hostid /mnt/etc/hostid

Enter the new system:

chroot /mnt /bin/bash

All following installation commands run inside the chroot, until the explicit exit in “First boot”. Configure the Ubuntu repositories, including Universe and security updates, before installing packages:

        
        
        
    
rm -f /etc/apt/sources.list
mkdir -p /etc/apt/sources.list.d
cat > /etc/apt/sources.list.d/ubuntu.sources <<'EOF'
Types: deb
URIs: http://archive.ubuntu.com/ubuntu/
Suites: resolute resolute-updates resolute-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu/
Suites: resolute-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
EOF
export LANG=C.UTF-8
export LC_ALL=C.UTF-8

Install the required packages:

        
        
        
    
apt update

apt install -y \
    linux-generic \
    ubuntu-server-minimal \
    openssh-server \
    sudo \
    zfsutils-linux \
    zfs-initramfs \
    dosfstools \
    efibootmgr \
    curl \
    ca-certificates \
    binutils \
    locales \
    netplan.io

Configuring the host ID

A stable /etc/hostid is important for root-on-ZFS systems.

Verify the ID copied from the live environment:

hostid

Verify:

ls -l /etc/hostid

Hostname, user and SSH

Set the hostname:

        
echo nas > /etc/hostname
printf '127.0.1.1\tnas\n' >> /etc/hosts

Create an administrative user:

        
useradd \
    --create-home \
    --shell /bin/bash \
    --groups sudo \
    nasadmin

passwd nasadmin

Enable SSH:

systemctl enable ssh

Network configuration

Example /etc/netplan/01-netcfg.yaml:

        
network:
  version: 2
  renderer: networkd

  ethernets:
    enp1s0:
      dhcp4: true

Set appropriate permissions:

        
chmod 600 /etc/netplan/01-netcfg.yaml
netplan generate

Creating zpool.cache

There is a small but important detail here.

The pool was created with:

-R /mnt

which sets an alternate root and effectively leaves the pool without a persistent cache file.

Create one explicitly:

        
mkdir -p /etc/zfs

zpool set \
    cachefile=/etc/zfs/zpool.cache \
    rpool

Verify:

ls -lh /etc/zfs/zpool.cache

Enable ZFS services for the installed system. enable creates the links needed at boot; it does not try to start a systemd instance inside the chroot:

systemctl enable zfs.target zfs-import-cache zfs-mount zfs-import.target

Rebuild the initramfs:

update-initramfs -u -k all

Check that both the host ID and pool cache were included:

        
lsinitramfs /boot/initrd.img-* \
    | grep -E 'hostid|zpool.cache'

You should see:

etc/hostid
etc/zfs/zpool.cache

Installing ZFSBootMenu

Installing the EFI binary

I use the official prebuilt ZFSBootMenu EFI binary.

Mount the primary ESP at:

/boot/efi

For example:

mkdir -p /boot/efi
mount /dev/disk/by-uuid/AAAA-BBBB /boot/efi

Create the directories:

        
mkdir -p \
    /boot/efi/EFI/ZBM \
    /boot/efi/EFI/BOOT

Download a pinned ZFSBootMenu 3.1.0 release and verify its published SHA256 before installing it. This example selects the x86_64 recovery image with Linux 6.12; its kernel is separate from Ubuntu’s:

        
        
        
    
ZBM_VERSION=v3.1.0
ZBM_IMAGE="zfsbootmenu-recovery-x86_64-${ZBM_VERSION}-linux6.12.EFI"
ZBM_URL="https://github.com/zbm-dev/zfsbootmenu/releases/download/${ZBM_VERSION}"
ZBM_DOWNLOAD=$(mktemp -d)
cd "$ZBM_DOWNLOAD"
curl --fail --location --remote-name "${ZBM_URL}/${ZBM_IMAGE}"
curl --fail --location --remote-name "${ZBM_URL}/sha256.txt"
grep -F "(${ZBM_IMAGE})" sha256.txt > selected.sha256
sha256sum --check selected.sha256 || exit 1
install -m 0644 "$ZBM_IMAGE" /boot/efi/EFI/ZBM/VMLINUZ.EFI

The checksum detects a corrupt download. Upstream also documents signature verification in Binary Releases.

Install zbm-kcl from the source commit tagged v3.1.0; binutils supplies the objcopy command it uses to edit EFI command lines. The release’s sha256.txt covers the EFI images and tarballs, not this standalone script. The digest below was calculated from the pinned upstream source:

        
        
        
    
ZBM_COMMIT=7174e420590a270ece3c6426bdb1d5cdd9ef27b0
curl --fail --location --output zbm-kcl \
    "https://raw.githubusercontent.com/zbm-dev/zfsbootmenu/${ZBM_COMMIT}/bin/zbm-kcl" || exit 1
printf '%s  %s\n' \
    '16edae3eee5df9a0b133734bc4d8a8cb68ca73ac30f98cb21cea3212b051ff01' \
    zbm-kcl | sha256sum --check - || exit 1
install -m 0755 zbm-kcl /usr/local/sbin/zbm-kcl
zbm-kcl /boot/efi/EFI/ZBM/VMLINUZ.EFI

Keep an unmodified recovery image and create the standard UEFI fallback copy:

        
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/BOOT/BOOTX64.EFI
cd /

Mounting only the primary ESP

Only the primary ESP is mounted permanently.

Example /etc/fstab:

UUID=AAAA-BBBB /boot/efi vfat defaults,nofail,umask=0077,x-systemd.device-timeout=5s 0 0

The key option is:

nofail

If the disk containing the primary ESP disappears, the operating system must still be able to boot from another ESP.

Creating UEFI boot entries

Create a UEFI entry for every disk.

Use the stable disk paths exported before entering the chroot:

        
        
        
    
i=1
for disk in "$D1" "$D2" "$D3" "$D4"; do
    efibootmgr --create --disk "$disk" --part 1 \
      --label "ZFSBootMenu HDD${i}" --loader '\EFI\ZBM\VMLINUZ.EFI'
    i=$((i + 1))
done

The backup ESPs will receive their EFI files in the synchronization step below.

Verify:

efibootmgr

The firmware should have entries similar to:

        
ZFSBootMenu HDD1
ZFSBootMenu HDD2
ZFSBootMenu HDD3
ZFSBootMenu HDD4

In addition, every ESP contains:

EFI/BOOT/BOOTX64.EFI

so the firmware can also use the generic UEFI fallback path.

Synchronizing ESPs with zbm-esp-sync

ZFS protects the contents of the pool.

It does not replicate FAT32 ESPs.

For that purpose I use a small tool I wrote called zbm-esp-sync.

The design is intentionally simple:

        
ESP HDD1 = master
       │
       ├── ESP HDD2
       ├── ESP HDD3
       └── ESP HDD4

Only these trees are synchronized:

EFI/ZBM
EFI/BOOT

The utility verifies that the target really is an EFI System Partition and that the filesystem is VFAT.

It does not modify partition tables or format drives.

Install the build dependencies:

        
apt install -y \
    golang-go \
    git \
    make \
    util-linux

Clone and build:

        
        
        
    
git clone --branch v0.1.1 --depth 1 \
    https://gitlab.com/tty8747/zbm-esp-sync \
    /usr/local/src/zbm-esp-sync

cd /usr/local/src/zbm-esp-sync

go test ./...
make VERSION=v0.1.1 build
make VERSION=v0.1.1 install
zbm-esp-sync --version
cd /

The upstream install target places the binary in /usr/local/sbin and installs the systemd units. Replace all example UUIDs in /etc/zbm-esp-sync/config.yaml:

        
        
        
    
master: /dev/disk/by-uuid/AAAA-BBBB

esp:
  - /dev/disk/by-uuid/AAAA-BBBB
  - /dev/disk/by-uuid/CCCC-DDDD
  - /dev/disk/by-uuid/EEEE-FFFF
  - /dev/disk/by-uuid/GGGG-HHHH

paths:
  - EFI/ZBM
  - EFI/BOOT

Test first:

sudo zbm-esp-sync refresh --dry-run

Then synchronize:

        
sudo zbm-esp-sync refresh
sudo zbm-esp-sync verify
sudo zbm-esp-sync status

A healthy setup should look roughly like:

        
ESP                          ROLE    MOUNTED  FILES  CHECKSUM  STATE
/dev/disk/by-uuid/...        master  yes      ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current

Enable automatic synchronization for the next boot. Do not start services with --now inside the chroot:

systemctl enable zbm-esp-sync.path

The path unit watches the master ESP for changes to:

EFI/ZBM
EFI/BOOT

When files change, synchronization is triggered automatically.

What happens when the primary ESP disappears

If HDD1 is physically missing, the command:

sudo zbm-esp-sync status

may report something similar to:

master ... not a block device

This is expected.

zbm-esp-sync is not involved in booting Linux.

The NAS can continue running on the remaining disks, but I prefer not to update ZFSBootMenu while the master ESP is absent.

The degraded-pool boot problem

The first physical failure test exposed an interesting issue.

With HDD1 removed, ZFSBootMenu itself started successfully from another disk, but the ZFS pool was not automatically imported.

The screen repeatedly showed:

Unable to import pool

From the ZFSBootMenu recovery shell, this worked:

zpool import -f -N rpool

After that, Ubuntu booted normally.

That tells us that the redundancy of the RAIDZ2 pool itself is fine, but the import policy is blocking unattended degraded boot.

For this standalone NAS I changed the ZFSBootMenu import policy to:

zbm.import_policy=force

Warning

zbm.import_policy=force makes ZFSBootMenu use zpool import -f. Enable it only when the pool cannot be active on another host; importing the same pool on two hosts can corrupt data. See the ZFSBootMenu import policy documentation.

Changing the ZFSBootMenu import policy

For a prebuilt ZFSBootMenu EFI binary, the command line can be edited using zbm-kcl.

Inspect the current value:

        
sudo zbm-kcl \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI

Change the import policy:

        
sudo zbm-kcl \
    -r zbm.import_policy \
    -a zbm.import_policy=force \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI

Verify:

        
sudo zbm-kcl \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI

In my setup the result is:

quiet loglevel=0 nomodeset zbm.import_policy=force

Update the generic fallback copy:

        
sudo cp \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI \
    /boot/efi/EFI/BOOT/BOOTX64.EFI

Then synchronize all ESPs:

sudo zbm-esp-sync refresh
sudo zbm-esp-sync verify

I intentionally keep:

VMLINUZ-BACKUP.EFI

with the normal hostid-based import policy.

That gives me an emergency ZFSBootMenu image that does not force-import pools.

First boot

Before leaving the chroot, check the root dataset, boot files and ESP copies:

        
zpool get bootfs rpool
zfs list
ls -l /boot/vmlinuz-* /boot/initrd.img-*
zbm-esp-sync verify
efibootmgr

Exit to the live shell, unmount the installation tree and export the pool. If unmount or export fails, stop and resolve the busy mount before rebooting:

        
exit
cd /
umount --recursive /mnt
zpool export rpool

After both commands succeed, reboot and remove the installation medium:

reboot

        
        
        
    
findmnt /
zpool status
hostid
sudo systemctl start zbm-esp-sync.service
sudo systemctl enable --now zbm-esp-sync.path
systemctl status zbm-esp-sync.path
sudo journalctl -u zbm-esp-sync.service --no-pager -n 30
sudo zbm-esp-sync verify

Confirm normal boot with all disks present before testing missing disks.

Testing fault tolerance

RAIDZ2 theory is useful, but boot redundancy should be tested physically.

Always shut the NAS down before removing drives:

sudo poweroff

One-disk failure

First I removed HDD1.

The system successfully booted from another ESP.

The pool became:

        
        
        
    
rpool       DEGRADED
  raidz2-0  DEGRADED
    HDD1    UNAVAIL
    HDD2    ONLINE
    HDD3    ONLINE
    HDD4    ONLINE

The filesystems were still mounted normally:

/      rpool/ROOT/ubuntu
/home  rpool/home

SSH came up without manual intervention.

After reinstalling HDD1, ZFS performed a very small resilver and the pool returned to:

ONLINE

Two-disk failure

The more important test was removing HDD1 and HDD2 at the same time.

Only HDD3 and HDD4 remained.

The boot sequence was:

        
        
        
    
UEFI
 ↓
ESP on HDD3
 ↓
ZFSBootMenu
 ↓
zbm.import_policy=force
 ↓
rpool DEGRADED
 ↓
Ubuntu
 ↓
SSH

The pool looked like this:

        
rpool       DEGRADED
  raidz2-0  DEGRADED
    HDD1    UNAVAIL
    HDD2    UNAVAIL
    HDD3    ONLINE
    HDD4    ONLINE

errors: No known data errors

This confirmed not only that RAIDZ2 can operate with two missing drives, but also that the entire boot path remains functional.

That includes:

        
        
        
    
firmware
ESP
ZFSBootMenu
pool import
root-on-ZFS
systemd
network
SSH

ZFS tuning for a NAS

Will zbm-esp-sync wear out the ESPs?

No.

There are two separate questions here.

First, these ESPs live on mechanical HDDs, not NAND flash.

There is no SSD-style TBW endurance concern.

Second, zbm-esp-sync does not rewrite the partitions continuously.

Normal operation looks like:

        
        
        
    
ZFSBootMenu unchanged
        ↓
nothing is written

ZFSBootMenu updated
        ↓
master ESP changes
        ↓
systemd.path triggers
        ↓
one synchronization run
        ↓
backup ESPs are updated

The tool compares content and only updates changed files.

It writes temporary files, renames them into place and verifies the resulting data.

It never modifies GPT or formats filesystems.

A more relevant HDD wear issue is excessive power cycling or aggressive spindown.

For mechanical disks, I would pay much more attention to:

        
Start_Stop_Count
Load_Cycle_Count
Power_Cycle_Count
Power_On_Hours
Temperature

than to a few dozen megabytes written to an ESP during a bootloader update.

Should I create a RAM disk for ZFS cache?

No.

ZFS already has a native RAM cache called ARC.

On this NAS with 16 GB of RAM, the system reported approximately:

        
Total RAM:      15.4 GiB
ARC max:        14.4 GiB
ARC current:    ~143 MiB

The ARC was still tiny because the NAS had just been installed and there was almost no workload.

There is no benefit in doing this:

        
        
        
    
RAM
 ↓
tmpfs
 ↓
L2ARC
 ↓
ZFS

ZFS already does:

        
        
        
    
RAM
 ├── ARC
 │    └── read cache
 │
 └── dirty data
      └── write buffering
             ↓
            ZFS

L2ARC is meant to be a second-level read cache, usually on SSD or NVMe, when the working set no longer fits into RAM.

It is not a replacement for ARC.

Likewise, a SLOG is not a generic write cache.

For now I leave ARC tuning at its defaults:

zfs_arc_max=0
zfs_arc_min=0

This allows OpenZFS to manage memory dynamically.

Useful commands:

zarcsummary

and:

zarcstat 5

Dataset layout for NAS data

I prefer not to store all user data directly inside one generic dataset.

Separate datasets make it possible to apply different snapshot policies, quotas and filesystem properties.

For example:

        
        
        
    
sudo zfs create \
    -o mountpoint=/srv/data \
    rpool/data

sudo zfs create \
    -o mountpoint=/srv/data/shared \
    -o atime=off \
    rpool/data/shared

sudo zfs create \
    -o mountpoint=/srv/data/documents \
    -o atime=off \
    rpool/data/documents

sudo zfs create \
    -o mountpoint=/srv/data/media \
    -o atime=off \
    rpool/data/media

sudo zfs create \
    -o mountpoint=/srv/data/backups \
    -o atime=off \
    rpool/data/backups

The result is:

        
        
        
    
rpool
├── ROOT
│   └── ubuntu
├── home
└── data
    ├── shared
    ├── documents
    ├── media
    └── backups

For Samba-oriented datasets I use:

        
acltype=posixacl
xattr=sa
compression=lz4

Disabling atime is useful for datasets where applications do not care about file access timestamps.

I would not blindly set a large recordsize everywhere.

recordsize should be tuned per workload.

Large sequential media or backup files may benefit from a larger value, but mixed file workloads are often better left at the default.

Data protection

ZFS snapshots

RAIDZ2 protects against physical disk failure.

It does not protect against:

        
rm -rf
accidental overwrite
ransomware
broken application logic
operator mistakes

Snapshots help with these problems.

A manual snapshot:

        
SNAPSHOT="manual-$(date -u +%Y%m%dT%H%M%SZ)"
sudo zfs snapshot -r "rpool/data@${SNAPSHOT}"

The -r option includes shared, documents, media and backups. Without it, only the parent dataset is snapshotted. Verify all child snapshots:

zfs list -t snapshot -r rpool/data

Snapshots are very cheap to create because they are initially just references to existing blocks.

However, a snapshot is not a backup.

If the entire pool disappears, the snapshots disappear with it.

Automatic snapshots with Sanoid

For automated retention policies, I use Sanoid.

Install it:

sudo apt install sanoid

Example /etc/sanoid/sanoid.conf:

        
        
        
    
[rpool/data]
    use_template = nas
    recursive = yes

[rpool/ROOT/ubuntu]
    use_template = system

[template_nas]
    frequently = 0
    hourly = 24
    daily = 30
    weekly = 8
    monthly = 12
    yearly = 0
    autosnap = yes
    autoprune = yes

[template_system]
    frequently = 0
    hourly = 0
    daily = 7
    weekly = 4
    monthly = 3
    yearly = 0
    autosnap = yes
    autoprune = yes

This keeps approximately:

        
        
        
    
NAS data:
hourly
daily
weekly
monthly

Ubuntu root:
daily
weekly
monthly

Enable the timer:

sudo systemctl enable --now sanoid.timer

Check it:

systemctl status sanoid.timer
sudo sanoid --monitor-snapshots

For off-host backup, Sanoid is commonly paired with Syncoid and ZFS replication.

Regular ZFS scrub

RAIDZ2 provides redundancy against disk failures. Regular scrubs complement that protection by detecting latent corruption: a scrub reads allocated data, verifies checksums and repairs damaged blocks from redundancy when possible. See the OpenZFS scrub documentation.

Start a manual scrub:

sudo zpool scrub rpool

Wait until it finishes:

sudo zpool scrub -w rpool

Then check:

sudo zpool status -v

A healthy result looks like:

scan: scrub repaired 0B ... with 0 errors
errors: No known data errors

Ubuntu ships systemd timers for automatic scrubs.

For a home NAS with large spinning disks, I use a monthly scrub:

        
sudo systemctl enable --now \
    zfs-scrub-monthly@rpool.timer

Verify:

systemctl list-timers 'zfs-scrub*'

A weekly full scrub may be unnecessary for a large mostly-static home NAS, while monthly is a reasonable starting point.

Disk health monitoring

SMART monitoring

ZFS and SMART solve different problems.

ZFS sees things like:

        
READ errors
WRITE errors
checksum errors

SMART sees the internal state of the physical drive.

Install smartmontools:

        
sudo apt install smartmontools
sudo systemctl enable --now smartmontools.service

Inspect a disk using a stable by-id path:

        
sudo smartctl -x \
    /dev/disk/by-id/ata-DISK

For WD HDDs, I pay particular attention to:

        
sudo smartctl -A \
    /dev/disk/by-id/ata-DISK \
    | grep -E \
'Power_On_Hours|Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable|UDMA_CRC_Error_Count|Start_Stop_Count|Load_Cycle_Count|Temperature'

A healthy new drive should generally start around:

        
Reallocated_Sector_Ct   0
Current_Pending_Sector  0
Offline_Uncorrectable   0
UDMA_CRC_Error_Count    0

A single SMART value is not always enough to condemn a disk.

What matters is the trend together with ZFS errors and SMART self-test results.

Scheduled SMART self-tests

smartd can run regular self-tests.

A simple /etc/smartd.conf example:

DEVICESCAN -a -s (S/../.././02|L/../../7/04:003)

Short tests start daily in the 02:00–03:00 window, using the NAS’s local time. Long tests are scheduled on Sundays, initially in the 04:00–05:00 window; :003 shifts their start windows by three hours for each successive device. For four detected drives, the long-test windows start at 04:00, 07:00, 10:00 and 13:00. Actual starts depend on the daemon’s polling cycle, and long tests may still overlap if they take more than three hours. This syntax is described in smartd.conf(5).

DEVICESCAN applies to all discovered devices, including any supported USB DOM. Use explicit device entries if only the four pool HDDs should be tested, and check the resulting schedule below.

After editing the configuration:

sudo systemctl restart smartmontools

Check the schedule:

sudo smartd -q showtests

Check logs:

journalctl -u smartmontools

Prometheus monitoring and alerting

Why monitoring matters

RAID without alerts is dangerous.

The failure sequence you want to avoid is:

        
        
        
    
HDD1 fails
 ↓
RAIDZ2 keeps working
 ↓
nobody notices
 ↓
HDD2 fails weeks later
 ↓
still nobody notices
 ↓
HDD3 fails
 ↓
pool is lost

The first degraded state should result in an alert within minutes.

I use Prometheus and Alertmanager for this.

Prometheus itself should preferably run somewhere other than the NAS.

If the NAS loses power completely, an external monitoring server can still detect that it disappeared.

Installing Prometheus exporters

On the NAS:

        
sudo apt install \
    prometheus-node-exporter \
    prometheus-smartctl-exporter

Enable them:

        
sudo systemctl enable --now \
    prometheus-node-exporter

sudo systemctl enable --now \
    prometheus-smartctl-exporter

Check node exporter:

curl -s localhost:9100/metrics | head

Check SMART exporter:

curl -s localhost:9633/metrics | head

Monitoring ZFS pool state

Node exporter has a ZFS collector.

A particularly useful metric is:

node_zfs_zpool_state

Check it:

        
curl -s localhost:9100/metrics \
    | grep node_zfs_zpool_state

You may see something similar to:

node_zfs_zpool_state{state="online",zpool="rpool"} 1
node_zfs_zpool_state{state="degraded",zpool="rpool"} 0

This makes ZFS pool-state alerting straightforward.

SMART metrics

The SMART exporter exposes metrics such as:

        
        
        
    
smartctl_device
smartctl_device_smart_status
smartctl_device_status
smartctl_device_smartctl_exit_status
smartctl_device_temperature
smartctl_device_attribute

Inspect the actual output generated by the version installed on your system:

        
curl -s localhost:9633/metrics \
    | grep smartctl_device_smart_status

and:

        
curl -s localhost:9633/metrics \
    | grep smartctl_device_temperature

Inspect the device inventory and diagnostic status as well:

        
curl --fail --silent --show-error localhost:9633/metrics \
    | grep -E '^smartctl_device(_status|_smartctl_exit_status)?\{'

I always inspect the real exporter output before writing PromQL rules, because exact labels and ATA attributes can vary between drive models. Compare health results with device status and the smartctl exit status when diagnosing a failure. The latter is a bitmask, so a nonzero value does not necessarily mean the command could not read the device.

In the upstream metric example, model_name and serial_number are labels of smartctl_device, not of every SMART metric. Verify those labels locally before using the WD model selector below; replace it if your pool uses different drives.

Prometheus scrape configuration

Example:

        
        
        
    
scrape_configs:
  - job_name: nas-node
    static_configs:
      - targets:
          - nas:9100

  - job_name: nas-smart
    static_configs:
      - targets:
          - nas:9633

Alert when the NAS disappears

        
- alert: NASDown
  expr: up{job="nas-node"} == 0
  for: 2m

  labels:
    severity: critical

  annotations:
    summary: "NAS is unreachable"

This alert must live outside the NAS itself.

Alert when the SMART exporter disappears

The SMART exporter is a separate scrape target. Node exporter can remain healthy while SMART monitoring is unavailable:

        
- alert: SMARTExporterDown
  expr: up{job="nas-smart"} == 0
  for: 2m

  labels:
    severity: warning

  annotations:
    summary: "SMART exporter on NAS is unreachable"

Keep both targets in the scrape configuration. Removing a target also removes its up series, so these availability alerts no longer cover it.

Alert when the ZFS pool is not ONLINE

        
        
        
    
- alert: ZFSPoolNotOnline
  expr: |
    (
      node_zfs_zpool_state{
        job="nas-node",
        zpool="rpool",
        state="online"
      } != 1
    )
    or
    (
      up{job="nas-node"} == 1
      unless on (job, instance)
      node_zfs_zpool_state{
        job="nas-node",
        zpool="rpool",
        state="online"
      }
    )

  for: 1m

  labels:
    severity: critical

  annotations:
    summary: "ZFS pool rpool is not ONLINE or is missing"

A removed or failed HDD should cause this alert quickly. The second branch also detects a missing pool-state metric for each reachable nas-node target; NASDown covers failed scrapes. The PromQL unless operator matches by job and instance, so a healthy NAS cannot hide a missing metric on another one. These examples assume every target in this job has rpool.

Alert when expected HDD metrics disappear

For this four-WD60EFRX setup, check the number of matching devices per NAS. An ungrouped count could combine drives from different machines. If no matching devices remain, count returns no series; the or branch below supplies zero for each configured target:

        
        
        
    
- alert: SMARTPoolDisksMissing
  expr: |
    (
      (
        count by (job, instance) (
          smartctl_device{job="nas-smart", model_name=~".*WD60EFRX.*"}
        )
        or on (job, instance)
        (0 * up{job="nas-smart"})
      ) < 4
    )
    and on (job, instance)
    (up{job="nas-smart"} == 1)

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "SMART exporter reports fewer than four expected pool HDDs"

SMARTExporterDown covers an unreachable exporter. This inventory rule checks only that four matching devices are reported; it does not establish their health or ZFS membership. If other disks share the same model, select the four expected serial numbers instead. Update the selector after replacing a disk.

SMART health alert

        
- alert: SMARTHealthFailed
  expr: |
    smartctl_device_smart_status{job="nas-smart"} == 0

  for: 2m

  labels:
    severity: critical

  annotations:
    summary: "SMART health check failed"

This covers every device exposed by the nas-smart job. If the exporter also sees a USB DOM, NVMe drive or other storage, restrict the rule to the pool HDDs when that is the intended scope. To filter by model or serial, join the health metric to smartctl_device with and on (job, instance, device); adding a model_name filter directly to a metric without that label would hide it. Missing device series are handled separately by the inventory rule above.

Alert on bad or pending sectors

Example:

        
        
        
    
- alert: SMARTBadSectors
  expr: |
    smartctl_device_attribute{
      attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",
      attribute_value_type="raw"
    } > 0

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "SMART reports bad or pending sectors"

Depending on your exporter version, inspect the actual metric labels first.

Alert on growing SATA CRC errors

CRC errors are especially useful as a rate rather than an absolute counter.

Example:

        
        
        
    
- alert: SATACRCErrorIncreasing
  expr: |
    increase(
      smartctl_device_attribute{
        attribute_name="UDMA_CRC_Error_Count",
        attribute_value_type="raw"
      }[1h]
    ) > 0

  labels:
    severity: warning

  annotations:
    summary: "SATA CRC error counter is increasing"

A growing CRC counter can point to:

        
SATA cabling
backplane problems
connector problems
controller issues

rather than platter failure.

HDD temperature alert

Example:

        
- alert: HDDTemperatureHigh
  expr: |
    smartctl_device_temperature{
      temperature_type="current"
    } > 50

  for: 10m

  labels:
    severity: warning

  annotations:
    summary: "HDD temperature is high"

The 50°C value is an example warning threshold.

Use the specification of your actual HDD model when choosing final limits.

Monitoring the boot ESP replicas

Manual ESP verification

This NAS has another failure domain that a typical ZFS system does not have:

4 independent ESPs

So I also want to know whether they are still synchronized.

Manual verification:

sudo zbm-esp-sync verify

A failed verification returns a non-zero status.

I would not run a full SHA256 verification every minute, especially if HDD spindown is enabled.

Once per day is sufficient.

Exporting ESP status to Prometheus

Node exporter includes a textfile collector. The script below uses /var/lib/prometheus/node-exporter, the packaged collector directory. Verify the service configuration on the installed version:

systemctl cat prometheus-node-exporter
cat /etc/default/prometheus-node-exporter

If you override its arguments, retain --collector.textfile.directory=/var/lib/prometheus/node-exporter. Restart the exporter after changing them and confirm the metrics appear below.

Create a small monitoring script:

        
        
        
    
sudo tee /usr/local/sbin/zbm-esp-monitor >/dev/null <<'EOF'
#!/usr/bin/env bash

set -euo pipefail

DIR=/var/lib/prometheus/node-exporter
OUT="$DIR/zbm-esp.prom"
install -d -m 0755 "$DIR"
TMP=$(mktemp "$DIR/.zbm-esp.prom.XXXXXX")
trap 'rm -f "$TMP"' EXIT

if /usr/local/sbin/zbm-esp-sync verify >/dev/null 2>&1; then
    ok=1
else
    ok=0
fi

{
    echo '# HELP zbm_esp_sync_ok ZFSBootMenu ESP replicas match the master'
    echo '# TYPE zbm_esp_sync_ok gauge'
    echo "zbm_esp_sync_ok $ok"

    echo '# HELP zbm_esp_sync_last_check_timestamp_seconds Last ESP verification time'
    echo '# TYPE zbm_esp_sync_last_check_timestamp_seconds gauge'
    echo "zbm_esp_sync_last_check_timestamp_seconds $(date +%s)"
} > "$TMP"

chmod 0644 "$TMP"
mv "$TMP" "$OUT"
EOF

sudo chmod 0755 \
    /usr/local/sbin/zbm-esp-monitor

Verify the actual binary location:

command -v zbm-esp-sync

and adjust the script if required.

Daily ESP verification with systemd

Create a service:

        
        
        
    
sudo tee \
  /etc/systemd/system/zbm-esp-monitor.service \
  >/dev/null <<'EOF'
[Unit]
Description=Verify ZFSBootMenu EFI replicas

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/zbm-esp-monitor
EOF

Create a timer:

        
        
        
    
sudo tee \
  /etc/systemd/system/zbm-esp-monitor.timer \
  >/dev/null <<'EOF'
[Unit]
Description=Periodic ZFSBootMenu ESP verification

[Timer]
OnCalendar=*-*-* 06:15:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable it:

        
sudo systemctl daemon-reload

sudo systemctl start zbm-esp-monitor.service
sudo systemctl enable --now \
    zbm-esp-monitor.timer

Check:

        
systemctl list-timers \
    zbm-esp-monitor.timer

Verify the metrics:

        
curl -s localhost:9100/metrics \
    | grep '^zbm_esp'

Expected output:

zbm_esp_sync_ok 1
zbm_esp_sync_last_check_timestamp_seconds ...

Alert when ESPs are out of sync

        
- alert: ZFSBootMenuESPsOutOfSync
  expr: zbm_esp_sync_ok != 1

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "ZFSBootMenu ESP replicas are not synchronized"

Also monitor whether the check itself has stopped running:

        
        
        
    
- alert: ZFSBootMenuESPCheckStale
  expr: |
    (time() - zbm_esp_sync_last_check_timestamp_seconds{job="nas-node"} > 90000)
    or
    (up{job="nas-node"} == 1
      unless on (job, instance)
      zbm_esp_sync_last_check_timestamp_seconds{job="nas-node"})

  labels:
    severity: warning

  annotations:
    summary: "ZFSBootMenu ESP verification is stale or missing"

The second branch detects a missing metric for each healthy NAS scrape target. NASDown handles targets that cannot be scraped. Keep the NAS target in the Prometheus configuration: removing a target also removes its up series.

ZFS Event Daemon

I also keep ZED enabled:

        
sudo systemctl enable --now \
    zfs-zed.service

Inspect events manually:

sudo zpool events -v

ZED gives another independent channel for reacting to ZFS events.

The monitoring stack therefore looks like this:

        
        
        
    
                 NAS
                  │
        ┌─────────┼─────────┐
        │         │         │
       ZFS       SMART     ESPs
        │         │         │
 node_exporter    │    zbm-esp-monitor
        │         │         │
        └─────────┼─────────┘
                  │
              Prometheus
                  │
             Alertmanager
                  │
           email / Telegram

And independently:

        
ZFS kernel
    ↓
   ZED

Power management

Suspend and Wake-on-LAN

Ubuntu Server can suspend if the platform supports it.

Check available power states:

cat /sys/power/state
cat /sys/power/mem_sleep

Suspend manually:

sudo systemctl suspend

For network wake-up, use Wake-on-LAN.

Install ethtool:

sudo apt install ethtool

Check NIC capabilities:

        
sudo ethtool enp1s0 \
    | grep -i Wake-on

For example:

Supports Wake-on: pumbg
Wake-on: d

If g is present, Magic Packet wake-up is supported.

Enable it:

sudo ethtool -s enp1s0 wol g

Netplan can keep the setting persistent:

        
network:
  version: 2
  renderer: networkd

  ethernets:
    enp1s0:
      dhcp4: true
      wakeonlan: true

There is an important limitation here.

A sleeping server generally will not wake merely because somebody tries to open an SMB share.

The usual workflow is:

        
client sends WoL Magic Packet
        ↓
NAS wakes up
        ↓
Samba/SSH becomes available

For a NAS that needs to remain reachable at all times, normal idle may be preferable to full suspend.

I would also avoid overly aggressive HDD spindown, because SMART, monitoring, ZFS metadata updates and snapshots may repeatedly wake the drives and generate excessive start/stop cycles.

Final architecture

The final boot architecture is:

        
        
        
    
                    TerraMaster
                         │
                ┌────────┴────────┐
                │                 │
              Boot               Data
                │                 │
         4 independent ESPs      rpool
                │                 │
         ZFSBootMenu            RAIDZ2
                │          ┌──────┼──────┐
         zbm-esp-sync       │      │      │
                │         HDD1   HDD2   HDD3/HDD4
                │
       automatic force import
                │
        rpool/ROOT/ubuntu

Operationally:

        
        
        
    
RAIDZ2
  │
  ├── monthly scrub
  │
  ├── SMART monitoring
  │     ├── short tests
  │     └── staggered long tests
  │
  ├── datasets
  │     ├── shared
  │     ├── documents
  │     ├── media
  │     └── backups
  │
  ├── Sanoid snapshots
  │
  ├── node_exporter
  │
  ├── smartctl_exporter
  │
  ├── ESP monitoring
  │
  ├── ZED
  │
  └── Prometheus + Alertmanager

What RAIDZ2 does not solve

RAIDZ2 is not a backup.

It protects against disk failures.

It does not protect against:

        
        
        
    
accidental deletion
ransomware
application corruption
administrator mistakes
NAS theft
fire
power supply failure affecting several drives
complete pool loss

Snapshots help with some logical failures, but they still live on the same pool.

A complete design should eventually look like:

        
        
        
    
              Primary NAS
                  │
        RAIDZ2 + snapshots
                  │
           ZFS replication
                  │
                  ▼
        separate physical NAS
          or backup server

For a ZFS-based system, zfs send / zfs receive or Syncoid are natural choices.

Conclusion

My original goal was to build a NAS that does not rely on a separate system SSD and can survive the loss of any two HDDs.

The result is a system where Ubuntu itself lives inside the RAIDZ2 pool, while every disk contains an independently bootable EFI System Partition.

ZFS provides data redundancy.

ZFSBootMenu provides a clean root-on-ZFS boot path.

Four ESPs eliminate the single boot-device failure point.

zbm-esp-sync keeps the boot partitions synchronized.

zbm.import_policy=force allows this standalone NAS to import a degraded pool without manual recovery.

Monthly scrubs detect latent corruption.

SMART monitors physical drive health.

Sanoid provides snapshot retention.

Prometheus and Alertmanager make sure that the first failed disk does not remain unnoticed.

And there is no need to build a RAM disk for ZFS caching, because ARC already provides the correct in-memory cache layer.

Most importantly, this was not only tested in theory.

The NAS successfully booted in all of these states:

        
4 disks present
    → ONLINE

1 disk missing
    → DEGRADED
    → boot successful

2 disks missing
    → DEGRADED
    → boot successful

With two physical HDDs removed, the machine still booted from an ESP on one of the remaining disks, ZFSBootMenu imported the RAIDZ2 pool automatically, Ubuntu mounted rpool/ROOT/ubuntu, networking came up, and SSH became available without manual intervention.

That is exactly the failure mode I wanted: losing two disks should result in a degraded NAS that needs maintenance, not an unbootable server that needs a rescue USB.