ZFS RAIDZ2 with ZFSBootMenu: A Fault-Tolerant 4-Disk NAS

2026-09-15 4975 words 24 minutes

Contents

In my previous article, I described how to install Ubuntu on a ZFS mirror and boot it using ZFSBootMenu.

This time, the goal is more ambitious: build a proper four-disk NAS where both the operating system and user data live on a single ZFS RAIDZ2 pool, while still being able to boot the machine after losing one or even two physical disks.

I also wanted to avoid a dedicated boot SSD.

The final design looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


HDD1
├── 1 GiB ESP
└── ZFS ─┐

HDD2     │
├── ESP  │
└── ZFS ─┤
         │
HDD3     ├── rpool / RAIDZ2
├── ESP  │
└── ZFS ─┤
         │
HDD4     │
├── ESP  │
└── ZFS ─┘

Each disk has its own EFI System Partition, containing a copy of ZFSBootMenu.

The ZFS pool itself contains Ubuntu and NAS data.

The system was also tested by physically removing disks, including a two-disk failure scenario.

Hardware and software

My setup is:

Component	Configuration
NAS	TerraMaster F4-425
HDDs	4 × WD Red WD60EFRX, 6 TB
RAM	16 GB
OS	Ubuntu Server 26.04 LTS
Kernel	Linux 7.0.x
OpenZFS	2.4.x
Pool layout	RAIDZ2
Bootloader	ZFSBootMenu
ESPs	One 1 GiB ESP per HDD

The internal TerraMaster USB DOM is left untouched.

The important design decision is that every HDD is independently bootable.

A disk therefore looks like this:

1
2
3


Disk
├── GPT partition 1: EFI System Partition
└── GPT partition 2: ZFS

And each ESP contains:

1
2


EFI/ZBM/VMLINUZ.EFI
EFI/BOOT/BOOTX64.EFI

The second file is the standard UEFI fallback path.

Installing Ubuntu on RAIDZ2

Preparing the disks

First, identify the drives.

Do not rely on /dev/sda, /dev/sdb, and similar names when creating the ZFS pool.

Use stable /dev/disk/by-id paths instead.

For example:

1
2


ls -l /dev/disk/by-id/
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE

Define the disks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


D1=/dev/disk/by-id/ata-DISK1
D2=/dev/disk/by-id/ata-DISK2
D3=/dev/disk/by-id/ata-DISK3
D4=/dev/disk/by-id/ata-DISK4

DISKS=(
  "$D1"
  "$D2"
  "$D3"
  "$D4"
)

The following commands destroy the existing partition tables, so double-check the selected disks before continuing.

Create GPT tables and two partitions per disk:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


for d in "${DISKS[@]}"; do
    wipefs -a "$d"
    sgdisk --zap-all "$d"

    sgdisk \
      -n 1:1M:+1G \
      -t 1:EF00 \
      -c 1:EFI \
      "$d"

    sgdisk \
      -n 2:0:0 \
      -t 2:6A85CF4D-1DD2-11B2-99A6-080020736631 \
      -c 2:ZFS \
      "$d"
done

partprobe
udevadm settle

Verify:

1

lsblk -o NAME,SIZE,PARTTYPE,FSTYPE

Creating the four EFI System Partitions

Format partition 1 on every disk:

1
2
3
4


mkfs.vfat -F 32 -n ZBM1 "${D1}-part1"
mkfs.vfat -F 32 -n ZBM2 "${D2}-part1"
mkfs.vfat -F 32 -n ZBM3 "${D3}-part1"
mkfs.vfat -F 32 -n ZBM4 "${D4}-part1"

Then get their UUIDs:

1

lsblk -o PATH,FSTYPE,LABEL,UUID,PARTTYPE

These UUIDs will later be used by /etc/fstab and zbm-esp-sync.

Creating the RAIDZ2 pool

From the Ubuntu live environment, install the required tools:

1
2
3
4
5
6
7


apt update

apt install -y \
    debootstrap \
    gdisk \
    dosfstools \
    zfsutils-linux

Create the pool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


zpool create -f \
    -o ashift=12 \
    -o compatibility=openzfs-2.3-linux \
    -O compression=lz4 \
    -O acltype=posixacl \
    -O xattr=sa \
    -O relatime=on \
    -O mountpoint=none \
    -R /mnt \
    rpool \
    raidz2 \
    "${D1}-part2" \
    "${D2}-part2" \
    "${D3}-part2" \
    "${D4}-part2"

I use:

1

ashift=12

for 4 KiB sectors.

Compression is enabled with:

1

compression=lz4

For Linux systems using POSIX ACLs, I use:

1
2


acltype=posixacl
xattr=sa

Verify the pool:

1
2


zpool status
zpool get ashift,compatibility rpool

Creating the root datasets

Create a container for boot environments:

1
2
3
4


zfs create \
    -o canmount=off \
    -o mountpoint=none \
    rpool/ROOT

Create the Ubuntu root filesystem:

1
2
3
4


zfs create \
    -o canmount=noauto \
    -o mountpoint=/ \
    rpool/ROOT/ubuntu

Mount it:

1

zfs mount rpool/ROOT/ubuntu

Because the pool was imported with:

1

-R /mnt

the root filesystem appears under /mnt.

Create a separate /home dataset:

1
2
3


zfs create \
    -o mountpoint=/home \
    rpool/home

Set the boot filesystem:

1

zpool set bootfs=rpool/ROOT/ubuntu rpool

Set the kernel command line inherited by boot environments:

1
2
3


zfs set \
    org.zfsbootmenu:commandline="quiet" \
    rpool/ROOT

There is an important distinction here.

This property:

1

org.zfsbootmenu:commandline

controls the Linux kernel launched by ZFSBootMenu.

Parameters such as:

1

zbm.import_policy=force

control ZFSBootMenu itself and are stored in the command line embedded inside the ZFSBootMenu EFI executable.

Installing Ubuntu 26.04 with debootstrap

Bootstrap the system:

1

debootstrap resolute /mnt

Bind the required virtual filesystems:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


mount --rbind /dev  /mnt/dev
mount --make-rslave /mnt/dev

mount --rbind /proc /mnt/proc
mount --make-rslave /mnt/proc

mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys

mount --rbind /run /mnt/run
mount --make-rslave /mnt/run

Copy DNS configuration:

1

cp -L /etc/resolv.conf /mnt/etc/resolv.conf

Enter the new system:

1

chroot /mnt /bin/bash

Install the required packages:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apt update

apt install -y \
    linux-generic \
    ubuntu-server-minimal \
    openssh-server \
    sudo \
    zfsutils-linux \
    zfs-initramfs \
    dosfstools \
    efibootmgr \
    curl \
    locales \
    netplan.io

Configuring the host ID

A stable /etc/hostid is important for root-on-ZFS systems.

Generate it:

1
2


zgenhostid
hostid

Verify:

1

ls -l /etc/hostid

Hostname, user and SSH

Set the hostname:

1

echo nas > /etc/hostname

Create an administrative user:

1
2
3
4
5
6
7


useradd \
    --create-home \
    --shell /bin/bash \
    --groups sudo \
    goto

passwd goto

Enable SSH:

1

systemctl enable ssh

Network configuration

Example /etc/netplan/01-netcfg.yaml:

1
2
3
4
5
6
7


network:
  version: 2
  renderer: networkd

  ethernets:
    enp1s0:
      dhcp4: true

Set appropriate permissions:

1

chmod 600 /etc/netplan/01-netcfg.yaml

Creating zpool.cache

There is a small but important detail here.

The pool was created with:

1

-R /mnt

which sets an alternate root and effectively leaves the pool without a persistent cache file.

Create one explicitly:

1
2
3
4
5


mkdir -p /etc/zfs

zpool set \
    cachefile=/etc/zfs/zpool.cache \
    rpool

Verify:

1

ls -lh /etc/zfs/zpool.cache

Rebuild the initramfs:

1

update-initramfs -u -k all

Check that both the host ID and pool cache were included:

1
2


lsinitramfs /boot/initrd.img-* \
    | grep -E 'hostid|zpool.cache'

You should see:

1
2


etc/hostid
etc/zfs/zpool.cache

Installing ZFSBootMenu

Installing the EFI binary

I use the official prebuilt ZFSBootMenu EFI binary.

Mount the primary ESP at:

1

/boot/efi

For example:

1
2


mkdir -p /boot/efi
mount /dev/disk/by-uuid/AAAA-BBBB /boot/efi

Create the directories:

1
2
3


mkdir -p \
    /boot/efi/EFI/ZBM \
    /boot/efi/EFI/BOOT

Place the ZFSBootMenu EFI executable at:

1

/boot/efi/EFI/ZBM/VMLINUZ.EFI

Create the standard UEFI fallback copy:

1
2
3


cp \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI \
    /boot/efi/EFI/BOOT/BOOTX64.EFI

I also keep:

1

EFI/ZBM/VMLINUZ-BACKUP.EFI

as an additional recovery copy.

Mounting only the primary ESP

Only the primary ESP is mounted permanently.

Example /etc/fstab:

1

UUID=AAAA-BBBB /boot/efi vfat defaults,nofail,umask=0077,x-systemd.device-timeout=5s 0 0

The key option is:

1

nofail

If the disk containing the primary ESP disappears, the operating system must still be able to boot from another ESP.

Creating UEFI boot entries

Create a UEFI entry for every disk.

For example:

1
2
3
4
5
6


efibootmgr \
    --create \
    --disk /dev/sda \
    --part 1 \
    --label 'ZFSBootMenu HDD1' \
    --loader '\EFI\ZBM\VMLINUZ.EFI'

Repeat for all four drives.

Verify:

1

efibootmgr

The firmware should have entries similar to:

1
2
3
4


ZFSBootMenu HDD1
ZFSBootMenu HDD2
ZFSBootMenu HDD3
ZFSBootMenu HDD4

In addition, every ESP contains:

1

EFI/BOOT/BOOTX64.EFI

so the firmware can also use the generic UEFI fallback path.

Synchronizing ESPs with zbm-esp-sync

ZFS protects the contents of the pool.

It does not replicate FAT32 ESPs.

For that purpose I use a small tool I wrote called zbm-esp-sync.

The design is intentionally simple:

1
2
3
4
5


ESP HDD1 = master
       │
       ├── ESP HDD2
       ├── ESP HDD3
       └── ESP HDD4

Only these trees are synchronized:

1
2


EFI/ZBM
EFI/BOOT

The utility verifies that the target really is an EFI System Partition and that the filesystem is VFAT.

It does not modify partition tables or format drives.

Install the build dependencies:

1
2
3
4
5


apt install -y \
    golang-go \
    git \
    make \
    util-linux

Clone and build:

1
2
3
4
5
6
7
8
9


git clone \
    https://gitlab.com/tty8747/zbm-esp-sync \
    /usr/local/src/zbm-esp-sync

cd /usr/local/src/zbm-esp-sync

go test ./...
make build
sudo make install

Example /etc/zbm-esp-sync/config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


master: /dev/disk/by-uuid/AAAA-BBBB

esp:
  - /dev/disk/by-uuid/AAAA-BBBB
  - /dev/disk/by-uuid/CCCC-DDDD
  - /dev/disk/by-uuid/EEEE-FFFF
  - /dev/disk/by-uuid/GGGG-HHHH

paths:
  - EFI/ZBM
  - EFI/BOOT

Test first:

1

sudo zbm-esp-sync refresh --dry-run

Then synchronize:

1
2
3


sudo zbm-esp-sync refresh
sudo zbm-esp-sync verify
sudo zbm-esp-sync status

A healthy setup should look roughly like:

1
2
3
4
5


ESP                          ROLE    MOUNTED  FILES  CHECKSUM  STATE
/dev/disk/by-uuid/...        master  yes      ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current

Enable automatic synchronization:

1
2
3
4
5
6
7


sudo systemctl daemon-reload

sudo systemctl start \
    zbm-esp-sync.service

sudo systemctl enable --now \
    zbm-esp-sync.path

The path unit watches the master ESP for changes to:

1
2


EFI/ZBM
EFI/BOOT

When files change, synchronization is triggered automatically.

What happens when the primary ESP disappears

If HDD1 is physically missing, the command:

1

sudo zbm-esp-sync status

may report something similar to:

1

master ... not a block device

This is expected.

zbm-esp-sync is not involved in booting Linux.

The NAS can continue running on the remaining disks, but I prefer not to update ZFSBootMenu while the master ESP is absent.

The degraded-pool boot problem

The first physical failure test exposed an interesting issue.

With HDD1 removed, ZFSBootMenu itself started successfully from another disk, but the ZFS pool was not automatically imported.

The screen repeatedly showed:

1

Unable to import pool

From the ZFSBootMenu recovery shell, this worked:

1

zpool import -f -N rpool

After that, Ubuntu booted normally.

That tells us that the redundancy of the RAIDZ2 pool itself is fine, but the import policy is blocking unattended degraded boot.

For this standalone NAS I changed the ZFSBootMenu import policy to:

1

zbm.import_policy=force

This causes ZFSBootMenu to use a forced pool import.

This should only be done when you are certain the same pool cannot simultaneously be imported on another machine.

Changing the ZFSBootMenu import policy

For a prebuilt ZFSBootMenu EFI binary, the command line can be edited using zbm-kcl.

Inspect the current value:

1
2


sudo zbm-kcl \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI

Change the im[118;1:3uport policy:

1
2
3
4


sudo zbm-kcl \
    -r zbm.import_policy \
    -a zbm.import_policy=force \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI

Verify:

1
2


sudo zbm-kcl \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI

In my setup the result is:

1

quiet loglevel=0 nomodeset zbm.import_policy=force

Update the generic fallback copy:

1
2
3


sudo cp \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI \
    /boot/efi/EFI/BOOT/BOOTX64.EFI

Then synchronize all ESPs:

1
2


sudo zbm-esp-sync refresh
sudo zbm-esp-sync verify

I intentionally keep:

1

VMLINUZ-BACKUP.EFI

with the normal hostid-based import policy.

That gives me an emergency ZFSBootMenu image that does not force-import pools.

Testing fault tolerance

RAIDZ2 theory is useful, but boot redundancy should be tested physically.

Always shut the NAS down before removing drives:

1

sudo poweroff

One-disk failure

First I removed HDD1.

The system successfully booted from another ESP.

The pool became:

1
2
3
4
5
6


rpool       DEGRADED
  raidz2-0  DEGRADED
    HDD1    UNAVAIL
    HDD2    ONLINE
    HDD3    ONLINE
    HDD4    ONLINE

The filesystems were still mounted normally:

1
2


/      rpool/ROOT/ubuntu
/home  rpool/home

SSH came up without manual intervention.

After reinstalling HDD1, ZFS performed a very small resilver and the pool returned to:

1

ONLINE

Two-disk failure

The more important test was removing HDD1 and HDD2 at the same time.

Only HDD3 and HDD4 remained.

The boot sequence was:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


UEFI
 ↓
ESP on HDD3
 ↓
ZFSBootMenu
 ↓
zbm.import_policy=force
 ↓
rpool DEGRADED
 ↓
Ubuntu
 ↓
SSH

The pool looked like this:

1
2
3
4
5
6
7
8


rpool       DEGRADED
  raidz2-0  DEGRADED
    HDD1    UNAVAIL
    HDD2    UNAVAIL
    HDD3    ONLINE
    HDD4    ONLINE

errors: No known data errors

This confirmed not only that RAIDZ2 can operate with two missing drives, but also that the entire boot path remains functional.

That includes:

1
2
3
4
5
6
7
8


firmware
ESP
ZFSBootMenu
pool import
root-on-ZFS
systemd
network
SSH

ZFS tuning for a NAS

Will zbm-esp-sync wear out the ESPs?

No.

There are two separate questions here.

First, these ESPs live on mechanical HDDs, not NAND flash.

There is no SSD-style TBW endurance concern.

Second, zbm-esp-sync does not rewrite the partitions continuously.

Normal operation looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


ZFSBootMenu unchanged
        ↓
nothing is written

ZFSBootMenu updated
        ↓
master ESP changes
        ↓
systemd.path triggers
        ↓
one synchronization run
        ↓
backup ESPs are updated

The tool compares content and only updates changed files.

It writes temporary files, renames them into place and verifies the resulting data.

It never modifies GPT or formats filesystems.

A more relevant HDD wear issue is excessive power cycling or aggressive spindown.

For mechanical disks, I would pay much more attention to:

1
2
3
4
5


Start_Stop_Count
Load_Cycle_Count
Power_Cycle_Count
Power_On_Hours
Temperature

than to a few dozen megabytes written to an ESP during a bootloader update.

Should I create a RAM disk for ZFS cache?

No.

ZFS already has a native RAM cache called ARC.

On this NAS with 16 GB of RAM, the system reported approximately:

1
2
3


Total RAM:      15.4 GiB
ARC max:        14.4 GiB
ARC current:    ~143 MiB

The ARC was still tiny because the NAS had just been installed and there was almost no workload.

There is no benefit in doing this:

1
2
3
4
5
6
7


RAM
 ↓
tmpfs
 ↓
L2ARC
 ↓
ZFS

ZFS already does:

1
2
3
4
5
6
7
8


RAM
 ├── ARC
 │    └── read cache
 │
 └── dirty data
      └── write buffering
             ↓
            ZFS

L2ARC is meant to be a second-level read cache, usually on SSD or NVMe, when the working set no longer fits into RAM.

It is not a replacement for ARC.

Likewise, a SLOG is not a generic write cache.

For now I leave ARC tuning at its defaults:

1
2


zfs_arc_max=0
zfs_arc_min=0

This allows OpenZFS to manage memory dynamically.

Useful commands:

1

zarcsummary

and:

1

zarcstat 5

Dataset layout for NAS data

I prefer not to store all user data directly inside one generic dataset.

Separate datasets make it possible to apply different snapshot policies, quotas and filesystem properties.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


sudo zfs create \
    -o mountpoint=/srv/data \
    rpool/data

sudo zfs create \
    -o mountpoint=/srv/data/shared \
    -o atime=off \
    rpool/data/shared

sudo zfs create \
    -o mountpoint=/srv/data/documents \
    -o atime=off \
    rpool/data/documents

sudo zfs create \
    -o mountpoint=/srv/data/media \
    -o atime=off \
    rpool/data/media

sudo zfs create \
    -o mountpoint=/srv/data/backups \
    -o atime=off \
    rpool/data/backups

The result is:

1
2
3
4
5
6
7
8
9


rpool
├── ROOT
│   └── ubuntu
├── home
└── data
    ├── shared
    ├── documents
    ├── media
    └── backups

For Samba-oriented datasets I use:

1
2
3


acltype=posixacl
xattr=sa
compression=lz4

Disabling atime is useful for datasets where applications do not care about file access timestamps.

I would not blindly set a large recordsize everywhere.

recordsize should be tuned per workload.

Large sequential media or backup files may benefit from a larger value, but mixed file workloads are often better left at the default.

Data protection

ZFS snapshots

RAIDZ2 protects against physical disk failure.

It does not protect against:

1
2
3
4
5


rm -rf
accidental overwrite
ransomware
broken application logic
operator mistakes

Snapshots help with these problems.

A manual snapshot:

1
2


sudo zfs snapshot \
    rpool/data@manual-$(date +%Y%m%d-%H%M)

List snapshots:

1

zfs list -t snapshot

Snapshots are very cheap to create because they are initially just references to existing blocks.

However, a snapshot is not a backup.

If the entire pool disappears, the snapshots disappear with it.

Automatic snapshots with Sanoid

For automated retention policies, I use Sanoid.

Install it:

1

sudo apt install sanoid

Example /etc/sanoid/sanoid.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


[rpool/data]
    use_template = nas
    recursive = yes

[rpool/ROOT/ubuntu]
    use_template = system

[template_nas]
    frequently = 0
    hourly = 24
    daily = 30
    weekly = 8
    monthly = 12
    yearly = 0
    autosnap = yes
    autoprune = yes

[template_system]
    frequently = 0
    hourly = 0
    daily = 7
    weekly = 4
    monthly = 3
    yearly = 0
    autosnap = yes
    autoprune = yes

This keeps approximately:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


NAS data:
24 hourly
30 daily
8 weekly
12 monthly

Ubuntu root:
7 daily
4 weekly
3 monthly

Enable the timer:

1

sudo systemctl enable --now sanoid.timer

Check it:

1
2


systemctl status sanoid.timer
sudo sanoid --monitor-snapshots

For off-host backup, Sanoid is commonly paired with Syncoid and ZFS replication.

Regular ZFS scrub

RAIDZ2 is only useful if data integrity is actually checked periodically.

A scrub reads allocated data, verifies checksums and reconstructs damaged blocks from redundancy when possible.

Start a manual scrub:

1

sudo zpool scrub rpool

Wait until it finishes:

1

sudo zpool scrub -w rpool

Then check:

1

sudo zpool status -v

A healthy result looks like:

1
2


scan: scrub repaired 0B ... with 0 errors
errors: No known data errors

Ubuntu ships systemd timers for automatic scrubs.

For a home NAS with large spinning disks, I use a monthly scrub:

1
2


sudo systemctl enable --now \
    zfs-scrub-monthly@rpool.timer

Verify:

1

systemctl list-timers 'zfs-scrub*'

A weekly full scrub may be unnecessary for a large mostly-static home NAS, while monthly is a reasonable starting point.

Disk health monitoring

SMART monitoring

ZFS and SMART solve different problems.

ZFS sees things like:

1
2
3


READ errors
WRITE errors
checksum errors

SMART sees the internal state of the physical drive.

Install smartmontools:

1
2


sudo apt install smartmontools
sudo systemctl enable --now smartmontools.service

Inspect a disk using a stable by-id path:

1
2


sudo smartctl -x \
    /dev/disk/by-id/ata-DISK

For WD HDDs, I pay particular attention to:

1
2
3
4


sudo smartctl -A \
    /dev/disk/by-id/ata-DISK \
    | grep -E \
'Power_On_Hours|Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable|UDMA_CRC_Error_Count|Start_Stop_Count|Load_Cycle_Count|Temperature'

A healthy new drive should generally start around:

1
2
3
4


Reallocated_Sector_Ct   0
Current_Pending_Sector  0
Offline_Uncorrectable   0
UDMA_CRC_Error_Count    0

A single SMART value is not always enough to condemn a disk.

What matters is the trend together with ZFS errors and SMART self-test results.

Scheduled SMART self-tests

smartd can run regular self-tests.

A simple /etc/smartd.conf example:

1

DEVICESCAN -a -s (S/../.././02|L/../../7/04:003)

This schedules short tests regularly and long tests on a weekly basis.

For multi-disk NAS systems, it is useful to stagger long tests instead of making every HDD scan its entire surface at exactly the same time.

After editing the configuration:

1

sudo systemctl restart smartmontools

Check the schedule:

1

sudo smartd -q showtests

Check logs:

1

journalctl -u smartmontools

Prometheus monitoring and alerting

Why monitoring matters

RAID without alerts is dangerous.

The failure sequence you want to avoid is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


HDD1 fails
 ↓
RAIDZ2 keeps working
 ↓
nobody notices
 ↓
HDD2 fails weeks later
 ↓
still nobody notices
 ↓
HDD3 fails
 ↓
pool is lost

The first degraded state should result in an alert within minutes.

I use Prometheus and Alertmanager for this.

Prometheus itself should preferably run somewhere other than the NAS.

If the NAS loses power completely, an external monitoring server can still detect that it disappeared.

Installing Prometheus exporters

On the NAS:

1
2
3


sudo apt install \
    prometheus-node-exporter \
    prometheus-smartctl-exporter

Enable them:

1
2
3
4
5


sudo systemctl enable --now \
    prometheus-node-exporter

sudo systemctl enable --now \
    prometheus-smartctl-exporter

Check node exporter:

1

curl -s localhost:9100/metrics | head

Check SMART exporter:

1

curl -s localhost:9633/metrics | head

Monitoring ZFS pool state

Node exporter has a ZFS collector.

A particularly useful metric is:

1

node_zfs_zpool_state

Check it:

1
2


curl -s localhost:9100/metrics \
    | grep node_zfs_zpool_state

You may see something similar to:

1
2


node_zfs_zpool_state{state="online",zpool="rpool"} 1
node_zfs_zpool_state{state="degraded",zpool="rpool"} 0

This makes ZFS pool-state alerting straightforward.

SMART metrics

The SMART exporter exposes metrics such as:

1
2
3
4


smartctl_device
smartctl_device_smart_status
smartctl_device_temperature
smartctl_device_attribute

Inspect the actual output generated by the version installed on your system:

1
2


curl -s localhost:9633/metrics \
    | grep smartctl_device_smart_status

and:

1
2


curl -s localhost:9633/metrics \
    | grep smartctl_device_temperature

I always inspect the real exporter output before writing PromQL rules, because exact labels and ATA attributes can vary between drive models.

Prometheus scrape configuration

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


scrape_configs:
  - job_name: nas-node
    static_configs:
      - targets:
          - nas:9100

  - job_name: nas-smart
    static_configs:
      - targets:
          - nas:9633

Alert when the NAS disappears

1
2
3
4
5
6
7
8
9


- alert: NASDown
  expr: up{job="nas-node"} == 0
  for: 2m

  labels:
    severity: critical

  annotations:
    summary: "NAS is unreachable"

This alert must live outside the NAS itself.

Alert when the ZFS pool is not ONLINE

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


- alert: ZFSPoolNotOnline
  expr: |
    node_zfs_zpool_state{
      zpool="rpool",
      state="online"
    } != 1    

  for: 1m

  labels:
    severity: critical

  annotations:
    summary: "ZFS pool rpool is not ONLINE"

A removed or failed HDD should cause this alert quickly.

SMART health alert

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


- alert: SMARTHealthFailed
  expr: |
    smartctl_device_smart_status == 0    

  for: 2m

  labels:
    severity: critical

  annotations:
    summary: "SMART health check failed"

Alert on bad or pending sectors

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


- alert: SMARTBadSectors
  expr: |
    smartctl_device_attribute{
      attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",
      attribute_value_type="raw"
    } > 0    

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "SMART reports bad or pending sectors"

Depending on your exporter version, inspect the actual metric labels first.

Alert on growing SATA CRC errors

CRC errors are especially useful as a rate rather than an absolute counter.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


- alert: SATACRCErrorIncreasing
  expr: |
    increase(
      smartctl_device_attribute{
        attribute_name="UDMA_CRC_Error_Count",
        attribute_value_type="raw"
      }[1h]
    ) > 0    

  labels:
    severity: warning

  annotations:
    summary: "SATA CRC error counter is increasing"

A growing CRC counter can point to:

1
2
3
4


SATA cabling
backplane problems
connector problems
controller issues

rather than platter failure.

HDD temperature alert

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


- alert: HDDTemperatureHigh
  expr: |
    smartctl_device_temperature{
      temperature_type="current"
    } > 50    

  for: 10m

  labels:
    severity: warning

  annotations:
    summary: "HDD temperature is high"

The 50°C value is an example warning threshold.

Use the specification of your actual HDD model when choosing final limits.

Monitoring the boot ESP replicas

Manual ESP verification

This NAS has another failure domain that a typical ZFS system does not have:

1

4 independent ESPs

So I also want to know whether they are still synchronized.

Manual verification:

1

sudo zbm-esp-sync verify

A failed verification returns a non-zero status.

I would not run a full SHA256 verification every minute, especially if HDD spindown is enabled.

Once per day is sufficient.

Exporting ESP status to Prometheus

Node exporter includes a textfile collector.

Create a small monitoring script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


sudo tee /usr/local/sbin/zbm-esp-monitor >/dev/null <<'EOF'
#!/usr/bin/env bash

set -u

DIR=/var/lib/prometheus/node-exporter
OUT="$DIR/zbm-esp.prom"
TMP=$(mktemp "$DIR/.zbm-esp.prom.XXXXXX")

mkdir -p "$DIR"

if /usr/local/bin/zbm-esp-sync verify >/dev/null 2>&1; then
    ok=1
else
    ok=0
fi

{
    echo '# HELP zbm_esp_sync_ok ZFSBootMenu ESP replicas match the master'
    echo '# TYPE zbm_esp_sync_ok gauge'
    echo "zbm_esp_sync_ok $ok"

    echo '# HELP zbm_esp_sync_last_check_timestamp_seconds Last ESP verification time'
    echo '# TYPE zbm_esp_sync_last_check_timestamp_seconds gauge'
    echo "zbm_esp_sync_last_check_timestamp_seconds $(date +%s)"
} > "$TMP"

chmod 0644 "$TMP"
mv "$TMP" "$OUT"
EOF

sudo chmod 0755 \
    /usr/local/sbin/zbm-esp-monitor

Verify the actual binary location:

1

command -v zbm-esp-sync

and adjust the script if required.

Daily ESP verification with systemd

Create a service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


sudo tee \
  /etc/systemd/system/zbm-esp-monitor.service \
  >/dev/null <<'EOF'
[Unit]
Description=Verify ZFSBootMenu EFI replicas

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/zbm-esp-monitor
EOF

Create a timer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


sudo tee \
  /etc/systemd/system/zbm-esp-monitor.timer \
  >/dev/null <<'EOF'
[Unit]
Description=Periodic ZFSBootMenu ESP verification

[Timer]
OnCalendar=*-*-* 06:15:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable it:

1
2
3
4


sudo systemctl daemon-reload

sudo systemctl enable --now \
    zbm-esp-monitor.timer

Check:

1
2


systemctl list-timers \
    zbm-esp-monitor.timer

Verify the metrics:

1
2


curl -s localhost:9100/metrics \
    | grep '^zbm_esp'

Expected output:

1
2


zbm_esp_sync_ok 1
zbm_esp_sync_last_check_timestamp_seconds ...

Alert when ESPs are out of sync

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


- alert: ZFSBootMenuESPsOutOfSync
  expr: zbm_esp_sync_ok != 1

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "ZFSBootMenu ESP replicas are not synchronized"

Also monitor whether the check itself has stopped running:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


- alert: ZFSBootMenuESPCheckStale
  expr: |
    time() -
    zbm_esp_sync_last_check_timestamp_seconds
    > 90000    

  labels:
    severity: warning

  annotations:
    summary: "ZFSBootMenu ESP verification is stale"

ZFS Event Daemon

I also keep ZED enabled:

1
2


sudo systemctl enable --now \
    zfs-zed.service

Inspect events manually:

1

sudo zpool events -v

ZED gives another independent channel for reacting to ZFS events.

The monitoring stack therefore looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


                 NAS
                  │
        ┌─────────┼─────────┐
        │         │         │
       ZFS       SMART     ESPs
        │         │         │
 node_exporter    │    zbm-esp-monitor
        │         │         │
        └─────────┼─────────┘
                  │
              Prometheus
                  │
             Alertmanager
                  │
           email / Telegram

And independently:

1
2
3


ZFS kernel
    ↓
   ZED

Power management

Suspend and Wake-on-LAN

Ubuntu Server can suspend if the platform supports it.

Check available power states:

1
2


cat /sys/power/state
cat /sys/power/mem_sleep

Suspend manually:

1

sudo systemctl suspend

For network wake-up, use Wake-on-LAN.

Install ethtool:

1

sudo apt install ethtool

Check NIC capabilities:

1
2


sudo ethtool enp1s0 \
    | grep -i Wake-on

For example:

1
2


Supports Wake-on: pumbg
Wake-on: d

If g is present, Magic Packet wake-up is supported.

Enable it:

1

sudo ethtool -s enp1s0 wol g

Netplan can keep the setting persistent:

1
2
3
4
5
6
7
8


network:
  version: 2
  renderer: networkd

  ethernets:
    enp1s0:
      dhcp4: true
      wakeonlan: true

There is an important limitation here.

A sleeping server generally will not wake merely because somebody tries to open an SMB share.

The usual workflow is:

1
2
3
4
5


client sends WoL Magic Packet
        ↓
NAS wakes up
        ↓
Samba/SSH becomes available

For a NAS that needs to remain reachable at all times, normal idle may be preferable to full suspend.

I would also avoid overly aggressive HDD spindown, because SMART, monitoring, ZFS metadata updates and snapshots may repeatedly wake the drives and generate excessive start/stop cycles.

Final architecture

The final boot architecture is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


                    TerraMaster
                         │
                ┌────────┴────────┐
                │                 │
              Boot               Data
                │                 │
         4 independent ESPs      rpool
                │                 │
         ZFSBootMenu            RAIDZ2
                │          ┌──────┼──────┐
         zbm-esp-sync       │      │      │
                │         HDD1   HDD2   HDD3/HDD4
                │
       automatic force import
                │
        rpool/ROOT/ubuntu

Operationally:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


RAIDZ2
  │
  ├── monthly scrub
  │
  ├── SMART monitoring
  │     ├── short tests
  │     └── staggered long tests
  │
  ├── datasets
  │     ├── shared
  │     ├── documents
  │     ├── media
  │     └── backups
  │
  ├── Sanoid snapshots
  │
  ├── node_exporter
  │
  ├── smartctl_exporter
  │
  ├── ESP monitoring
  │
  ├── ZED
  │
  └── Prometheus + Alertmanager

What RAIDZ2 does not solve

RAIDZ2 is not a backup.

It protects against disk failures.

It does not protect against:

1
2
3
4
5
6
7
8


accidental deletion
ransomware
application corruption
administrator mistakes
NAS theft
fire
power supply failure affecting several drives
complete pool loss

Snapshots help with some logical failures, but they still live on the same pool.

A complete design should eventually look like:

1
2
3
4
5
6
7
8
9


              Primary NAS
                  │
        RAIDZ2 + snapshots
                  │
           ZFS replication
                  │
                  ▼
        separate physical NAS
          or backup server

For a ZFS-based system, zfs send / zfs receive or Syncoid are natural choices.

Conclusion

My original goal was to build a NAS that does not rely on a separate system SSD and can survive the loss of any two HDDs.

The result is a system where Ubuntu itself lives inside the RAIDZ2 pool, while every disk contains an independently bootable EFI System Partition.

ZFS provides data redundancy.

ZFSBootMenu provides a clean root-on-ZFS boot path.

Four ESPs eliminate the single boot-device failure point.

zbm-esp-sync keeps the boot partitions synchronized.

zbm.import_policy=force allows this standalone NAS to import a degraded pool without manual recovery.

Monthly scrubs detect latent corruption.

SMART monitors physical drive health.

Sanoid provides snapshot retention.

Prometheus and Alertmanager make sure that the first failed disk does not remain unnoticed.

And there is no need to build a RAM disk for ZFS caching, because ARC already provides the correct in-memory cache layer.

Most importantly, this was not only tested in theory.

The NAS successfully booted in all of these states:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


4 disks present
    → ONLINE

1 disk missing
    → DEGRADED
    → boot successful

2 disks missing
    → DEGRADED
    → boot successful

With two physical HDDs removed, the machine still booted from an ESP on one of the remaining disks, ZFSBootMenu imported the RAIDZ2 pool automatically, Ubuntu mounted rpool/ROOT/ubuntu, networking came up, and SSH became available without manual intervention.

That is exactly the failure mode I wanted: losing two disks should result in a degraded NAS that needs maintenance, not an unbootable server that needs a rescue USB.