# ZFS RAIDZ2 with ZFSBootMenu: A Fault-Tolerant 4-Disk NAS In my previous article, I described how to install Ubuntu on a ZFS mirror and boot it using ZFSBootMenu. This time, the goal is more ambitious: build a proper four-disk NAS where both the operating system and user data live on a single ZFS RAIDZ2 pool, while still being able to boot the machine after losing one or even two physical disks. I also wanted to avoid a dedicated boot SSD. The final design looks like this: ```text HDD1 ├── 1 GiB ESP └── ZFS ─┐ HDD2 │ ├── ESP │ └── ZFS ─┤ │ HDD3 ├── rpool / RAIDZ2 ├── ESP │ └── ZFS ─┤ │ HDD4 │ ├── ESP │ └── ZFS ─┘ ``` Each disk has its own EFI System Partition, containing a copy of ZFSBootMenu. The ZFS pool itself contains Ubuntu and NAS data. The system was also tested by physically removing disks, including a two-disk failure scenario. ## Hardware and software My setup is: | Component | Configuration | |---|---| | NAS | TerraMaster F4-425 | | HDDs | 4 × WD Red WD60EFRX, 6 TB | | RAM | 16 GB | | OS | Ubuntu Server 26.04 LTS | | Kernel | Linux 7.0.x | | OpenZFS | 2.4.x | | Pool layout | RAIDZ2 | | Bootloader | ZFSBootMenu | | ESPs | One 1 GiB ESP per HDD | The internal TerraMaster USB DOM is left untouched. The important design decision is that every HDD is independently bootable. A disk therefore looks like this: ```text Disk ├── GPT partition 1: EFI System Partition └── GPT partition 2: ZFS ``` And each ESP contains: ```text EFI/ZBM/VMLINUZ.EFI EFI/BOOT/BOOTX64.EFI ``` The second file is the standard UEFI fallback path. ## Installing Ubuntu on RAIDZ2 ### Preparing the disks First, identify the drives. Do not rely on `/dev/sda`, `/dev/sdb`, and similar names when creating the ZFS pool. Use stable `/dev/disk/by-id` paths instead. For example: ```bash ls -l /dev/disk/by-id/ lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE ``` Define the disks: ```bash D1=/dev/disk/by-id/ata-DISK1 D2=/dev/disk/by-id/ata-DISK2 D3=/dev/disk/by-id/ata-DISK3 D4=/dev/disk/by-id/ata-DISK4 DISKS=( "$D1" "$D2" "$D3" "$D4" ) ``` The following commands destroy the existing partition tables, so double-check the selected disks before continuing. Create GPT tables and two partitions per disk: ```bash for d in "${DISKS[@]}"; do wipefs -a "$d" sgdisk --zap-all "$d" sgdisk \ -n 1:1M:+1G \ -t 1:EF00 \ -c 1:EFI \ "$d" sgdisk \ -n 2:0:0 \ -t 2:6A85CF4D-1DD2-11B2-99A6-080020736631 \ -c 2:ZFS \ "$d" done partprobe udevadm settle ``` Verify: ```bash lsblk -o NAME,SIZE,PARTTYPE,FSTYPE ``` ### Creating the four EFI System Partitions Format partition 1 on every disk: ```bash mkfs.vfat -F 32 -n ZBM1 "${D1}-part1" mkfs.vfat -F 32 -n ZBM2 "${D2}-part1" mkfs.vfat -F 32 -n ZBM3 "${D3}-part1" mkfs.vfat -F 32 -n ZBM4 "${D4}-part1" ``` Then get their UUIDs: ```bash lsblk -o PATH,FSTYPE,LABEL,UUID,PARTTYPE ``` These UUIDs will later be used by `/etc/fstab` and `zbm-esp-sync`. ### Creating the RAIDZ2 pool From the Ubuntu live environment, install the required tools: ```bash apt update apt install -y \ debootstrap \ gdisk \ dosfstools \ zfsutils-linux ``` Create the pool: ```bash zpool create -f \ -o ashift=12 \ -o compatibility=openzfs-2.3-linux \ -O compression=lz4 \ -O acltype=posixacl \ -O xattr=sa \ -O relatime=on \ -O mountpoint=none \ -R /mnt \ rpool \ raidz2 \ "${D1}-part2" \ "${D2}-part2" \ "${D3}-part2" \ "${D4}-part2" ``` I use: ```text ashift=12 ``` for 4 KiB sectors. Compression is enabled with: ```text compression=lz4 ``` For Linux systems using POSIX ACLs, I use: ```text acltype=posixacl xattr=sa ``` Verify the pool: ```bash zpool status zpool get ashift,compatibility rpool ``` ### Creating the root datasets Create a container for boot environments: ```bash zfs create \ -o canmount=off \ -o mountpoint=none \ rpool/ROOT ``` Create the Ubuntu root filesystem: ```bash zfs create \ -o canmount=noauto \ -o mountpoint=/ \ rpool/ROOT/ubuntu ``` Mount it: ```bash zfs mount rpool/ROOT/ubuntu ``` Because the pool was imported with: ```text -R /mnt ``` the root filesystem appears under `/mnt`. Create a separate `/home` dataset: ```bash zfs create \ -o mountpoint=/home \ rpool/home ``` Set the boot filesystem: ```bash zpool set bootfs=rpool/ROOT/ubuntu rpool ``` Set the kernel command line inherited by boot environments: ```bash zfs set \ org.zfsbootmenu:commandline="quiet" \ rpool/ROOT ``` There is an important distinction here. This property: ```text org.zfsbootmenu:commandline ``` controls the Linux kernel launched by ZFSBootMenu. Parameters such as: ```text zbm.import_policy=force ``` control ZFSBootMenu itself and are stored in the command line embedded inside the ZFSBootMenu EFI executable. ### Installing Ubuntu 26.04 with debootstrap Bootstrap the system: ```bash debootstrap resolute /mnt ``` Bind the required virtual filesystems: ```bash mount --rbind /dev /mnt/dev mount --make-rslave /mnt/dev mount --rbind /proc /mnt/proc mount --make-rslave /mnt/proc mount --rbind /sys /mnt/sys mount --make-rslave /mnt/sys mount --rbind /run /mnt/run mount --make-rslave /mnt/run ``` Copy DNS configuration: ```bash cp -L /etc/resolv.conf /mnt/etc/resolv.conf ``` Enter the new system: ```bash chroot /mnt /bin/bash ``` Install the required packages: ```bash apt update apt install -y \ linux-generic \ ubuntu-server-minimal \ openssh-server \ sudo \ zfsutils-linux \ zfs-initramfs \ dosfstools \ efibootmgr \ curl \ locales \ netplan.io ``` ### Configuring the host ID A stable `/etc/hostid` is important for root-on-ZFS systems. Generate it: ```bash zgenhostid hostid ``` Verify: ```bash ls -l /etc/hostid ``` ### Hostname, user and SSH Set the hostname: ```bash echo nas > /etc/hostname ``` Create an administrative user: ```bash useradd \ --create-home \ --shell /bin/bash \ --groups sudo \ goto passwd goto ``` Enable SSH: ```bash systemctl enable ssh ``` ### Network configuration Example `/etc/netplan/01-netcfg.yaml`: ```yaml network: version: 2 renderer: networkd ethernets: enp1s0: dhcp4: true ``` Set appropriate permissions: ```bash chmod 600 /etc/netplan/01-netcfg.yaml ``` ### Creating zpool.cache There is a small but important detail here. The pool was created with: ```text -R /mnt ``` which sets an alternate root and effectively leaves the pool without a persistent cache file. Create one explicitly: ```bash mkdir -p /etc/zfs zpool set \ cachefile=/etc/zfs/zpool.cache \ rpool ``` Verify: ```bash ls -lh /etc/zfs/zpool.cache ``` Rebuild the initramfs: ```bash update-initramfs -u -k all ``` Check that both the host ID and pool cache were included: ```bash lsinitramfs /boot/initrd.img-* \ | grep -E 'hostid|zpool.cache' ``` You should see: ```text etc/hostid etc/zfs/zpool.cache ``` ## Installing ZFSBootMenu ### Installing the EFI binary I use the official prebuilt ZFSBootMenu EFI binary. Mount the primary ESP at: ```text /boot/efi ``` For example: ```bash mkdir -p /boot/efi mount /dev/disk/by-uuid/AAAA-BBBB /boot/efi ``` Create the directories: ```bash mkdir -p \ /boot/efi/EFI/ZBM \ /boot/efi/EFI/BOOT ``` Place the ZFSBootMenu EFI executable at: ```text /boot/efi/EFI/ZBM/VMLINUZ.EFI ``` Create the standard UEFI fallback copy: ```bash cp \ /boot/efi/EFI/ZBM/VMLINUZ.EFI \ /boot/efi/EFI/BOOT/BOOTX64.EFI ``` I also keep: ```text EFI/ZBM/VMLINUZ-BACKUP.EFI ``` as an additional recovery copy. ### Mounting only the primary ESP Only the primary ESP is mounted permanently. Example `/etc/fstab`: ```fstab UUID=AAAA-BBBB /boot/efi vfat defaults,nofail,umask=0077,x-systemd.device-timeout=5s 0 0 ``` The key option is: ```text nofail ``` If the disk containing the primary ESP disappears, the operating system must still be able to boot from another ESP. ### Creating UEFI boot entries Create a UEFI entry for every disk. For example: ```bash efibootmgr \ --create \ --disk /dev/sda \ --part 1 \ --label 'ZFSBootMenu HDD1' \ --loader '\EFI\ZBM\VMLINUZ.EFI' ``` Repeat for all four drives. Verify: ```bash efibootmgr ``` The firmware should have entries similar to: ```text ZFSBootMenu HDD1 ZFSBootMenu HDD2 ZFSBootMenu HDD3 ZFSBootMenu HDD4 ``` In addition, every ESP contains: ```text EFI/BOOT/BOOTX64.EFI ``` so the firmware can also use the generic UEFI fallback path. ### Synchronizing ESPs with zbm-esp-sync ZFS protects the contents of the pool. It does not replicate FAT32 ESPs. For that purpose I use a small tool I wrote called `zbm-esp-sync`. The design is intentionally simple: ```text ESP HDD1 = master │ ├── ESP HDD2 ├── ESP HDD3 └── ESP HDD4 ``` Only these trees are synchronized: ```text EFI/ZBM EFI/BOOT ``` The utility verifies that the target really is an EFI System Partition and that the filesystem is VFAT. It does not modify partition tables or format drives. Install the build dependencies: ```bash apt install -y \ golang-go \ git \ make \ util-linux ``` Clone and build: ```bash git clone \ https://gitlab.com/tty8747/zbm-esp-sync \ /usr/local/src/zbm-esp-sync cd /usr/local/src/zbm-esp-sync go test ./... make build sudo make install ``` Example `/etc/zbm-esp-sync/config.yaml`: ```yaml master: /dev/disk/by-uuid/AAAA-BBBB esp: - /dev/disk/by-uuid/AAAA-BBBB - /dev/disk/by-uuid/CCCC-DDDD - /dev/disk/by-uuid/EEEE-FFFF - /dev/disk/by-uuid/GGGG-HHHH paths: - EFI/ZBM - EFI/BOOT ``` Test first: ```bash sudo zbm-esp-sync refresh --dry-run ``` Then synchronize: ```bash sudo zbm-esp-sync refresh sudo zbm-esp-sync verify sudo zbm-esp-sync status ``` A healthy setup should look roughly like: ```text ESP ROLE MOUNTED FILES CHECKSUM STATE /dev/disk/by-uuid/... master yes ok ok current /dev/disk/by-uuid/... backup no ok ok current /dev/disk/by-uuid/... backup no ok ok current /dev/disk/by-uuid/... backup no ok ok current ``` Enable automatic synchronization: ```bash sudo systemctl daemon-reload sudo systemctl start \ zbm-esp-sync.service sudo systemctl enable --now \ zbm-esp-sync.path ``` The path unit watches the master ESP for changes to: ```text EFI/ZBM EFI/BOOT ``` When files change, synchronization is triggered automatically. ### What happens when the primary ESP disappears If HDD1 is physically missing, the command: ```bash sudo zbm-esp-sync status ``` may report something similar to: ```text master ... not a block device ``` This is expected. `zbm-esp-sync` is not involved in booting Linux. The NAS can continue running on the remaining disks, but I prefer not to update ZFSBootMenu while the master ESP is absent. ### The degraded-pool boot problem The first physical failure test exposed an interesting issue. With HDD1 removed, ZFSBootMenu itself started successfully from another disk, but the ZFS pool was not automatically imported. The screen repeatedly showed: ```text Unable to import pool ``` From the ZFSBootMenu recovery shell, this worked: ```bash zpool import -f -N rpool ``` After that, Ubuntu booted normally. That tells us that the redundancy of the RAIDZ2 pool itself is fine, but the import policy is blocking unattended degraded boot. For this standalone NAS I changed the ZFSBootMenu import policy to: ```text zbm.import_policy=force ``` This causes ZFSBootMenu to use a forced pool import. This should only be done when you are certain the same pool cannot simultaneously be imported on another machine. ### Changing the ZFSBootMenu import policy For a prebuilt ZFSBootMenu EFI binary, the command line can be edited using `zbm-kcl`. Inspect the current value: ```bash sudo zbm-kcl \ /boot/efi/EFI/ZBM/VMLINUZ.EFI ``` Change the im[118;1:3uport policy: ```bash sudo zbm-kcl \ -r zbm.import_policy \ -a zbm.import_policy=force \ /boot/efi/EFI/ZBM/VMLINUZ.EFI ``` Verify: ```bash sudo zbm-kcl \ /boot/efi/EFI/ZBM/VMLINUZ.EFI ``` In my setup the result is: ```text quiet loglevel=0 nomodeset zbm.import_policy=force ``` Update the generic fallback copy: ```bash sudo cp \ /boot/efi/EFI/ZBM/VMLINUZ.EFI \ /boot/efi/EFI/BOOT/BOOTX64.EFI ``` Then synchronize all ESPs: ```bash sudo zbm-esp-sync refresh sudo zbm-esp-sync verify ``` I intentionally keep: ```text VMLINUZ-BACKUP.EFI ``` with the normal hostid-based import policy. That gives me an emergency ZFSBootMenu image that does not force-import pools. ## Testing fault tolerance RAIDZ2 theory is useful, but boot redundancy should be tested physically. Always shut the NAS down before removing drives: ```bash sudo poweroff ``` ### One-disk failure First I removed HDD1. The system successfully booted from another ESP. The pool became: ```text rpool DEGRADED raidz2-0 DEGRADED HDD1 UNAVAIL HDD2 ONLINE HDD3 ONLINE HDD4 ONLINE ``` The filesystems were still mounted normally: ```text / rpool/ROOT/ubuntu /home rpool/home ``` SSH came up without manual intervention. After reinstalling HDD1, ZFS performed a very small resilver and the pool returned to: ```text ONLINE ``` ### Two-disk failure The more important test was removing HDD1 and HDD2 at the same time. Only HDD3 and HDD4 remained. The boot sequence was: ```text UEFI ↓ ESP on HDD3 ↓ ZFSBootMenu ↓ zbm.import_policy=force ↓ rpool DEGRADED ↓ Ubuntu ↓ SSH ``` The pool looked like this: ```text rpool DEGRADED raidz2-0 DEGRADED HDD1 UNAVAIL HDD2 UNAVAIL HDD3 ONLINE HDD4 ONLINE errors: No known data errors ``` This confirmed not only that RAIDZ2 can operate with two missing drives, but also that the entire boot path remains functional. That includes: ```text firmware ESP ZFSBootMenu pool import root-on-ZFS systemd network SSH ``` ## ZFS tuning for a NAS ### Will zbm-esp-sync wear out the ESPs? No. There are two separate questions here. First, these ESPs live on mechanical HDDs, not NAND flash. There is no SSD-style TBW endurance concern. Second, `zbm-esp-sync` does not rewrite the partitions continuously. Normal operation looks like: ```text ZFSBootMenu unchanged ↓ nothing is written ZFSBootMenu updated ↓ master ESP changes ↓ systemd.path triggers ↓ one synchronization run ↓ backup ESPs are updated ``` The tool compares content and only updates changed files. It writes temporary files, renames them into place and verifies the resulting data. It never modifies GPT or formats filesystems. A more relevant HDD wear issue is excessive power cycling or aggressive spindown. For mechanical disks, I would pay much more attention to: ```text Start_Stop_Count Load_Cycle_Count Power_Cycle_Count Power_On_Hours Temperature ``` than to a few dozen megabytes written to an ESP during a bootloader update. ### Should I create a RAM disk for ZFS cache? No. ZFS already has a native RAM cache called ARC. On this NAS with 16 GB of RAM, the system reported approximately: ```text Total RAM: 15.4 GiB ARC max: 14.4 GiB ARC current: ~143 MiB ``` The ARC was still tiny because the NAS had just been installed and there was almost no workload. There is no benefit in doing this: ```text RAM ↓ tmpfs ↓ L2ARC ↓ ZFS ``` ZFS already does: ```text RAM ├── ARC │ └── read cache │ └── dirty data └── write buffering ↓ ZFS ``` L2ARC is meant to be a second-level read cache, usually on SSD or NVMe, when the working set no longer fits into RAM. It is not a replacement for ARC. Likewise, a SLOG is not a generic write cache. For now I leave ARC tuning at its defaults: ```text zfs_arc_max=0 zfs_arc_min=0 ``` This allows OpenZFS to manage memory dynamically. Useful commands: ```bash zarcsummary ``` and: ```bash zarcstat 5 ``` ### Dataset layout for NAS data I prefer not to store all user data directly inside one generic dataset. Separate datasets make it possible to apply different snapshot policies, quotas and filesystem properties. For example: ```bash sudo zfs create \ -o mountpoint=/srv/data \ rpool/data sudo zfs create \ -o mountpoint=/srv/data/shared \ -o atime=off \ rpool/data/shared sudo zfs create \ -o mountpoint=/srv/data/documents \ -o atime=off \ rpool/data/documents sudo zfs create \ -o mountpoint=/srv/data/media \ -o atime=off \ rpool/data/media sudo zfs create \ -o mountpoint=/srv/data/backups \ -o atime=off \ rpool/data/backups ``` The result is: ```text rpool ├── ROOT │ └── ubuntu ├── home └── data ├── shared ├── documents ├── media └── backups ``` For Samba-oriented datasets I use: ```text acltype=posixacl xattr=sa compression=lz4 ``` Disabling `atime` is useful for datasets where applications do not care about file access timestamps. I would not blindly set a large `recordsize` everywhere. `recordsize` should be tuned per workload. Large sequential media or backup files may benefit from a larger value, but mixed file workloads are often better left at the default. ## Data protection ### ZFS snapshots RAIDZ2 protects against physical disk failure. It does not protect against: ```text rm -rf accidental overwrite ransomware broken application logic operator mistakes ``` Snapshots help with these problems. A manual snapshot: ```bash sudo zfs snapshot \ rpool/data@manual-$(date +%Y%m%d-%H%M) ``` List snapshots: ```bash zfs list -t snapshot ``` Snapshots are very cheap to create because they are initially just references to existing blocks. However, a snapshot is not a backup. If the entire pool disappears, the snapshots disappear with it. ### Automatic snapshots with Sanoid For automated retention policies, I use Sanoid. Install it: ```bash sudo apt install sanoid ``` Example `/etc/sanoid/sanoid.conf`: ```ini [rpool/data] use_template = nas recursive = yes [rpool/ROOT/ubuntu] use_template = system [template_nas] frequently = 0 hourly = 24 daily = 30 weekly = 8 monthly = 12 yearly = 0 autosnap = yes autoprune = yes [template_system] frequently = 0 hourly = 0 daily = 7 weekly = 4 monthly = 3 yearly = 0 autosnap = yes autoprune = yes ``` This keeps approximately: ```text NAS data: 24 hourly 30 daily 8 weekly 12 monthly Ubuntu root: 7 daily 4 weekly 3 monthly ``` Enable the timer: ```bash sudo systemctl enable --now sanoid.timer ``` Check it: ```bash systemctl status sanoid.timer sudo sanoid --monitor-snapshots ``` For off-host backup, Sanoid is commonly paired with Syncoid and ZFS replication. ### Regular ZFS scrub RAIDZ2 is only useful if data integrity is actually checked periodically. A scrub reads allocated data, verifies checksums and reconstructs damaged blocks from redundancy when possible. Start a manual scrub: ```bash sudo zpool scrub rpool ``` Wait until it finishes: ```bash sudo zpool scrub -w rpool ``` Then check: ```bash sudo zpool status -v ``` A healthy result looks like: ```text scan: scrub repaired 0B ... with 0 errors errors: No known data errors ``` Ubuntu ships systemd timers for automatic scrubs. For a home NAS with large spinning disks, I use a monthly scrub: ```bash sudo systemctl enable --now \ zfs-scrub-monthly@rpool.timer ``` Verify: ```bash systemctl list-timers 'zfs-scrub*' ``` A weekly full scrub may be unnecessary for a large mostly-static home NAS, while monthly is a reasonable starting point. ## Disk health monitoring ### SMART monitoring ZFS and SMART solve different problems. ZFS sees things like: ```text READ errors WRITE errors checksum errors ``` SMART sees the internal state of the physical drive. Install `smartmontools`: ```bash sudo apt install smartmontools sudo systemctl enable --now smartmontools.service ``` Inspect a disk using a stable by-id path: ```bash sudo smartctl -x \ /dev/disk/by-id/ata-DISK ``` For WD HDDs, I pay particular attention to: ```bash sudo smartctl -A \ /dev/disk/by-id/ata-DISK \ | grep -E \ 'Power_On_Hours|Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable|UDMA_CRC_Error_Count|Start_Stop_Count|Load_Cycle_Count|Temperature' ``` A healthy new drive should generally start around: ```text Reallocated_Sector_Ct 0 Current_Pending_Sector 0 Offline_Uncorrectable 0 UDMA_CRC_Error_Count 0 ``` A single SMART value is not always enough to condemn a disk. What matters is the trend together with ZFS errors and SMART self-test results. ### Scheduled SMART self-tests `smartd` can run regular self-tests. A simple `/etc/smartd.conf` example: ```text DEVICESCAN -a -s (S/../.././02|L/../../7/04:003) ``` This schedules short tests regularly and long tests on a weekly basis. For multi-disk NAS systems, it is useful to stagger long tests instead of making every HDD scan its entire surface at exactly the same time. After editing the configuration: ```bash sudo systemctl restart smartmontools ``` Check the schedule: ```bash sudo smartd -q showtests ``` Check logs: ```bash journalctl -u smartmontools ``` ## Prometheus monitoring and alerting ### Why monitoring matters RAID without alerts is dangerous. The failure sequence you want to avoid is: ```text HDD1 fails ↓ RAIDZ2 keeps working ↓ nobody notices ↓ HDD2 fails weeks later ↓ still nobody notices ↓ HDD3 fails ↓ pool is lost ``` The first degraded state should result in an alert within minutes. I use Prometheus and Alertmanager for this. Prometheus itself should preferably run somewhere other than the NAS. If the NAS loses power completely, an external monitoring server can still detect that it disappeared. ### Installing Prometheus exporters On the NAS: ```bash sudo apt install \ prometheus-node-exporter \ prometheus-smartctl-exporter ``` Enable them: ```bash sudo systemctl enable --now \ prometheus-node-exporter sudo systemctl enable --now \ prometheus-smartctl-exporter ``` Check node exporter: ```bash curl -s localhost:9100/metrics | head ``` Check SMART exporter: ```bash curl -s localhost:9633/metrics | head ``` ### Monitoring ZFS pool state Node exporter has a ZFS collector. A particularly useful metric is: ```text node_zfs_zpool_state ``` Check it: ```bash curl -s localhost:9100/metrics \ | grep node_zfs_zpool_state ``` You may see something similar to: ```text node_zfs_zpool_state{state="online",zpool="rpool"} 1 node_zfs_zpool_state{state="degraded",zpool="rpool"} 0 ``` This makes ZFS pool-state alerting straightforward. ### SMART metrics The SMART exporter exposes metrics such as: ```text smartctl_device smartctl_device_smart_status smartctl_device_temperature smartctl_device_attribute ``` Inspect the actual output generated by the version installed on your system: ```bash curl -s localhost:9633/metrics \ | grep smartctl_device_smart_status ``` and: ```bash curl -s localhost:9633/metrics \ | grep smartctl_device_temperature ``` I always inspect the real exporter output before writing PromQL rules, because exact labels and ATA attributes can vary between drive models. ### Prometheus scrape configuration Example: ```yaml scrape_configs: - job_name: nas-node static_configs: - targets: - nas:9100 - job_name: nas-smart static_configs: - targets: - nas:9633 ``` ### Alert when the NAS disappears ```yaml - alert: NASDown expr: up{job="nas-node"} == 0 for: 2m labels: severity: critical annotations: summary: "NAS is unreachable" ``` This alert must live outside the NAS itself. ### Alert when the ZFS pool is not ONLINE ```yaml - alert: ZFSPoolNotOnline expr: | node_zfs_zpool_state{ zpool="rpool", state="online" } != 1 for: 1m labels: severity: critical annotations: summary: "ZFS pool rpool is not ONLINE" ``` A removed or failed HDD should cause this alert quickly. ### SMART health alert ```yaml - alert: SMARTHealthFailed expr: | smartctl_device_smart_status == 0 for: 2m labels: severity: critical annotations: summary: "SMART health check failed" ``` ### Alert on bad or pending sectors Example: ```yaml - alert: SMARTBadSectors expr: | smartctl_device_attribute{ attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable", attribute_value_type="raw" } > 0 for: 5m labels: severity: warning annotations: summary: "SMART reports bad or pending sectors" ``` Depending on your exporter version, inspect the actual metric labels first. ### Alert on growing SATA CRC errors CRC errors are especially useful as a rate rather than an absolute counter. Example: ```yaml - alert: SATACRCErrorIncreasing expr: | increase( smartctl_device_attribute{ attribute_name="UDMA_CRC_Error_Count", attribute_value_type="raw" }[1h] ) > 0 labels: severity: warning annotations: summary: "SATA CRC error counter is increasing" ``` A growing CRC counter can point to: ```text SATA cabling backplane problems connector problems controller issues ``` rather than platter failure. ### HDD temperature alert Example: ```yaml - alert: HDDTemperatureHigh expr: | smartctl_device_temperature{ temperature_type="current" } > 50 for: 10m labels: severity: warning annotations: summary: "HDD temperature is high" ``` The `50°C` value is an example warning threshold. Use the specification of your actual HDD model when choosing final limits. ## Monitoring the boot ESP replicas ### Manual ESP verification This NAS has another failure domain that a typical ZFS system does not have: ```text 4 independent ESPs ``` So I also want to know whether they are still synchronized. Manual verification: ```bash sudo zbm-esp-sync verify ``` A failed verification returns a non-zero status. I would not run a full SHA256 verification every minute, especially if HDD spindown is enabled. Once per day is sufficient. ### Exporting ESP status to Prometheus Node exporter includes a textfile collector. Create a small monitoring script: ```bash sudo tee /usr/local/sbin/zbm-esp-monitor >/dev/null <<'EOF' #!/usr/bin/env bash set -u DIR=/var/lib/prometheus/node-exporter OUT="$DIR/zbm-esp.prom" TMP=$(mktemp "$DIR/.zbm-esp.prom.XXXXXX") mkdir -p "$DIR" if /usr/local/bin/zbm-esp-sync verify >/dev/null 2>&1; then ok=1 else ok=0 fi { echo '# HELP zbm_esp_sync_ok ZFSBootMenu ESP replicas match the master' echo '# TYPE zbm_esp_sync_ok gauge' echo "zbm_esp_sync_ok $ok" echo '# HELP zbm_esp_sync_last_check_timestamp_seconds Last ESP verification time' echo '# TYPE zbm_esp_sync_last_check_timestamp_seconds gauge' echo "zbm_esp_sync_last_check_timestamp_seconds $(date +%s)" } > "$TMP" chmod 0644 "$TMP" mv "$TMP" "$OUT" EOF sudo chmod 0755 \ /usr/local/sbin/zbm-esp-monitor ``` Verify the actual binary location: ```bash command -v zbm-esp-sync ``` and adjust the script if required. ### Daily ESP verification with systemd Create a service: ```bash sudo tee \ /etc/systemd/system/zbm-esp-monitor.service \ >/dev/null <<'EOF' [Unit] Description=Verify ZFSBootMenu EFI replicas [Service] Type=oneshot ExecStart=/usr/local/sbin/zbm-esp-monitor EOF ``` Create a timer: ```bash sudo tee \ /etc/systemd/system/zbm-esp-monitor.timer \ >/dev/null <<'EOF' [Unit] Description=Periodic ZFSBootMenu ESP verification [Timer] OnCalendar=*-*-* 06:15:00 Persistent=true [Install] WantedBy=timers.target EOF ``` Enable it: ```bash sudo systemctl daemon-reload sudo systemctl enable --now \ zbm-esp-monitor.timer ``` Check: ```bash systemctl list-timers \ zbm-esp-monitor.timer ``` Verify the metrics: ```bash curl -s localhost:9100/metrics \ | grep '^zbm_esp' ``` Expected output: ```text zbm_esp_sync_ok 1 zbm_esp_sync_last_check_timestamp_seconds ... ``` ### Alert when ESPs are out of sync ```yaml - alert: ZFSBootMenuESPsOutOfSync expr: zbm_esp_sync_ok != 1 for: 5m labels: severity: warning annotations: summary: "ZFSBootMenu ESP replicas are not synchronized" ``` Also monitor whether the check itself has stopped running: ```yaml - alert: ZFSBootMenuESPCheckStale expr: | time() - zbm_esp_sync_last_check_timestamp_seconds > 90000 labels: severity: warning annotations: summary: "ZFSBootMenu ESP verification is stale" ``` ## ZFS Event Daemon I also keep ZED enabled: ```bash sudo systemctl enable --now \ zfs-zed.service ``` Inspect events manually: ```bash sudo zpool events -v ``` ZED gives another independent channel for reacting to ZFS events. The monitoring stack therefore looks like this: ```text NAS │ ┌─────────┼─────────┐ │ │ │ ZFS SMART ESPs │ │ │ node_exporter │ zbm-esp-monitor │ │ │ └─────────┼─────────┘ │ Prometheus │ Alertmanager │ email / Telegram ``` And independently: ```text ZFS kernel ↓ ZED ``` ## Power management ### Suspend and Wake-on-LAN Ubuntu Server can suspend if the platform supports it. Check available power states: ```bash cat /sys/power/state cat /sys/power/mem_sleep ``` Suspend manually: ```bash sudo systemctl suspend ``` For network wake-up, use Wake-on-LAN. Install `ethtool`: ```bash sudo apt install ethtool ``` Check NIC capabilities: ```bash sudo ethtool enp1s0 \ | grep -i Wake-on ``` For example: ```text Supports Wake-on: pumbg Wake-on: d ``` If `g` is present, Magic Packet wake-up is supported. Enable it: ```bash sudo ethtool -s enp1s0 wol g ``` Netplan can keep the setting persistent: ```yaml network: version: 2 renderer: networkd ethernets: enp1s0: dhcp4: true wakeonlan: true ``` There is an important limitation here. A sleeping server generally will not wake merely because somebody tries to open an SMB share. The usual workflow is: ```text client sends WoL Magic Packet ↓ NAS wakes up ↓ Samba/SSH becomes available ``` For a NAS that needs to remain reachable at all times, normal idle may be preferable to full suspend. I would also avoid overly aggressive HDD spindown, because SMART, monitoring, ZFS metadata updates and snapshots may repeatedly wake the drives and generate excessive start/stop cycles. ## Final architecture The final boot architecture is: ```text TerraMaster │ ┌────────┴────────┐ │ │ Boot Data │ │ 4 independent ESPs rpool │ │ ZFSBootMenu RAIDZ2 │ ┌──────┼──────┐ zbm-esp-sync │ │ │ │ HDD1 HDD2 HDD3/HDD4 │ automatic force import │ rpool/ROOT/ubuntu ``` Operationally: ```text RAIDZ2 │ ├── monthly scrub │ ├── SMART monitoring │ ├── short tests │ └── staggered long tests │ ├── datasets │ ├── shared │ ├── documents │ ├── media │ └── backups │ ├── Sanoid snapshots │ ├── node_exporter │ ├── smartctl_exporter │ ├── ESP monitoring │ ├── ZED │ └── Prometheus + Alertmanager ``` ## What RAIDZ2 does not solve RAIDZ2 is not a backup. It protects against disk failures. It does not protect against: ```text accidental deletion ransomware application corruption administrator mistakes NAS theft fire power supply failure affecting several drives complete pool loss ``` Snapshots help with some logical failures, but they still live on the same pool. A complete design should eventually look like: ```text Primary NAS │ RAIDZ2 + snapshots │ ZFS replication │ ▼ separate physical NAS or backup server ``` For a ZFS-based system, `zfs send` / `zfs receive` or Syncoid are natural choices. ## Conclusion My original goal was to build a NAS that does not rely on a separate system SSD and can survive the loss of any two HDDs. The result is a system where Ubuntu itself lives inside the RAIDZ2 pool, while every disk contains an independently bootable EFI System Partition. ZFS provides data redundancy. ZFSBootMenu provides a clean root-on-ZFS boot path. Four ESPs eliminate the single boot-device failure point. `zbm-esp-sync` keeps the boot partitions synchronized. `zbm.import_policy=force` allows this standalone NAS to import a degraded pool without manual recovery. Monthly scrubs detect latent corruption. SMART monitors physical drive health. Sanoid provides snapshot retention. Prometheus and Alertmanager make sure that the first failed disk does not remain unnoticed. And there is no need to build a RAM disk for ZFS caching, because ARC already provides the correct in-memory cache layer. Most importantly, this was not only tested in theory. The NAS successfully booted in all of these states: ```text 4 disks present → ONLINE 1 disk missing → DEGRADED → boot successful 2 disks missing → DEGRADED → boot successful ``` With two physical HDDs removed, the machine still booted from an ESP on one of the remaining disks, ZFSBootMenu imported the RAIDZ2 pool automatically, Ubuntu mounted `rpool/ROOT/ubuntu`, networking came up, and SSH became available without manual intervention. That is exactly the failure mode I wanted: losing two disks should result in a degraded NAS that needs maintenance, not an unbootable server that needs a rescue USB.