Installing Debian on a RAID with LUKS encryption, ZFS root, and booting from USB with Detached Header

LUKS encryption with Detached Header file on USB drive 5504x3096 luks-encrypted-brick.jpg
LUKS encryption with Detached Header file on USB drive

Introduction from afar

Not long ago I felt nostalgic inside FreeBSD again, everything is wonderful, everything is familiar, everything is convenient.
There is just one point that completely rules it out from desktop use, at least for me.
On almost all my laptops, FreeBSD does not support either sleep or standby modes (s2disk/s2ram).
And I couldn’t do anything with the hardware/drivers/ACPI, but I tried a lot.

Without standby mode, it is completely impossible to use a laptop, since after transportation, you need to load everything again, turn it on, and restore a complex work session.
And I only reboot workstations after applying updates that require it.

One of the many nice little things that Beastie has that Debian lacks is ZFS Boot Environments, this is somewhat more convenient than, say, LVM snapshots.
And the second is GEOM_ELI, which supports not only, like LUKS, the OR password OR key mode, but also supports password WITH key mode.

I thought and thought, and decided to deploy Debian from scratch, taking into account all the tools I use, my experience, and, importantly, my habits.

Debian is still my main system, so the only way to get a hardware (physical) encryption key in addition to the password is to make it from the header and place it on a bootable USB flash drive.

Habits


And makes us rather bear those ills we have.
Than fly to others that we know not of.

mdadm raid1

This is the base layer of all my disk systems and it is a habit that is difficult to break.
Even if the workstation only has one disk, it will be based on a degraded RAID1.
Although there are always at least two disks in systems.
This allows you to create a bootable and functional copy of your system and data in just a few minutes.

1
2
3
4
5
mdadm --grow   /dev/md1 --raid-devices=3
mdadm --manage /dev/md1 --add /dev/sdx1
# watch -n1 cat /proc/mdstat
mdadm --manage /dev/md1 --fail /dev/sdx1
mdadm --manage /dev/md1 --remove /dev/sdx1

The disk can be connected via any bus; I have cloned a mirror via eSATA, USB2/3, EXPRESSCARD 54, Thunderbolt, and even on a mounted /dev/loop file.

A system deployed on good equipment and customized to your needs that you can trust is as valuable as your data.
Because writing a small project can take 2-3 months of work, and setting up the system during the process can take up to half a year.

Let me point out right away that no dd, rsync, cpio, or other programs can correctly create a backup from a running system as conveniently as RAID.

LUKS

This is the second level, immediately above the Raid mirror.
Its purpose does not need to be explained.

LVM

Sometimes you need “Hibernate mode” (s2disk), so you need SWAP, and since you want to have one password, then LVM is necessary.

Yes, mdadm, LVM and LUKS can be replaced with ZFS alone, but those are habits.
For a good fifteen years there hasn’t been a single problem with them.
There were power failures/laptop resets, multiple Debian release updates, HDD crashes, but the system always booted.

Looking ahead

As a result, the disk system will look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
# usb
sda                       8:16   1  14.9G  0 disk
├─sda1                    8:17   1  1000K  0 part
├─sda2                    8:18   1   512M  0 part   /boot/efi
└─sda3                    8:19   1     4G  0 part   /boot

# nvme
nvme0n1                 259:0    0   3.6T  0 disk
└─nvme0n1p1             259:1    0   904G  0 part
  └─md1                   9:1    0 903.9G  0 raid1
    └─md1_crypt         253:0    0 903.9G  0 crypt
      ├─lvm_system-swap 253:1    0    96G  0 lvm
      └─lvm_system-zfs  253:2    0 807.9G  0 lvm    zroot

Deployment


The Debian installer does not provide the ability to perform a custom installation that meets all requirements.
Therefore, it was decided to install using the good old debootstrap under the LiveCD system.
In my case it is debian-live-13.3.0-amd64-kde.iso.
The LiveCD image was downloaded, all hashes and GPG signatures were verified.

Debian LiveCD

sudo -i
apt install apt-transport-https
nano /etc/apt/sources.list

1
2
3
4
# trixie
deb https://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb https://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware
1
2
3
4
5
6
7
8
9
10
apt update
apt install cryptsetup mdadm \
            lvm2 \
            linux-headers-amd64 \
            debootstrap \
            gdisk \
            zfsutils-linux \
            openssh-server \
            linux-headers-6.12.63+deb13-amd64 \
            linux-headers-6.12.63+deb13-common

If you want to continue the installation via ssh:

1
2
3
4
# nano /etc/ssh/sshd_config
printf '%s\n' "PermitRootLogin yes" >> /etc/ssh/sshd_config
systemctl restart sshd.service
passwd

Before installation the entire disk was overwritten with urandom, you can skip this step.

1
dd if=/dev/urandom of=/dev/nvme0n1 bs=1M count=4096 oflag=sync status=progress

Disk partitioning

1
2
3
4
5
6
7
8
parted -a optimal /dev/nvme0n1
mklabel gpt
unit MiB
p free
mkpart 'raid' 1 925697
set 1 raid on
align-check optimal 1
align-check minimal 1

mdadm

1
2
mdadm --verbose --create /dev/md1 --level=1 --raid-devices=2 missing /dev/nvme0n1p1
mdadm --detail --scan

luks2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
mkdir /tmp/keys
chmod 700 /tmp/keys
sync
dd if=/dev/random of=/tmp/keys/md1_header bs=1M count=16
sync
printf '3' > /proc/sys/vm/drop_caches
chmod 400 /tmp/keys/md1_header
# Wipe 1Gb and test /dev/md1
dd if=/dev/urandom of=/dev/md1 bs=1M count=1024 oflag=sync status=progress
# LUKS2 init
cryptsetup  --verbose \
            --cipher "aes-xts-plain64" \
            --key-size=512 \
            --hash=sha512 \
            --use-random \
            --iter-time=3000 \
            --type luks2 \
            --pbkdf argon2id \
            --pbkdf-memory 4194304 \
            --pbkdf-parallel 4 \
            --verify-passphrase \
            luksFormat /dev/md1 \
            --header /tmp/keys/md1_header \
            --align-payload=8192
# Check
cryptsetup luksDump /tmp/keys/md1_header
# Open
cryptsetup luksOpen /dev/md1 --allow-discards --header /tmp/keys/md1_header md1_crypt

LVM

1
2
3
4
5
6
7
8
9
10
11
12
pvcreate /dev/mapper/md1_crypt
pvdisplay -v
vgcreate lvm_system /dev/mapper/md1_crypt
vgdisplay
vgscan
# vgchange -a y
# swap partition
lvcreate -L98304    -n swap lvm_system
vgdisplay lvm_system
# zroot partition
lvcreate -l100%FREE -n zfs  lvm_system
vgdisplay lvm_system

ZFS

modprobe zfs

If you see an error, it’s likely that the linux-headers is not installed.

1
2
3
4
5
6
7
8
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.12.63+deb13-amd64

apt install linux-headers-amd64
# or
# apt install linux-headers-6.12.63+deb13-amd64 linux-headers-6.12.63+deb13-common
apt reinstall zfs-dkms

modprobe zfs

Creating ZFS pool and datasets.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
zpool create  -o ashift=12 \
              -o autotrim=on \
              -O acltype=posixacl \
              -O xattr=sa \
              -O dnodesize=auto \
              -O compression=off \
              -O normalization=formD \
              -O utf8only=on \
              -O checksum=fletcher4 \
              -O dedup=off \
              -O atime=off \
              -O canmount=off \
              -O mountpoint=/ \
              -R /mnt \
              zroot /dev/lvm_system/zfs

zfs create  -o canmount=off     -o mountpoint=none  zroot/ROOT
zfs create  -o canmount=noauto  -o mountpoint=/     zroot/ROOT/trixie
zfs mount zroot/ROOT/trixie

zfs create  -o canmount=on            -o atime=off                                          zroot/data
zfs create                            -o atime=on                                           zroot/home
zfs create  -o mountpoint=/root       -o atime=on                                           zroot/home/root
zfs create  -o canmount=off                                                                 zroot/usr
zfs create  -o mountpoint=/usr/local                                                        zroot/usr/local
zfs create  -o mountpoint=/usr/src    -o exec=off     -o setuid=off     -o compression=lz4  zroot/usr/src
zfs create  -o canmount=off                                                                 zroot/var
zfs create  -o mountpoint=/var/log    -o exec=off                       -o compression=lz4  zroot/var/log
zfs create  -o mountpoint=/var/mail   -o atime=on     -o exec=off                           zroot/var/mail
1
2
3
4
chmod 700 /mnt/root
mkdir /mnt/run
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock

Partitioning a bootable USB-drive

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Wipe USB
dd if=/dev/urandom of=/dev/sda bs=1M count=4096 oflag=sync status=progress

sgdisk -Z
sgdisk -o /dev/sda
# sgdisk --list-types
sgdisk  -a1 -n1:24K:+1000K  -t1:EF02    /dev/sda
sgdisk      -n2:1M:+512M    -t2:EF00    /dev/sda
sgdisk      -n3:514M:+4G    -t3:8300    /dev/sda
sgdisk      -c 1:boot                   /dev/sda
sgdisk  -A  1:set:2                     /dev/sda

# /boot - ext4
mkfs.ext4 /dev/sda3
mkdir     /mnt/boot
mount     /dev/sda3 /mnt/boot
mkdir     /mnt/boot/efi
# (/boot/efi)/EFI - fat32
mkdosfs -F 32 -s 1 -n EFI /dev/sda2
mount     /dev/sda2 /mnt/boot/efi

Debotstrap and system pre-configuration

debootstrap trixie /mnt

1
2
3
4
5
6
7
8
mkdir /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/
# I didn't have this file
# cp: cannot stat '/etc/zfs/zpool.cache': No such file or directory

hostname deb
hostname > /mnt/etc/hostname
printf '127.0.1.1       deb' >> /mnt/etc/hosts

ip addr show
nano /mnt/etc/network/interfaces.d/eth0

1
2
auto eth0
iface eth0 inet dhcp

nano /mnt/etc/apt/sources.list

1
2
3
4
# trixie
deb http://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware

Mounting virtual file systems.

1
2
3
4
5
# mkdir /mnt/dev /mnt/proc /mnt/sys
mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
mount -t devpts devpts /mnt/dev/pts

Copy luks header

1
2
3
4
5
6
7
8
9
10
mkdir /mnt/etc/.crypto_data
cp /tmp/keys/md1_header /mnt/etc/.crypto_data/md1_header
chmod 700 /mnt/etc/.crypto_data
chmod 400 /mnt/etc/.crypto_data/md1_header

# Make a second temporary copy of md1_header in /boot on USB
# If you can't boot the first time,
# then you don't need to extract this file from the initrd image
# to mount md1_crypt manually.
cp /tmp/keys/md1_header /mnt/boot/md1_header

Chrooting into the installed system

chroot /mnt bash --login

Converting sources to https.

apt update
apt install apt-transport-https ca-certificates
nano /etc/apt/sources.list

1
2
3
4
# trixie
deb https://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb https://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware

Install and configure the necessary packages.

1
2
3
4
5
6
7
8
9
apt update
apt install console-setup locales
dpkg-reconfigure locales tzdata keyboard-configuration console-setup

apt install linux-headers-amd64 linux-image-amd64
apt install cryptsetup mdadm lvm2 debootstrap gdisk zfsutils-linux openssh-server
apt install dpkg-dev
apt install zfs-initramfs
echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf

Generating mdadm configurations and backups for LVM.

mdadm --detail --scan > /etc/mdadm/mdadm.conf
vgcfgbackup

Adding a hook for initramfs

This script will add the md1_header file into initial ram disk (initrd).

nano /etc/initramfs-tools/hooks/crypto-header.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/sh

PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
# get pre-requisites
prereqs)
prereqs
exit 0
;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /etc/.crypto_data/md1_header /md1_header

chmod 750 /etc/initramfs-tools/hooks/crypto-header.sh

fstab, crypttab, modules

For ZFS, you don’t need to specify anything in fstab.
nano /etc/fstab

1
2
3
UUID=b5bd933b-51e9-4693-bf3b-d13307bbd885     /boot           ext4    discard,noatime,nodiratime                      0 2
UUID=4813-4190                                /boot/efi       vfat    defaults                                        0 0
/dev/lvm_system/swap                          none            swap    sw                                              0 0

nano /etc/crypttab

1
2
# <target name>   <source device>   <key file>   <options>     # ,x-initrd.attach
md1_crypt /dev/md1 none luks,discard,header=/md1_header,initramfs

nano /etc/initramfs-tools/modules

1
2
3
4
5
6
md_mod
# raid1
zfs
spl
nls_cp437
nls_ascii

Adjusting ZFS mount points

1
2
3
4
5
6
7
8
9
10
11
mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/zroot
zed -F &
# Verify
cat /etc/zfs/zfs-list.cache/zroot
# Switch to foreground
fg
# Press Ctrl+C & remove '/mnt'
sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*
# Make snapshot
zfs snapshot zroot/ROOT/trixie@install

Setting up GRUB and initrd

apt install cryptsetup cryptsetup-initramfs
apt install systemd-timesyncd
apt install grub-pc
apt install dosfstools

If you see this error, it is normal for ZFS root.

1
2
cryptsetup: ERROR: Couldn't resolve device zroot/ROOT/trixie
cryptsetup: WARNING: Couldn't determine root device

update-initramfs -u -k all

nano /etc/default/grub

1
2
3
4
5
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`( . /etc/os-release && echo ${NAME} )`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="root=ZFS=zroot/ROOT/trixie net.ifnames=0 biosdevname=0"

update-grub

legacy BIOS MBR

grub-install /dev/sda

UEFI

1
2
3
4
5
6
7
8
apt install grub-efi-amd64 shim-signed
update-grub
grub-install \
            --target=x86_64-efi \
            --efi-directory=/boot/efi \
            --bootloader-id=debian \
            --recheck \
            --no-floppy

Setting up users

1
2
3
4
5
6
7
8
# Root password
passwd root

# Regular user
groupadd --gid 1000 youruser
adduser --home /home/youruser --ingroup youruser --uid 1000 youruser
chmod -R ugo-xX,u=rwX,go-rwXx /home/youruser
usermod -aG cdrom,floppy,audio,dip,video,plugdev,netdev,scanner,bluetooth,lpadmin youruser

Exit & reboot

Ctrl+D or exit

You have exited the chroot installation environment and returned to the LiveCD system. Unmount all ZFS file systems.

1
2
3
4
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zfs umount -a
umount /mnt
zpool export -a

Type reboot and you may be able to boot into your new system from a bootable USB flash drive.

A little more detail about GRUB UEFI

Bootable USB flash drive.

blkid /dev/sda*

1
2
3
4
/dev/sdb:  PTUUID="70f483d1-859e-44de-9a93-ea46acdc886b" PTTYPE="gpt"/dev/sdb1: PARTLABEL="boot" PARTUUID="da022c89-87be-421a-bc48-cfd30136666e"
/dev/sdb1: PARTLABEL="boot" PARTUUID="110dda04-a5e1-4c56-832c-9573c82c59ee"
/dev/sdb2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="4813-4190" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="a88f91e5-2f12-411f-8d32-2ea944c231b6"
/dev/sdb3: UUID="b5bd933b-51e9-4693-bf3b-d13307bbd885" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="0bae36b2-e05a-42d9-9567-589b9499d114"

Contents of the vfat file system for /EFI.

tree /boot/efi

1
2
3
4
5
6
7
8
9
10
11
12
13
/boot/efi
└── EFI
    ├── BOOT
    │   ├── BOOTX64.EFI
    │   ├── grubx64.efi
    │   └── mmx64.efi
    └── debian
        ├── BOOTX64.CSV
        ├── fbx64.efi
        ├── grub.cfg
        ├── grubx64.efi
        ├── mmx64.efi
        └── shimx64.efi

cat /boot/efi/EFI/debian/grub.cfg

1
2
3
search.fs_uuid b5bd933b-51e9-4693-bf3b-d13307bbd885 root hd1,gpt3 
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

fs_uuid corresponds /dev/sdb3.

If the new system boots successfully


You can remove md1_header from /boot, because it is already contained inside all /boot/initrd.img*.
rm -f /boot/md1_header

And make some copies of your bootable flash drive.
Decide for yourself whether to use dd or copy files, remember about grub-install.
Without this flash drive, or more precisely without md1_header, your data will become a brick.

If you were unable to boot


Boot back into LiveCD.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
nano /etc/apt/sources.list
# trixie
deb http://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware
apt update
apt install cryptsetup mdadm lvm2 linux-headers-generic debootstrap gdisk openssh-server linux-headers-6.12.63+deb13-amd64 linux-headers-6.12.63+deb13-common
apt install linux-headers-6.12.63+deb13-amd64 linux-headers-6.12.63+deb13-common
apt install zfsutils-linux zfs-dkms

# nano /etc/ssh/sshd_config
# printf '%s\n' "PermitRootLogin yes" >> /etc/ssh/sshd_config
# systemctl restart sshd.service

mdadm --detail --scan
mdadm --stop /dev/md127
mdadm --assemble /dev/md1 /dev/nvme0n1p1

mkdir /tmp/usb
mount /dev/sda3 /tmp/usb
cp /tmp/usb/md1_header /tmp/md1_header
umount /dev/sda3

cryptsetup luksOpen /dev/md1 --allow-discards --header /tmp/md1_header md1_crypt

pvscan
vgchange -a y
modprobe zfs

zpool import -f zroot
# zpool import -d /dev/mapper/md1_crypt zroot

zfs set mountpoint=legacy zroot/ROOT/debian
mount -t zfs zroot/ROOT/debian /mnt
# or
# zfs mount zroot/ROOT/debian

mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
mount -t devpts devpts /mnt/dev/pts

mount /dev/sda3 /mnt/boot
mount /dev/sda2 /mnt/boot/efi

chroot /mnt bash --login

And make the necessary corrections.

Debuging initrd.img


To verify that /md1_header was copied into the initrd by the hook script, you can use unmkinitramfs.

1
2
3
4
5
6
mkdir /tmp/debug
cp /boot/initrd.img-6.12.63+deb13-amd64 /tmp/debug/
cd /tmp/debug/
unmkinitramfs -v initrd.img-6.12.63+deb13-amd64 .
# header exists?
ls -la ./md1_header

I hope I haven’t forgotten anything. Based on Debian Trixie Root on ZFS.

Original post on SecOps.it Blog Installing Debian on a RAID with LUKS encryption, ZFS root, and booting from USB with Detached Header