Installing Debian on a RAID with LUKS encryption, ZFS root, and booting from USB with Detached Header
Introduction from afar
Not long ago I felt nostalgic inside FreeBSD again, everything is wonderful, everything is familiar, everything is convenient.
There is just one point that completely rules it out from desktop use, at least for me.
On almost all my laptops, FreeBSD does not support either sleep or standby modes (s2disk/s2ram).
And I couldn’t do anything with the hardware/drivers/ACPI, but I tried a lot.
Without standby mode, it is completely impossible to use a laptop, since after transportation, you need to load everything again, turn it on, and restore a complex work session.
And I only reboot workstations after applying updates that require it.
One of the many nice little things that Beastie has that Debian lacks is ZFS Boot Environments, this is somewhat more convenient than, say, LVM snapshots.
And the second is GEOM_ELI, which supports not only, like LUKS, the OR password OR key mode, but also supports password WITH key mode.
I thought and thought, and decided to deploy Debian from scratch, taking into account all the tools I use, my experience, and, importantly, my habits.
Debian is still my main system, so the only way to get a hardware (physical) encryption key in addition to the password is to make it from the header and place it on a bootable USB flash drive.
Habits
And makes us rather bear those ills we have.
Than fly to others that we know not of.
mdadm raid1
This is the base layer of all my disk systems and it is a habit that is difficult to break.
Even if the workstation only has one disk, it will be based on a degraded RAID1.
Although there are always at least two disks in systems.
This allows you to create a bootable and functional copy of your system and data in just a few minutes.
1 2 3 4 5 | mdadm --grow /dev/md1 --raid-devices=3 mdadm --manage /dev/md1 --add /dev/sdx1 # watch -n1 cat /proc/mdstat mdadm --manage /dev/md1 --fail /dev/sdx1 mdadm --manage /dev/md1 --remove /dev/sdx1 |
The disk can be connected via any bus; I have cloned a mirror via eSATA, USB2/3, EXPRESSCARD 54, Thunderbolt, and even on a mounted /dev/loop file.
A system deployed on good equipment and customized to your needs that you can trust is as valuable as your data.
Because writing a small project can take 2-3 months of work, and setting up the system during the process can take up to half a year.
Let me point out right away that no dd, rsync, cpio, or other programs can correctly create a backup from a running system as conveniently as RAID.
LUKS
This is the second level, immediately above the Raid mirror.
Its purpose does not need to be explained.
LVM
Sometimes you need “Hibernate mode” (s2disk), so you need SWAP, and since you want to have one password, then LVM is necessary.
Yes, mdadm, LVM and LUKS can be replaced with ZFS alone, but those are habits.
For a good fifteen years there hasn’t been a single problem with them.
There were power failures/laptop resets, multiple Debian release updates, HDD crashes, but the system always booted.
Looking ahead
As a result, the disk system will look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 | # usb sda 8:16 1 14.9G 0 disk ├─sda1 8:17 1 1000K 0 part ├─sda2 8:18 1 512M 0 part /boot/efi └─sda3 8:19 1 4G 0 part /boot # nvme nvme0n1 259:0 0 3.6T 0 disk └─nvme0n1p1 259:1 0 904G 0 part └─md1 9:1 0 903.9G 0 raid1 └─md1_crypt 253:0 0 903.9G 0 crypt ├─lvm_system-swap 253:1 0 96G 0 lvm └─lvm_system-zfs 253:2 0 807.9G 0 lvm zroot |
Deployment
The Debian installer does not provide the ability to perform a custom installation that meets all requirements.
Therefore, it was decided to install using the good old debootstrap under the LiveCD system.
In my case it is debian-live-13.3.0-amd64-kde.iso.
The LiveCD image was downloaded, all hashes and GPG signatures were verified.
Debian LiveCD
sudo -i
apt install apt-transport-https
nano /etc/apt/sources.list
1 2 3 4 | # trixie
deb https://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb https://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware
|
1 2 3 4 5 6 7 8 9 10 | apt update apt install cryptsetup mdadm \ lvm2 \ linux-headers-amd64 \ debootstrap \ gdisk \ zfsutils-linux \ openssh-server \ linux-headers-6.12.63+deb13-amd64 \ linux-headers-6.12.63+deb13-common |
If you want to continue the installation via ssh:
1 2 3 4 | # nano /etc/ssh/sshd_config printf '%s\n' "PermitRootLogin yes" >> /etc/ssh/sshd_config systemctl restart sshd.service passwd |
Before installation the entire disk was overwritten with urandom, you can skip this step.
1 | dd if=/dev/urandom of=/dev/nvme0n1 bs=1M count=4096 oflag=sync status=progress |
Disk partitioning
1 2 3 4 5 6 7 8 | parted -a optimal /dev/nvme0n1 mklabel gpt unit MiB p free mkpart 'raid' 1 925697 set 1 raid on align-check optimal 1 align-check minimal 1 |
mdadm
1 2 | mdadm --verbose --create /dev/md1 --level=1 --raid-devices=2 missing /dev/nvme0n1p1 mdadm --detail --scan |
luks2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | mkdir /tmp/keys chmod 700 /tmp/keys sync dd if=/dev/random of=/tmp/keys/md1_header bs=1M count=16 sync printf '3' > /proc/sys/vm/drop_caches chmod 400 /tmp/keys/md1_header # Wipe 1Gb and test /dev/md1 dd if=/dev/urandom of=/dev/md1 bs=1M count=1024 oflag=sync status=progress # LUKS2 init cryptsetup --verbose \ --cipher "aes-xts-plain64" \ --key-size=512 \ --hash=sha512 \ --use-random \ --iter-time=3000 \ --type luks2 \ --pbkdf argon2id \ --pbkdf-memory 4194304 \ --pbkdf-parallel 4 \ --verify-passphrase \ luksFormat /dev/md1 \ --header /tmp/keys/md1_header \ --align-payload=8192 # Check cryptsetup luksDump /tmp/keys/md1_header # Open cryptsetup luksOpen /dev/md1 --allow-discards --header /tmp/keys/md1_header md1_crypt |
LVM
1 2 3 4 5 6 7 8 9 10 11 12 | pvcreate /dev/mapper/md1_crypt pvdisplay -v vgcreate lvm_system /dev/mapper/md1_crypt vgdisplay vgscan # vgchange -a y # swap partition lvcreate -L98304 -n swap lvm_system vgdisplay lvm_system # zroot partition lvcreate -l100%FREE -n zfs lvm_system vgdisplay lvm_system |
ZFS
modprobe zfs
If you see an error, it’s likely that the linux-headers is not installed.
1 2 3 4 5 6 7 8 | modprobe: FATAL: Module zfs not found in directory /lib/modules/6.12.63+deb13-amd64 apt install linux-headers-amd64 # or # apt install linux-headers-6.12.63+deb13-amd64 linux-headers-6.12.63+deb13-common apt reinstall zfs-dkms modprobe zfs |
Creating ZFS pool and datasets.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | zpool create -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl \ -O xattr=sa \ -O dnodesize=auto \ -O compression=off \ -O normalization=formD \ -O utf8only=on \ -O checksum=fletcher4 \ -O dedup=off \ -O atime=off \ -O canmount=off \ -O mountpoint=/ \ -R /mnt \ zroot /dev/lvm_system/zfs zfs create -o canmount=off -o mountpoint=none zroot/ROOT zfs create -o canmount=noauto -o mountpoint=/ zroot/ROOT/trixie zfs mount zroot/ROOT/trixie zfs create -o canmount=on -o atime=off zroot/data zfs create -o atime=on zroot/home zfs create -o mountpoint=/root -o atime=on zroot/home/root zfs create -o canmount=off zroot/usr zfs create -o mountpoint=/usr/local zroot/usr/local zfs create -o mountpoint=/usr/src -o exec=off -o setuid=off -o compression=lz4 zroot/usr/src zfs create -o canmount=off zroot/var zfs create -o mountpoint=/var/log -o exec=off -o compression=lz4 zroot/var/log zfs create -o mountpoint=/var/mail -o atime=on -o exec=off zroot/var/mail |
1 2 3 4 | chmod 700 /mnt/root mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock |
Partitioning a bootable USB-drive
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | # Wipe USB dd if=/dev/urandom of=/dev/sda bs=1M count=4096 oflag=sync status=progress sgdisk -Z sgdisk -o /dev/sda # sgdisk --list-types sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/sda sgdisk -n2:1M:+512M -t2:EF00 /dev/sda sgdisk -n3:514M:+4G -t3:8300 /dev/sda sgdisk -c 1:boot /dev/sda sgdisk -A 1:set:2 /dev/sda # /boot - ext4 mkfs.ext4 /dev/sda3 mkdir /mnt/boot mount /dev/sda3 /mnt/boot mkdir /mnt/boot/efi # (/boot/efi)/EFI - fat32 mkdosfs -F 32 -s 1 -n EFI /dev/sda2 mount /dev/sda2 /mnt/boot/efi |
Debotstrap and system pre-configuration
debootstrap trixie /mnt
1 2 3 4 5 6 7 8 | mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ # I didn't have this file # cp: cannot stat '/etc/zfs/zpool.cache': No such file or directory hostname deb hostname > /mnt/etc/hostname printf '127.0.1.1 deb' >> /mnt/etc/hosts |
ip addr show
nano /mnt/etc/network/interfaces.d/eth0
1 2 | auto eth0 iface eth0 inet dhcp |
nano /mnt/etc/apt/sources.list
1 2 3 4 | # trixie
deb http://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware
|
Mounting virtual file systems.
1 2 3 4 5 | # mkdir /mnt/dev /mnt/proc /mnt/sys mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t devpts devpts /mnt/dev/pts |
Copy luks header
1 2 3 4 5 6 7 8 9 10 | mkdir /mnt/etc/.crypto_data cp /tmp/keys/md1_header /mnt/etc/.crypto_data/md1_header chmod 700 /mnt/etc/.crypto_data chmod 400 /mnt/etc/.crypto_data/md1_header # Make a second temporary copy of md1_header in /boot on USB # If you can't boot the first time, # then you don't need to extract this file from the initrd image # to mount md1_crypt manually. cp /tmp/keys/md1_header /mnt/boot/md1_header |
Chrooting into the installed system
chroot /mnt bash --login
Converting sources to https.
apt update
apt install apt-transport-https ca-certificates
nano /etc/apt/sources.list
1 2 3 4 | # trixie
deb https://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware
deb https://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware
|
Install and configure the necessary packages.
1 2 3 4 5 6 7 8 9 | apt update apt install console-setup locales dpkg-reconfigure locales tzdata keyboard-configuration console-setup apt install linux-headers-amd64 linux-image-amd64 apt install cryptsetup mdadm lvm2 debootstrap gdisk zfsutils-linux openssh-server apt install dpkg-dev apt install zfs-initramfs echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf |
Generating mdadm configurations and backups for LVM.
mdadm --detail --scan > /etc/mdadm/mdadm.conf
vgcfgbackup
Adding a hook for initramfs
This script will add the md1_header file into initial ram disk (initrd).
nano /etc/initramfs-tools/hooks/crypto-header.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions copy_exec /etc/.crypto_data/md1_header /md1_header |
chmod 750 /etc/initramfs-tools/hooks/crypto-header.sh
fstab, crypttab, modules
For ZFS, you don’t need to specify anything in fstab.
nano /etc/fstab
1 2 3 | UUID=b5bd933b-51e9-4693-bf3b-d13307bbd885 /boot ext4 discard,noatime,nodiratime 0 2 UUID=4813-4190 /boot/efi vfat defaults 0 0 /dev/lvm_system/swap none swap sw 0 0 |
nano /etc/crypttab
1 2 | # <target name> <source device> <key file> <options> # ,x-initrd.attach md1_crypt /dev/md1 none luks,discard,header=/md1_header,initramfs |
nano /etc/initramfs-tools/modules
1 2 3 4 5 6 | md_mod
# raid1
zfs
spl
nls_cp437
nls_ascii
|
Adjusting ZFS mount points
1 2 3 4 5 6 7 8 9 10 11 | mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/zroot zed -F & # Verify cat /etc/zfs/zfs-list.cache/zroot # Switch to foreground fg # Press Ctrl+C & remove '/mnt' sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* # Make snapshot zfs snapshot zroot/ROOT/trixie@install |
Setting up GRUB and initrd
apt install cryptsetup cryptsetup-initramfs
apt install systemd-timesyncd
apt install grub-pc
apt install dosfstools
If you see this error, it is normal for ZFS root.
1 2 | cryptsetup: ERROR: Couldn't resolve device zroot/ROOT/trixie cryptsetup: WARNING: Couldn't determine root device |
update-initramfs -u -k all
nano /etc/default/grub
1 2 3 4 5 | GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`( . /etc/os-release && echo ${NAME} )` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="root=ZFS=zroot/ROOT/trixie net.ifnames=0 biosdevname=0" |
update-grub
legacy BIOS MBR
grub-install /dev/sda
UEFI
1 2 3 4 5 6 7 8 | apt install grub-efi-amd64 shim-signed update-grub grub-install \ --target=x86_64-efi \ --efi-directory=/boot/efi \ --bootloader-id=debian \ --recheck \ --no-floppy |
Setting up users
1 2 3 4 5 6 7 8 | # Root password passwd root # Regular user groupadd --gid 1000 youruser adduser --home /home/youruser --ingroup youruser --uid 1000 youruser chmod -R ugo-xX,u=rwX,go-rwXx /home/youruser usermod -aG cdrom,floppy,audio,dip,video,plugdev,netdev,scanner,bluetooth,lpadmin youruser |
Exit & reboot
Ctrl+D or exit
You have exited the chroot installation environment and returned to the LiveCD system. Unmount all ZFS file systems.
1 2 3 4 | mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} zfs umount -a umount /mnt zpool export -a |
Type reboot and you may be able to boot into your new system from a bootable USB flash drive.
A little more detail about GRUB UEFI
Bootable USB flash drive.
blkid /dev/sda*
1 2 3 4 | /dev/sdb: PTUUID="70f483d1-859e-44de-9a93-ea46acdc886b" PTTYPE="gpt"/dev/sdb1: PARTLABEL="boot" PARTUUID="da022c89-87be-421a-bc48-cfd30136666e" /dev/sdb1: PARTLABEL="boot" PARTUUID="110dda04-a5e1-4c56-832c-9573c82c59ee" /dev/sdb2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="4813-4190" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="a88f91e5-2f12-411f-8d32-2ea944c231b6" /dev/sdb3: UUID="b5bd933b-51e9-4693-bf3b-d13307bbd885" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="0bae36b2-e05a-42d9-9567-589b9499d114" |
Contents of the vfat file system for /EFI.
tree /boot/efi
1 2 3 4 5 6 7 8 9 10 11 12 13 | /boot/efi
└── EFI
├── BOOT
│ ├── BOOTX64.EFI
│ ├── grubx64.efi
│ └── mmx64.efi
└── debian
├── BOOTX64.CSV
├── fbx64.efi
├── grub.cfg
├── grubx64.efi
├── mmx64.efi
└── shimx64.efi
|
cat /boot/efi/EFI/debian/grub.cfg
1 2 3 | search.fs_uuid b5bd933b-51e9-4693-bf3b-d13307bbd885 root hd1,gpt3 set prefix=($root)'/grub' configfile $prefix/grub.cfg |
fs_uuid corresponds /dev/sdb3.
If the new system boots successfully
You can remove md1_header from /boot, because it is already contained inside all /boot/initrd.img*.
rm -f /boot/md1_header
And make some copies of your bootable flash drive.
Decide for yourself whether to use dd or copy files, remember about grub-install.
Without this flash drive, or more precisely without md1_header, your data will become a brick.
If you were unable to boot
Boot back into LiveCD.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | nano /etc/apt/sources.list # trixie deb http://deb.debian.org/debian/ trixie main contrib non-free non-free-firmware deb http://security.debian.org/debian-security/ trixie-security main contrib non-free non-free-firmware deb http://deb.debian.org/debian/ trixie-updates main contrib non-free non-free-firmware apt update apt install cryptsetup mdadm lvm2 linux-headers-generic debootstrap gdisk openssh-server linux-headers-6.12.63+deb13-amd64 linux-headers-6.12.63+deb13-common apt install linux-headers-6.12.63+deb13-amd64 linux-headers-6.12.63+deb13-common apt install zfsutils-linux zfs-dkms # nano /etc/ssh/sshd_config # printf '%s\n' "PermitRootLogin yes" >> /etc/ssh/sshd_config # systemctl restart sshd.service mdadm --detail --scan mdadm --stop /dev/md127 mdadm --assemble /dev/md1 /dev/nvme0n1p1 mkdir /tmp/usb mount /dev/sda3 /tmp/usb cp /tmp/usb/md1_header /tmp/md1_header umount /dev/sda3 cryptsetup luksOpen /dev/md1 --allow-discards --header /tmp/md1_header md1_crypt pvscan vgchange -a y modprobe zfs zpool import -f zroot # zpool import -d /dev/mapper/md1_crypt zroot zfs set mountpoint=legacy zroot/ROOT/debian mount -t zfs zroot/ROOT/debian /mnt # or # zfs mount zroot/ROOT/debian mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys mount -t devpts devpts /mnt/dev/pts mount /dev/sda3 /mnt/boot mount /dev/sda2 /mnt/boot/efi chroot /mnt bash --login |
And make the necessary corrections.
Debuging initrd.img
To verify that /md1_header was copied into the initrd by the hook script, you can use unmkinitramfs.
1 2 3 4 5 6 | mkdir /tmp/debug cp /boot/initrd.img-6.12.63+deb13-amd64 /tmp/debug/ cd /tmp/debug/ unmkinitramfs -v initrd.img-6.12.63+deb13-amd64 . # header exists? ls -la ./md1_header |
I hope I haven’t forgotten anything. Based on Debian Trixie Root on ZFS.
Original post on SecOps.it Blog • Installing Debian on a RAID with LUKS encryption, ZFS root, and booting from USB with Detached Header