Dan's notes

NFS Readonly Root Filesystem

Dan Drown -

I remember when Knoppix came out. It was innovative by making it possible to run a full Linux system off a read-only CD. It was useful in many different ways: a way to repair your system if it wasn't working normally, a secure OS to use if you didn't trust someone else's computer, and a way to try out Linux without having to deal with installer headaches.

I have a Windows desktop that I would occasionally like to run Linux on it. Normally, I use a lot of WSL but sometimes I need Linux with access to the hardware, rather than a VM. My choices for doing this would be to install to a local drive, use a live CD/USB, or netboot.

Since USB2.0 flash drives are very slow and I have a Linux server connected via 2.5G ethernet, I wanted to try out netbooting.

When netbooting, one problem to solve is transferring secrets. Things like the ssh host keys, the shadow password file, and the krb5 host keytab. There's a few ways to handle that, and I chose to use a USB flash drive. That way the secrets are not transferred over the network in clear text, as they would be with NFS or TFTP.

The OS itself will be exported read-only.  This means that multiple desktops can share it at the same time, they just each need their own USB flash drive.

I chose Fedora as my OS as I like their release schedule for Desktops.

Setting up the USB flash drive: partition

To setup the USB flash drive, I wrote a new GPT partition table. I created a 1GB EFI partition and a 14GB Linux partition.

[root@nfs ~]# fdisk /dev/disk/by-id/usb-SanDisk*9994*-0:0 
Welcome to fdisk (util-linux 2.36.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command. 
Command (m for help): p 
Disk /dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0: 14.91 GiB, 16008609792 bytes, 31266816 sectors 
Disk model: Ultra 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 512 bytes 
I/O size (minimum/optimal): 512 bytes / 512 bytes 
Disklabel type: gpt 
Disk identifier: 0926F9AC-DAC7-1144-B8FA-38F7894CCE49 
Device                                                           Start      End  Sectors  Size Type 
/dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0-part1  2048 31266782 31264735 14.9G Microsoft basic data
Command (m for help): g
Created a new GPT disklabel (GUID: 3586EEDF-45A2-7146-8AA7-4B32C369E357).
Command (m for help): n 
First sector (2048-31266782, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-31266782, default 31266782): +1G 
Created a new partition 1 of type 'Linux filesystem' and of size 1 GiB.
Partition #1 contains a ntfs signature. Do you want to remove the signature? [Y]es/[N]o: y 
The signature will be removed by a write command. 
Command (m for help): n 
Partition number (2-128, default 2): 
First sector (2099200-31266782, default 2099200): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2099200-31266782, default 31266782):
Created a new partition 2 of type 'Linux filesystem' and of size 13.9 GiB.
Command (m for help): t 
Partition number (1,2, default 2): 1 
Partition type or alias (type L to list all): 1 
Changed type of partition 'Linux filesystem' to 'EFI System'. 
Command (m for help): p 
Disk /dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0: 14.91 GiB, 16008609792 bytes, 31266816 sectors 
Disk model: Ultra 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 512 bytes 
I/O size (minimum/optimal): 512 bytes / 512 bytes 
Disklabel type: gpt 
Disk identifier: F8BDB994-FEBE-6745-BA7B-C945FF53BF93 
Device                                                             Start      End  Sectors  Size Type 
/dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0-part1    2048  2099199  2097152    1G EFI System 
/dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0-part2 2099200 31266782 29167583 13.9G Linux filesystem 
Filesystem/RAID signature on partition 1 will be wiped. 
Command (m for help): w 
The partition table has been altered. 
Calling ioctl() to re-read partition table. 
Syncing disks.

Setting up the USB flash drive: filesystems

I created a vfat filesystem on partition 1, and an ext2 filesystem on partition 2.  I then gave the ext2 filesystem the label "stateless-state" so the readonly-root package can find it. I gave the vfat filesystem the label "BOOT". I'll put data on them both in a later section.

[root@nfs ~]# mkfs.vfat /dev/disk/by-id/usb-SanDisk*9994*-part1
mkfs.fat 4.2 (2021-01-31)
[root@nfs ~]# mkfs.ext2 /dev/disk/by-id/usb-SanDisk*9994*-part2 
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 3645947 4k blocks and 912128 inodes
Filesystem UUID: 9387e5cc-5538-40af-84b2-8c6b494122d7 
Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208 
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
[root@nfs ~]# dosfslabel /dev/disk/by-id/usb-SanDisk*9994*-part1 BOOT
[root@nfs ~]# e2label /dev/disk/by-id/usb-SanDisk*9994*-part2 stateless-state

OS Image install

I started with these instructions and updated them for Fedora 35: https://fedoramagazine.org/how-to-build-a-netboot-server-part-1/

On my NFS server, I installed Fedora with the Xfce GUI Desktop to /srv/fc35. This uses about 5GB of space.

[root@nfs ~]# mkdir /srv/fc35 
[root@nfs ~]# dnf -y --releasever=35 --installroot=/srv/fc35 install fedora-release systemd passwd rootfiles sudo dracut dracut-network nfs-utils vim-minimal dnf readonly-root krb5-workstation
... # lots of output
Transaction Summary
========================================================
Install  243 Packages
... # lots more output
[root@nfs ~]# dnf -y --releasever=35 --installroot=/srv/fc35 group install "Xfce Desktop"
... # lots of output 
[root@nfs ~]# dnf -y --installroot=/srv/fc35 install firefox 
... # lots of output

Kernel custom boot config

Next, I setup the dracut/initramfs configuration and let dnf install the kernel‌.  The dracut settings will setup an initramfs that has the modules needed for a net boot.

[root@nfs ~]# echo 'hostonly=no' > /srv/fc35/etc/dracut.conf.d/hostonly.conf
[root@nfs ~]# echo 'add_dracutmodules+=" network nfs "' > /srv/fc35/etc/dracut.conf.d/netboot.conf
[root@nfs ~]# dnf --installroot=/srv/fc35 install kernel

Base OS Configuration

I configured the locale, configured the console messages, added my user, told journald to log to memory, and setup the DNS resolver IP:

[root@nfs ~]# echo 'LANG="en_US.UTF-8"' > /srv/fc35/etc/locale.conf 
[root@nfs ~]# echo 'kernel.printk = 0 4 1 7' > /srv/fc35/etc/sysctl.d/00-printk.conf
[root@nfs ~]# chroot /srv/fc35 useradd myuser
[root@nfs ~]# echo 'myuser ALL=(ALL) NOPASSWD: ALL' > /srv/fc35/etc/sudoers.d/myuser 
[root@nfs ~]# sed -i 's/^#Storage=auto$/Storage=volatile/' /srv/fc35/etc/systemd/journald.conf 
[root@nfs ~]# echo nameserver 10.1.1.1 >/srv/fc35/etc/resolv.conf

readonly-root configuration

The readonly-root package lets you redirect some writes to a different location.  This looks for the filesystem labeled "stateless-state" and bind-mounts the persistent state files (listed in /etc/statetab.d) there.  The temporary state files (listed in /etc/rwtab.d) are kept in a ramdisk.

[root@nfs ~]# cat <<EOF >/srv/fc35/etc/sysconfig/readonly-root 
# Set to 'yes' to mount the system filesystems read-only.
# NOTE: It's necessary to append 'ro' to mount options of '/' mount point in 
#       /etc/fstab as well, otherwise the READONLY option will not work. READONLY=yes 
# Set to 'yes' to mount various temporary state as either tmpfs 
# or on the block device labelled RW_LABEL. Implied by READONLY 
TEMPORARY_STATE=yes 
# Place to put a tmpfs for temporary scratch writable space 
RW_MOUNT=/var/lib/stateless/writable 
# Label on local filesystem which can be used for temporary scratch space 
#RW_LABEL=stateless-rw 
# Options to use for temporary mount
RW_OPTIONS= 
# Label for partition with persistent data 
STATE_LABEL=stateless-state
# Where to mount to the persistent data
STATE_MOUNT=/var/lib/stateless/state
# Options to use for persistent mount
STATE_OPTIONS= 
# NFS server to use for persistent data? 
CLIENTSTATE= 
# Use slave bind-mounts 
SLAVE_MOUNTS=yes
EOF 
[root@nfs ~]# echo dirs /var/lib/gssproxy >/srv/fc35/etc/rwtab.d/gssproxy 
[root@nfs ~]# echo dirs /root >/srv/fc35/etc/rwtab.d/home 
[root@nfs ~]# echo dirs /var/lib/logrotate >/srv/fc35/etc/rwtab.d/logrotate 
[root@nfs ~]# echo dirs /var/lib/sss >/srv/fc35/etc/rwtab.d/sssd 
[root@nfs ~]# echo dirs /var/lib/systemd >/srv/fc35/etc/rwtab.d/systemd 
[root@nfs ~]# echo dirs /var/lib/upower >/srv/fc35/etc/rwtab.d/upower 
[root@nfs ~]# mkdir /srv/fc35/var/lib/upower 
[root@nfs ~]# echo dirs /var/lib/lightdm-data >/srv/fc35/etc/rwtab.d/lightdm 
[root@nfs ~]# printf %s\\n /etc/ssh /etc/hostname >/srv/fc35/etc/statetab.d/network 
[root@nfs ~]# echo /etc/krb5.keytab >/srv/fc35/etc/statetab.d/security 
[root@nfs ~]# touch /srv/fc35/etc/hostname /srv/fc35/etc/krb5.keytab

sssd and krb5 configuration

For authentication, I configured sssd and krb5.  User and group info comes from the /etc/ files, while passwords are authenticated with krb5.  krb5 FAST is configured in sssd to protect against password offline brute force attacks.

[root@nfs ~]# cat <<EOF >/srv/fc35/etc/krb5.conf.d/lan 
[libdefaults] 
default_realm = LAN 
dns_lookup_kdc = true 
permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac 
default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac 
default_tkt_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac 
[realms] 
LAN = { 
  disable_encrypted_timestamp = true
}
[domain_realm] 
.lan = LAN 
EOF
[root@nfs ~]# chroot /srv/fc35 authselect select sssd --force
Backup stored at /var/lib/authselect/backups/2021-10-05-18-42-46.pBq1G9 
Profile "sssd" was selected. 
The following nsswitch maps are overwritten by the profile:
- passwd 
- group 
- netgroup 
- automount 
- services 
Make sure that SSSD service is configured and enabled. See SSSD documentation for more information. 
[root@nfs ~]# chroot /srv/fc35 systemctl enable sssd-pam.socket
Created symlink /etc/systemd/system/sssd.service.wants/sssd-pam.socket → /usr/lib/systemd/system/sssd-pam.socket. 
[root@nfs ~]# cat <<EOF >/srv/fc35/etc/sssd/sssd.conf 
[sssd] 
config_file_version = 2 
domains = default 
[domain/default] 
id_provider = files 
auth_provider = krb5 
krb5_realm = LAN 
krb5_use_fast = demand 
dns_discovery_domain = lan 
EOF

Time Sync configuration

Configured chrony with hardware timestamps, NTP interleaving, my 5 local NTP sources and the NTP pool:

[root@nfs ~]# cat <<EOF >/srv/fc35/etc/chrony.conf 
pool 2.fedora.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
ntsdumpdir /var/lib/chrony 
server ntp-1.lan minpoll 0 maxpoll 4 xleave 
server ntp-2.lan minpoll 0 maxpoll 4 xleave
server ntp-3.lan minpoll 0 maxpoll 4 xleave 
server ntp-4.lan minpoll 0 maxpoll 4 xleave 
server ntp-5.lan minpoll 0 maxpoll 4 xleave 
hwtimestamp * 
stratumweight 0 
EOF

export directories via NFS

The fc35 directory is exported read-only and with only IP address restrictions and root is allowed to read all files.  The home directory requires krb5 credentials and root is treated as an unknown user.

[root@nfs ~]# cat <<EOF >>/etc/exports 
/srv/fc35 10.1.1.0/255.255.255.0(ro,sec=sys,no_root_squash) 
/home 10.1.1.0/255.255.255.0(rw,sec=krb5) 
EOF
[root@nfs ~]# exportfs -var
exporting 10.1.1.0/255.255.255.0:/srv/fc35
exporting 10.1.1.0/255.255.255.0:/home

NFS username mapping

idmapd translates usernames between different machines for NFS

[root@nfs ~]# cat <<EOF >/etc/idmapd.conf
[General]
Domain = lan
[Mapping]
Nobody-User = nobody
Nobody-Group = nobody
EOF
[root@nfs ~]# cp /etc/idmapd.conf /srv/fc35/etc/idmapd.conf
[root@nfs ~]# systemctl restart nfs-idmapd

Disable services not needed for a read-only root

These services don't work properly for a read-only root.

[root@nfs ~]# chroot /srv/fc35 systemctl disable dmraid-activation.service 
Removed /etc/systemd/system/sysinit.target.wants/dmraid-activation.service. 
[root@nfs ~]# chroot /srv/fc35 systemctl disable flatpak-add-fedora-repos.service 
Removed /etc/systemd/system/multi-user.target.wants/flatpak-add-fedora-repos.service. 
[root@nfs ~]# chroot /srv/fc35 systemctl disable dnf-makecache.timer 
Removed /etc/systemd/system/timers.target.wants/dnf-makecache.timer.
[root@nfs ~]# chroot /srv/fc35 systemctl disable rsyslog
Removed /etc/systemd/system/multi-user.target.wants/rsyslog.service.

Setup fstab

I'm having fstab mount the USB boot partition to /boot, and mount the home directories over NFS

[root@nfs ~]# cat <<EOF >/srv/fc35/etc/fstab 
LABEL=BOOT /boot vfat defaults 1 2 
nfs.lan:/home /home nfs4 sec=krb5 0 0 
EOF

Make USB drive bootable

This sets up the UEFI environment on the USB flash drive. Normally /boot/efi is a seperate partition from /boot, but I've combined them together here.  I generated a grub.cfg file manually and put it in /boot/efi/fedora/grub.cfg

[root@nfs ~]# dnf install grub2-efi grub2-efi-modules shim
... # lots of output
[root@nfs ~]# mount /dev/disk/by-id/usb-SanDisk_*9994*-part1 /mnt
[root@nfs ~]# sh -c 'cd /srv/fc35;find . \! -type l -print0 | cpio -pdV0 /mnt'
.......................... 413133 blocks 
[root@nfs ~]# mv /mnt/efi/EFI/* /mnt/efi 
[root@nfs ~]# cat <<EOF >/mnt/efi/fedora/grub.cfg
set default="0"
function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}
load_video
set gfxpayload=keep 
insmod gzio 
insmod part_gpt 
insmod ext2 
set timeout=2 
search --no-floppy --set=root -u 37E0-C3FF 
menuentry 'Netboot 35' --class fedora --class gnu-linux --class gnu --class os { 
  linuxefi /vmlinuz-5.14.9-300.fc35.x86_64 ro ip=dhcp root=nfs4:nfs.lan:/srv/fc35 console=tty0 audit=0 selinux=0 quiet readonly 
  initrdefi /initramfs-5.14.9-300.fc35.x86_64.img 
}
EOF

‌Kerberos keytab

I created the Kerberos principal and then put the hostname and Kerberos keytab on the USB flash drive state partition.‌

[root@nfs ~]# umount /mnt
[root@nfs ~]# mount /dev/disk/by-id/usb-SanDisk_*9994*-part2 /mnt
[root@nfs ~]# mkdir /mnt/etc 
[root@nfs ~]# echo usb-1.lan >/mnt/etc/hostname
[root@nfs ~]# kadmin.local
Authenticating as principal host/admin@LAN with password. 
kadmin.local:  addprinc -randkey +requires_preauth host/usb-1.lan
No policy specified for host/usb-1.lan@LAN; defaulting to no policy
Principal "host/usb-1.lan@LAN" created. 
kadmin.local:  ktadd -k /mnt/etc/krb5.keytab host/usb-1.lan 
Entry for principal host/usb-1.lan with kvno 2, encryption type aes256-cts-hmac-sha384-192 added to keytab WRFILE:/mnt/etc/krb5.keytab.
Entry for principal host/usb-1.lan with kvno 2, encryption type aes128-cts-hmac-sha256-128 added to keytab WRFILE:/mnt/etc/krb5.keytab. 
Entry for principal host/usb-1.lan with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/mnt/etc/krb5.keytab. 
Entry for principal host/usb-1.lan with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/mnt/etc/krb5.keytab.

Booting

I unmounted the USB drive from the NFS server and plugged it into my desktop computer. Going into the bios, it detected the EFI partition on the USB drive and showed the "Fedora" label for it. Booting was a success!

Testing Home directory exports

Reading files off the /home directory on the netbooted system requires your user have a valid krb5 token. If you're sshing in rather than logging into the GUI, use the "GSSAPIDelegateCredentials yes" option.

[myuser@usb-1 ~]$ klist | grep nfs
10/09/2021 03:27:43  10/09/2021 13:27:43  nfs/nfs.lan@LAN
[myuser@usb-1 ~]$ ls -l /home
total 52
drwx--x--- 167 myuser myuser 12288 Oct  9 03:27 myuser
drwx------   7 nobody nobody  4096 Aug 19  2019 otheruser

If you don't have a token, even root won't grant access:

[myuser@usb-1 ~]$ sudo -u nobody ls -l /home/otheruser
ls: cannot access '/home/otheruser': Permission denied
[myuser@usb-1 ~]$ sudo ls -l $HOME
ls: cannot open directory '/home/myuser': Permission denied

Updates and kernels

Since the filesystem is read-only, the nfs server has to apply updates:

[root@nfs ~]# chroot /srv/fc35 dnf update
... # lots of output

Any kernel updated through dnf won't automatically make it to the USB drive, that will have to be maintained as well.