NFS Readonly Root Filesystem
Setting up a Fedora Readonly Root filesystem via NFS
I remember when Knoppix came out. It was innovative by making it possible to run a full Linux system off a read-only CD. It was useful in many different ways: a way to repair your system if it wasn't working normally, a secure OS to use if you didn't trust someone else's computer, and a way to try out Linux without having to deal with installer headaches.
I have a Windows desktop that I would occasionally like to run Linux on it. Normally, I use a lot of WSL but sometimes I need Linux with access to the hardware, rather than a VM. My choices for doing this would be to install to a local drive, use a live CD/USB, or netboot.
Since USB2.0 flash drives are very slow and I have a Linux server connected via 2.5G ethernet, I wanted to try out netbooting.
When netbooting, one problem to solve is transferring secrets. Things like the ssh host keys, the shadow password file, and the krb5 host keytab. There's a few ways to handle that, and I chose to use a USB flash drive. That way the secrets are not transferred over the network in clear text, as they would be with NFS or TFTP.
The OS itself will be exported read-only. This means that multiple desktops can share it at the same time, they just each need their own USB flash drive.
I chose Fedora as my OS as I like their release schedule for Desktops.
Setting up the USB flash drive: partition
To setup the USB flash drive, I wrote a new GPT partition table. I created a 1GB EFI partition and a 14GB Linux partition.
[root@nfs ~]# fdisk /dev/disk/by-id/usb-SanDisk*9994*-0:0
Welcome to fdisk (util-linux 2.36.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0: 14.91 GiB, 16008609792 bytes, 31266816 sectors
Disk model: Ultra
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0926F9AC-DAC7-1144-B8FA-38F7894CCE49
Device Start End Sectors Size Type
/dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0-part1 2048 31266782 31264735 14.9G Microsoft basic data
Command (m for help): g
Created a new GPT disklabel (GUID: 3586EEDF-45A2-7146-8AA7-4B32C369E357).
Command (m for help): n
First sector (2048-31266782, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-31266782, default 31266782): +1G
Created a new partition 1 of type 'Linux filesystem' and of size 1 GiB.
Partition #1 contains a ntfs signature. Do you want to remove the signature? [Y]es/[N]o: y
The signature will be removed by a write command.
Command (m for help): n
Partition number (2-128, default 2):
First sector (2099200-31266782, default 2099200):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2099200-31266782, default 31266782):
Created a new partition 2 of type 'Linux filesystem' and of size 13.9 GiB.
Command (m for help): t
Partition number (1,2, default 2): 1
Partition type or alias (type L to list all): 1
Changed type of partition 'Linux filesystem' to 'EFI System'.
Command (m for help): p
Disk /dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0: 14.91 GiB, 16008609792 bytes, 31266816 sectors
Disk model: Ultra
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F8BDB994-FEBE-6745-BA7B-C945FF53BF93
Device Start End Sectors Size Type
/dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0-part1 2048 2099199 2097152 1G EFI System
/dev/disk/by-id/usb-SanDisk_Ultra_4C530099941117103095-0:0-part2 2099200 31266782 29167583 13.9G Linux filesystem
Filesystem/RAID signature on partition 1 will be wiped.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.Setting up the USB flash drive: filesystems
I created a vfat filesystem on partition 1, and an ext2 filesystem on partition 2. I then gave the ext2 filesystem the label "stateless-state" so the readonly-root package can find it. I gave the vfat filesystem the label "BOOT". I'll put data on them both in a later section.
[root@nfs ~]# mkfs.vfat /dev/disk/by-id/usb-SanDisk*9994*-part1
mkfs.fat 4.2 (2021-01-31)
[root@nfs ~]# mkfs.ext2 /dev/disk/by-id/usb-SanDisk*9994*-part2
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 3645947 4k blocks and 912128 inodes
Filesystem UUID: 9387e5cc-5538-40af-84b2-8c6b494122d7
Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
[root@nfs ~]# dosfslabel /dev/disk/by-id/usb-SanDisk*9994*-part1 BOOT
[root@nfs ~]# e2label /dev/disk/by-id/usb-SanDisk*9994*-part2 stateless-stateOS Image install
I started with these instructions and updated them for Fedora 35: https://fedoramagazine.org/how-to-build-a-netboot-server-part-1/
On my NFS server, I installed Fedora with the Xfce GUI Desktop to /srv/fc35. This uses about 5GB of space.
[root@nfs ~]# mkdir /srv/fc35
[root@nfs ~]# dnf -y --releasever=35 --installroot=/srv/fc35 install fedora-release systemd passwd rootfiles sudo dracut dracut-network nfs-utils vim-minimal dnf readonly-root krb5-workstation
... # lots of output
Transaction Summary
========================================================
Install 243 Packages
... # lots more output
[root@nfs ~]# dnf -y --releasever=35 --installroot=/srv/fc35 group install "Xfce Desktop"
... # lots of output
[root@nfs ~]# dnf -y --installroot=/srv/fc35 install firefox
... # lots of outputKernel custom boot config
Next, I setup the dracut/initramfs configuration and let dnf install the kernel. The dracut settings will setup an initramfs that has the modules needed for a net boot.
[root@nfs ~]# echo 'hostonly=no' > /srv/fc35/etc/dracut.conf.d/hostonly.conf
[root@nfs ~]# echo 'add_dracutmodules+=" network nfs "' > /srv/fc35/etc/dracut.conf.d/netboot.conf
[root@nfs ~]# dnf --installroot=/srv/fc35 install kernelBase OS Configuration
I configured the locale, configured the console messages, added my user, told journald to log to memory, and setup the DNS resolver IP:
[root@nfs ~]# echo 'LANG="en_US.UTF-8"' > /srv/fc35/etc/locale.conf
[root@nfs ~]# echo 'kernel.printk = 0 4 1 7' > /srv/fc35/etc/sysctl.d/00-printk.conf
[root@nfs ~]# chroot /srv/fc35 useradd myuser
[root@nfs ~]# echo 'myuser ALL=(ALL) NOPASSWD: ALL' > /srv/fc35/etc/sudoers.d/myuser
[root@nfs ~]# sed -i 's/^#Storage=auto$/Storage=volatile/' /srv/fc35/etc/systemd/journald.conf
[root@nfs ~]# echo nameserver 10.1.1.1 >/srv/fc35/etc/resolv.confreadonly-root configuration
The readonly-root package lets you redirect some writes to a different location. This looks for the filesystem labeled "stateless-state" and bind-mounts the persistent state files (listed in /etc/statetab.d) there. The temporary state files (listed in /etc/rwtab.d) are kept in a ramdisk.
[root@nfs ~]# cat <<EOF >/srv/fc35/etc/sysconfig/readonly-root
# Set to 'yes' to mount the system filesystems read-only.
# NOTE: It's necessary to append 'ro' to mount options of '/' mount point in
# /etc/fstab as well, otherwise the READONLY option will not work. READONLY=yes
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
TEMPORARY_STATE=yes
# Place to put a tmpfs for temporary scratch writable space
RW_MOUNT=/var/lib/stateless/writable
# Label on local filesystem which can be used for temporary scratch space
#RW_LABEL=stateless-rw
# Options to use for temporary mount
RW_OPTIONS=
# Label for partition with persistent data
STATE_LABEL=stateless-state
# Where to mount to the persistent data
STATE_MOUNT=/var/lib/stateless/state
# Options to use for persistent mount
STATE_OPTIONS=
# NFS server to use for persistent data?
CLIENTSTATE=
# Use slave bind-mounts
SLAVE_MOUNTS=yes
EOF
[root@nfs ~]# echo dirs /var/lib/gssproxy >/srv/fc35/etc/rwtab.d/gssproxy
[root@nfs ~]# echo dirs /root >/srv/fc35/etc/rwtab.d/home
[root@nfs ~]# echo dirs /var/lib/logrotate >/srv/fc35/etc/rwtab.d/logrotate
[root@nfs ~]# echo dirs /var/lib/sss >/srv/fc35/etc/rwtab.d/sssd
[root@nfs ~]# echo dirs /var/lib/systemd >/srv/fc35/etc/rwtab.d/systemd
[root@nfs ~]# echo dirs /var/lib/upower >/srv/fc35/etc/rwtab.d/upower
[root@nfs ~]# mkdir /srv/fc35/var/lib/upower
[root@nfs ~]# echo dirs /var/lib/lightdm-data >/srv/fc35/etc/rwtab.d/lightdm
[root@nfs ~]# printf %s\\n /etc/ssh /etc/hostname >/srv/fc35/etc/statetab.d/network
[root@nfs ~]# echo /etc/krb5.keytab >/srv/fc35/etc/statetab.d/security
[root@nfs ~]# touch /srv/fc35/etc/hostname /srv/fc35/etc/krb5.keytabsssd and krb5 configuration
For authentication, I configured sssd and krb5. User and group info comes from the /etc/ files, while passwords are authenticated with krb5. krb5 FAST is configured in sssd to protect against password offline brute force attacks.
[root@nfs ~]# cat <<EOF >/srv/fc35/etc/krb5.conf.d/lan
[libdefaults]
default_realm = LAN
dns_lookup_kdc = true
permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
default_tkt_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
[realms]
LAN = {
disable_encrypted_timestamp = true
}
[domain_realm]
.lan = LAN
EOF
[root@nfs ~]# chroot /srv/fc35 authselect select sssd --force
Backup stored at /var/lib/authselect/backups/2021-10-05-18-42-46.pBq1G9
Profile "sssd" was selected.
The following nsswitch maps are overwritten by the profile:
- passwd
- group
- netgroup
- automount
- services
Make sure that SSSD service is configured and enabled. See SSSD documentation for more information.
[root@nfs ~]# chroot /srv/fc35 systemctl enable sssd-pam.socket
Created symlink /etc/systemd/system/sssd.service.wants/sssd-pam.socket → /usr/lib/systemd/system/sssd-pam.socket.
[root@nfs ~]# cat <<EOF >/srv/fc35/etc/sssd/sssd.conf
[sssd]
config_file_version = 2
domains = default
[domain/default]
id_provider = files
auth_provider = krb5
krb5_realm = LAN
krb5_use_fast = demand
dns_discovery_domain = lan
EOFTime Sync configuration
Configured chrony with hardware timestamps, NTP interleaving, my 5 local NTP sources and the NTP pool:
[root@nfs ~]# cat <<EOF >/srv/fc35/etc/chrony.conf
pool 2.fedora.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
ntsdumpdir /var/lib/chrony
server ntp-1.lan minpoll 0 maxpoll 4 xleave
server ntp-2.lan minpoll 0 maxpoll 4 xleave
server ntp-3.lan minpoll 0 maxpoll 4 xleave
server ntp-4.lan minpoll 0 maxpoll 4 xleave
server ntp-5.lan minpoll 0 maxpoll 4 xleave
hwtimestamp *
stratumweight 0
EOFexport directories via NFS
The fc35 directory is exported read-only and with only IP address restrictions and root is allowed to read all files. The home directory requires krb5 credentials and root is treated as an unknown user.
[root@nfs ~]# cat <<EOF >>/etc/exports
/srv/fc35 10.1.1.0/255.255.255.0(ro,sec=sys,no_root_squash)
/home 10.1.1.0/255.255.255.0(rw,sec=krb5)
EOF
[root@nfs ~]# exportfs -var
exporting 10.1.1.0/255.255.255.0:/srv/fc35
exporting 10.1.1.0/255.255.255.0:/homeNFS username mapping
idmapd translates usernames between different machines for NFS
[root@nfs ~]# cat <<EOF >/etc/idmapd.conf
[General]
Domain = lan
[Mapping]
Nobody-User = nobody
Nobody-Group = nobody
EOF
[root@nfs ~]# cp /etc/idmapd.conf /srv/fc35/etc/idmapd.conf
[root@nfs ~]# systemctl restart nfs-idmapdDisable services not needed for a read-only root
These services don't work properly for a read-only root.
[root@nfs ~]# chroot /srv/fc35 systemctl disable dmraid-activation.service
Removed /etc/systemd/system/sysinit.target.wants/dmraid-activation.service.
[root@nfs ~]# chroot /srv/fc35 systemctl disable flatpak-add-fedora-repos.service
Removed /etc/systemd/system/multi-user.target.wants/flatpak-add-fedora-repos.service.
[root@nfs ~]# chroot /srv/fc35 systemctl disable dnf-makecache.timer
Removed /etc/systemd/system/timers.target.wants/dnf-makecache.timer.
[root@nfs ~]# chroot /srv/fc35 systemctl disable rsyslog
Removed /etc/systemd/system/multi-user.target.wants/rsyslog.service.
Setup fstab
I'm having fstab mount the USB boot partition to /boot, and mount the home directories over NFS
[root@nfs ~]# cat <<EOF >/srv/fc35/etc/fstab
LABEL=BOOT /boot vfat defaults 1 2
nfs.lan:/home /home nfs4 sec=krb5 0 0
EOFMake USB drive bootable
This sets up the UEFI environment on the USB flash drive. Normally /boot/efi is a seperate partition from /boot, but I've combined them together here. I generated a grub.cfg file manually and put it in /boot/efi/fedora/grub.cfg
[root@nfs ~]# dnf install grub2-efi grub2-efi-modules shim
... # lots of output
[root@nfs ~]# mount /dev/disk/by-id/usb-SanDisk_*9994*-part1 /mnt
[root@nfs ~]# sh -c 'cd /srv/fc35;find . \! -type l -print0 | cpio -pdV0 /mnt'
.......................... 413133 blocks
[root@nfs ~]# mv /mnt/efi/EFI/* /mnt/efi
[root@nfs ~]# cat <<EOF >/mnt/efi/fedora/grub.cfg
set default="0"
function load_video {
insmod efi_gop
insmod efi_uga
insmod video_bochs
insmod video_cirrus
insmod all_video
}
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
set timeout=2
search --no-floppy --set=root -u 37E0-C3FF
menuentry 'Netboot 35' --class fedora --class gnu-linux --class gnu --class os {
linuxefi /vmlinuz-5.14.9-300.fc35.x86_64 ro ip=dhcp root=nfs4:nfs.lan:/srv/fc35 console=tty0 audit=0 selinux=0 quiet readonly
initrdefi /initramfs-5.14.9-300.fc35.x86_64.img
}
EOFKerberos keytab
I created the Kerberos principal and then put the hostname and Kerberos keytab on the USB flash drive state partition.
[root@nfs ~]# umount /mnt
[root@nfs ~]# mount /dev/disk/by-id/usb-SanDisk_*9994*-part2 /mnt
[root@nfs ~]# mkdir /mnt/etc
[root@nfs ~]# echo usb-1.lan >/mnt/etc/hostname
[root@nfs ~]# kadmin.local
Authenticating as principal host/admin@LAN with password.
kadmin.local: addprinc -randkey +requires_preauth host/usb-1.lan
No policy specified for host/usb-1.lan@LAN; defaulting to no policy
Principal "host/usb-1.lan@LAN" created.
kadmin.local: ktadd -k /mnt/etc/krb5.keytab host/usb-1.lan
Entry for principal host/usb-1.lan with kvno 2, encryption type aes256-cts-hmac-sha384-192 added to keytab WRFILE:/mnt/etc/krb5.keytab.
Entry for principal host/usb-1.lan with kvno 2, encryption type aes128-cts-hmac-sha256-128 added to keytab WRFILE:/mnt/etc/krb5.keytab.
Entry for principal host/usb-1.lan with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/mnt/etc/krb5.keytab.
Entry for principal host/usb-1.lan with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/mnt/etc/krb5.keytab.Booting
I unmounted the USB drive from the NFS server and plugged it into my desktop computer. Going into the bios, it detected the EFI partition on the USB drive and showed the "Fedora" label for it. Booting was a success!
Testing Home directory exports
Reading files off the /home directory on the netbooted system requires your user have a valid krb5 token. If you're sshing in rather than logging into the GUI, use the "GSSAPIDelegateCredentials yes" option.
[myuser@usb-1 ~]$ klist | grep nfs
10/09/2021 03:27:43 10/09/2021 13:27:43 nfs/nfs.lan@LAN
[myuser@usb-1 ~]$ ls -l /home
total 52
drwx--x--- 167 myuser myuser 12288 Oct 9 03:27 myuser
drwx------ 7 nobody nobody 4096 Aug 19 2019 otheruser
If you don't have a token, even root won't grant access:
[myuser@usb-1 ~]$ sudo -u nobody ls -l /home/otheruser
ls: cannot access '/home/otheruser': Permission denied
[myuser@usb-1 ~]$ sudo ls -l $HOME
ls: cannot open directory '/home/myuser': Permission deniedUpdates and kernels
Since the filesystem is read-only, the nfs server has to apply updates:
[root@nfs ~]# chroot /srv/fc35 dnf update
... # lots of outputAny kernel updated through dnf won't automatically make it to the USB drive, that will have to be maintained as well.