FreeBSD 11.1 on ZFS with full disk encryption

Starting with FreeBSD 11 the loader and kernel can be stored on an encrypted partition, which offers new possibilities of using full disk encrypted ZFS.

This document describes the manual installation of FreeBSD 11.1 on ZFS, using full disk encryption, including encrypted swap space.

In the following example I used my trustworthy HP N40L microserver to install FreeBSD 11.1 on a SanDisk 240 GB SSD attached to the on-board SATA bus, so that the four internal SATA enclosures can be used to put large data drives in. This explains the use of ada4 as the name of the internal SSD, as the discs in the 4 bay SATA enclosure are numbered from ada0 to ada3. It is important to set the drive order correctly in the BIOS and have the system boot from USB or the SSD accordingly. That way you can have the system boot from a USB flash drive while installing, whereas as soon as you remove the USB flash drive, the system will automatically boot from the internal SSD.

Boot installer from USB flash drive

1. Download FreeBSD-11.1-RELEASE-amd64-memstick.img from a FreeBSD mirror web site, e.g.
   ftp://ftp.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-memstick.img
2. Create bootable USB flash drive from downloaded image file, e.g.
   sudo dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/diskXXX bs=64k
   Important: change /dev/diskXXX to whatever device your USB flash drive actually is
3. Put the USB flash drive into one of the front side USB ports and boot into the FreeBSD installer
4. After the installer has loaded, exit to the Shell

Partition your disk

1. If you use some other system or disk, you have to substitue all occurences of ada4 with your actual device!
2. Destroy any old partition table probably already on the SSD
   gpart destroy -F ada4
3. Create GPT partition table
   gpart create -s gpt ada4
4. Add boot partition as the first partition and label it as boot0, which makes it easy to address that partition later on. 512 KB is large enough.
   Note: You can also align the partitions on 4K boundaries for newer drives or even use 1M alignment, which also is a multiple of 4K to enhance the performance. To do this you add -a 4K or -a 1m to the gpart statements below. In my tests however for that particular kind of server and SSD it showed no difference in performance, probably because due to encryption I/O is rather CPU bound.
   gpart add -l boot0 -t freebsd-boot -s 512K ada4
5. Add swap partition as the second partition and label it as swap0. As the machine contains 16 GB of RAM, I use swap space twice the size of RAM.
   gpart add -l swap0 -t freebsd-swap -a 1m -s 32G ada4
6. For my setup I wanted to divide the remaining space into two equal parts for two separate root pools, so I could later on install two different versions of FreeBSD at the same time. A 240 GB SSD provides ample space to do a dual boot setup. In case you only want to use one root pool for a single installation, you can omit the creation of the second root partition root1 and just use the remaining space on your drive to create one single root partition root0. We will however create both root partition, but only use the first one for this tutorial.
   gpart add -l root0 -t freebsd-zfs -s 200883180 ada4
gpart add -l root1 -t freebsd-zfs -s 200883180 ada4
7. Now your disk should look like this, with or without ada4p4 of course:
   gpart show ada4
   

=>       40  468877232  ada4  GPT  (224G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048   67108864     2  freebsd-swap  (32G)
   67110912  200883180     3  freebsd-zfs  (96G)
  267994092  200883180     4  freebsd-zfs  (96G)

Installation of bootcode

Now you have to install the bootcode onto the boot partition ada4p1. In our case we have to use gptzfsboot as we want to boot from a ZFS file system which resides on a disk using GPT:
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada4
partcode written to ada4p1
bootcode written to add4

Set up encrypted root partition

1. Initialize geli for the root partition
   Using -g tells the boot loader where to find the boot partition. As our boot partition is also encrypted, the boot loader has no means of deciding what partition is the actual boot partition, as without the password it cannot look into the partitions’ content.
   geli init -l 256 -g -b /dev/gpt/root0
   Important: Use a secure password!
2. Attach the geli provider for the root partition we just created
   geli attach /dev/gpt/root0

Create file systems on root partition

1. Create zpool named RPool
zpool create -m none RPool /dev/gpt/root0.eli
2. Set ZFS options
   zfs set atime=off RPool
zfs set checksum=on RPool
3. Create filesystems on ZFS root partition
   zfs create RPool/ROOT
zfs create -o mountpoint=/tmp/mnt/RPool RPool/ROOT/default
4. Create the rest of the file systems on the ZFS root partition. This is only an example and you may chose whatever ZFS options and hierarchies suit you best.
   zfs create -o mountpoint=/tmp/mnt/RPool/tmp -o setuid=off -o exec=on RPool/tmp
zfs create -o mountpoint=/tmp/mnt/RPool/usr RPool/usr
zfs create -o mountpoint=/tmp/mnt/RPool/usr/local RPool/usr/local
zfs create -o mountpoint=/tmp/mnt/RPool/usr/ports RPool/usr/ports
zfs create -o mountpoint=/tmp/mnt/RPool/usr/src RPool/usr/src
zfs create -o mountpoint=/tmp/mnt/RPool/var RPool/var
zfs create -o mountpoint=/tmp/mnt/RPool/var/log RPool/var/log
zfs create -o mountpoint=/tmp/mnt/RPool/var/mail RPool/mail

Installation of the basic FreeBSD system

1. Change working directory to our future root directory, temporarily mounted under /tmp/mnt/RPool:
   cd /tmp/mnt/RPool
2. You can decide which packages to install, however you should at least install kernel and base. I opted for an additional installation of src, which contains the sources.
   unxz -c /usr/freebsd-dist/kernel.txz | tar xpvf -
unxz -c /usr/freebsd-dist/base.txz | tar xpvf -
unxz -c /usr/freebsd-dist/src.txz | tar xpvf -
3. Change root directory to our future root directory.
   This is not necessary, but makes it easier to do the configuration.
   chroot /tmp/mnt/RPool
4. Customize loader.conf
echo 'aesni_load="YES"' > /boot/loader.conf
echo 'geom_eli_load="YES"' >> /boot/loader.conf
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'kern.geom.label.disk_ident.enable="1"' >> /boot/loader.conf
echo 'kern.geom.label.gptid.enable="1"' >> /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:RPool/ROOT/default"' >> /boot/loader.conf
5. Enable zfs in rc.conf
echo 'zfs_enable="YES"' > /etc/rc.conf
6. At this point you can also set up things like your keyboard mapping, time zone, hostname, network including nameserver etc. by adding additional options to /etc/rc.conf, /etc/resolv.conf or other configuration files accordingly. You can also opt to do that later after you booted into your freshly installed system.
7. Exit from the change root environment
   exit
8. Unmount all mounted ZFS based file systems
   zfs umount -a
cd
9. Setup mount points for the different zfs based file systems created earlier
   zfs set -o mountpoint=legacy RPool/ROOT/default
zfs set -o mountpoint=/tmp RPool/ROOT/tmp
zfs set -o mountpoint=/usr RPool/ROOT/usr
zfs set -o mountpoint=/usr/local RPool/ROOT/usr/local
zfs set -o mountpoint=/usr/ports RPool/ROOT/usr/ports
zfs set -o mountpoint=/usr/src RPool/ROOT/usr/src
zfs set -o mountpoint=/var RPool/ROOT/var
zfs set -o mountpoint=/var/log RPool/ROOT/var/log
zfs set -o mountpoint=/var/mail RPool/ROOT/var/mail

Reboot into installed system

Now we are ready to boot into our installed system:
reboot

Post-installation tasks

1. During the boot process you need your password for the encryption
2. Set root password (at the first boot you can log in into the system with an empty password by simply pressing return)
   passwd
3. Set up keyboard mapping in case you haven’t done this earlier by adding for example keymap="de" to your /etc/rc.conf

Activate enrypted swap space

As we didn’t setup our encrypted swap space during the system installation, now is a good time to do that:
geli onetime -d -e AES-XTS -l 256 -s 4096 /dev/gpt/swap0
We also need to add an entry to /etc/fstab so next time our swap space is activated automatically

# /etc/fstab
# device                        mountpoint      fstype  mountoptions    dump    pass
#
/dev/gpt/swap.eli               none            swap    sw              0       0

and have FreeBSD use the configured swap space for this incarnation
swapon -a
Check whether the swap space is actually in use:
swapinfo
should give you something like this:

Device          1K-blocks     Used    Avail Capacity
/dev/gpt/swap0.eli  33554432        0 33554432     0%

Install Ports Collection

After you have set up your network connection, you can install the Ports Collection with
portsnap fetch
portsnap extract
and install all the other software packages you need. This works exactly as with any other FreeBSD installation you may have installed and configured already earlier.

Add additional encrypted drives to your system

When you add additional drives to your system, it is important you also do this in a similar fashion to the boot and root drives earlier.
For this example I use ada0, which is the drive inserted into the first bay.

1. Destroy existing partition table
   gpart destroy -F ada0
2. Create GPT partition table
   gpart create -s gpt ada0
3. Add ZFS partition, label it accordingly and make it 4k aligned (or 1m, as you like)
   gpart add -t freebsd-zfs -l data0 -a 4k ada0
4. Create geli
   geli init -b -s 4096 -l 256 /dev/gpt/data0
5. Attach geli
   geli attach /dev/gpt/data0
6. Create new zpool on that partition
   zpool create -m none datapool0 /dev/gpt/data0.eli
7. Now you can start adding zfs file systems as you wish You can also create mirrored pools, RAID-Z based pools and so on, as you like. First you have to create the geli, attach those and then you can create your zpool on those attached geli providers, e.g.
   zpool create -m none datapool0 mirror /dev/gpt/data0a.eli /dev/gpt/data0b.eli