LinkedIn Sourceforge

Vincent's Blog

Pleasure in the job puts perfection in the work (Aristote)

Special FreeBSD install for a NAS

Posted on 2024-10-06 19:41:00 from Vincent in OpenBSD Nas

My last evaluation on other BSD has decided me to use ZFS for my NAS.

My OpenBSD NAS server is working since +8 years very well, but I'm at 85% of his capacity.

Despite it offers a pseudo-snaptshot system via rsync, this little machine becomes too small to manage and provide files of several GB we have today. So, why not replace the software when we have to replace the hardware ;).

In this post, I will not perform the standard install of FreeBSD. I will explain why I do that.


Introduction

I've never really work with FreeBSD, so I have to learn it. But I know a bit of ZFS because in one of past job I did the Sun Solaris Admin training. It was long time ago, but I still have colleagues working with ZFS. So, I know few concepts.

I do not repeat what I've describe in a previous blog's post, but my NAS respects the following requirements:
- Low power consumption
- HardWare able to run 24x7
- At least one interface of 2.5Gbps
- possibilities for 4 sata slots
- having one nvme slot

And compliant with last stable version of FreeBSD ;)

My idea is to use the nvme as Layer 2 caching system. To make is short, ZFS use the RAM to store files and asynchronously he is send them back to the disks. In such case the RAM of the machine is playing the role of Layer 1 of cache. With ZFS, you can define a Layer 2 of cache, slower than RAM, but faster than Disk. This approach is often used within big professional storage systems.
To be really correct, I should have 2 nvme for redundancies. But this is not a professional NAS

So, as you see the nvme storage is shared between the OS, the cache and the log.
This is explained in details by Lawrence Systems in this video: https://www.youtube.com/watch?v=M4DLChRXJog&t=881s

This specific reason force me to skip the standard installation mechanism.

But you will see, installing FreeBSD, like OpenBSD that I know very well, is just copying files.

Setup Freebsd

Size of the log: ZIL

In the asynchronous approach, before writing files on disks, ZFS store them in RAM. This provide a very high performance of ZFS, but with the risk that you could loose this info in case of power cut or in case of crash of the OS.
If you want to force a real "synchronous" approach, ZFS will write the file in the layer 2, what we call ZIL, before writing to disks. Because of his speed, this is where an nvme disk has a perfect match. Despite this is not the final destination for the files, they are safe is case of power cut or crash.
Since the Layer 2 cache receive every 5 seconds information coming from the RAM at the speed of the network interface, we can size the ZIL like this:

2.5Gbps x 5sec / 8 = 1.5G

In case I install a card of 10Gbps, I'll size the ZIL with 8GB.

Size of the cache:

In the other direction, ZFS has another caching called "log". We use quick disks to store files users are regularly asking. ZFS decides which files can goes in this zone. I did not find a clear rule to size it correctly. It depends of the size of the files your have and the Ethernet throughput you have.
In my case I will size it to 64GB

To have a kind of resilience, but having a good performance, I've decided to use the raid-Z1 disk's organisation. I let you check the perfect explanation made by Victor Mendon about the different disk's organisation and their impact on resilience and performance.

Installation

To perform this specific installation, I boot an USB key with the standard installation image. Nevertheless, I leave the installation screen to start a shell session

Since I have a X64 machine, I download from FreeBSD installation media the img for amd64.

To put it on an USB key, I do:

obsd:~ $ doas dd if=FreeBSD-14.1-RELEASE-amd64-mini-memstick.img of=/dev/rsd2c bs=1M

Be very careful to select the correct disk for you specific case.

Once you have booted the FreeBSD installer, leave to shell session and type the following commands:

Set the keyboard mapping to my needs (Belgian in my case)

kbdmap <be>

To check that we are well with an EFI board:

sysctl machdep.bootmethod

Slice the nvme disk (called /dev/nda0 in my case):

gpart destroy -F /dev/nda0
gpart create -s gpt /dev/nda0
gpart add -t efi -s 200M -l efi0 /dev/nda0
gpart add -t freebsd-swap -a 4k -s 48G -l swap0 /dev/nda0
gpart add -t freebsd-zfs -a 4k -s 8G -l zil0 /dev/nda0
gpart add -t freebsd-zfs -a 4k -s 64G -l l2arc0 /dev/nda0
gpart add -t freebsd-zfs -a 4k -l os0 /dev/nda0

Format the EFI file system:

newfs_msdos /dev/nda0p1

To be able to create subdirectory of /mnt and, later, to save network parameters on this USB key:

mount -uw /

Create the ZFS pool, datasets and inform which one if the bootfs:

zpool create -f -o ashit=12 -o altroot=/mnt -O mountpoint=none -O atime=off -O compression=lz4  rpool /dev/gpt/os0
zfs create                                              -o mountpoint=none       rpool/ROOT
zfs create                                              -o mountpoint=/          rpool/ROOT/default
zfs create -o compression=off                           -o monutpoint=none       rpool/usr 
zfs create -o compression=off                           -o mountpoint=none       rpool/var 
zfs create -o compression=off -o exec=on  -o setuid=off -o mountpoint=/tmp       rpool/tmp 
zfs create                    -o exec=off -o setuid=off -o mountpoint=/var/crash rpool/var/crash 
zfs create                    -o exec=off -o setuid=off -o mountpoint=/var/log   rpool/var/log 
zfs create   -o atime=on      -o exec=off -o setuid=off -o mountpoint=/var/mail  rpool/var/mail 
zfs create -o compression=off -o exec=on  -o setuid=off -o mountpoint=/var/tmp   rpool/var/tmp 
zfs create                                              -o mountpoint=/home      rpool/home
zfs create                                              -o mountpoint=/home/vi   rpool/home/vi

chmod 1777 /mnt/tmp
chmod 1777 /mnt/var/tmp

zpool set bootfs=rpool/ROOT/default rpool

Configure the network and download base.txz and kernel.txz:

bsdconfig <network>
cd /mnt
fetch https://download.freebsd.org/ftp/releases/amd64/   base.txz
fetch https://download.freebsd.org/ftp/releases/amd64/   kernel.txz

Install FreeBSD:

tar -C /mnt -xpjf  base.txz
tar -C /mnt -xpjf  kernel.txz

Setup EFI:

mkdir -p /mnt/boot/efi
mount -t msdosfs /dev/nda0p1 /mnt/boot/efi
mkdir -p /mnt/boot/efi/EFI/BOOT
cp /boot/loader.efi /mnt/boot/efi/EFI/BOOT/BOOTX64.EFI

Minimal config of our new system:

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"'  >> /mnt/boot/loader.conf

To setup the root password, activate the swap and define the hostname, we start a chroot session:

chroot /mnt
bsdconfig <passwd>, <timezone>
echo "/dev/nda0p2    none    swap    sw 0  0" >> /etc/fstab
echo 'hostname="fbsd-nas"' >> /etc/rc.conf
echo 'vfs.zfs.min_auto_ashift=12' >> /etc/sysctl.conf
exit

The system is now fully installed ;)

reboot

We have still few config to perform:

kbdmap <be> 
bsdconfig <network>
freebsd-update fetch
freebsd-update install
pkg install ncurses
pkg install terminfo-db

Create the zpool and dataset for the NAS:

mkdir /nas
zpool create -o ashift=12 -O mountpoint=none -O compression=lz4 -O atime=off naspool raidz1 /dev/ada0 /dev/ada1 /dev/ada2 /dev/ada3
zpool add naspool log /dev/gpt/zil0
zpool add naspool cache /dev/gpt/l2arc0
zfs create  -o exec=off -o setuid=off -o mountpoint=/nas/sharevar/log   rpool/var/log

Conclusion

I have a working machine. Different file systems / datasets seams to have the correct behaviour.

I can perform the required snapshots. I think not too far from the standard FreeBSD install.

Feel free to react if you find something incorrect ;).
It's my first weeks with FreeBSD, so I have maybe missed some important elements.



2, 0
displayed: 2026



What is the first vowel of the word Python?