Thank you for visiting!
My little window on internet allowing me to share several of my passions
Categories:
- FreeBSD
- OpenBSD
- VM
- High Availability
- vdcron
- My Sysupgrade
- FreeBSD
- Nas
- DragonflyBSD
- fapws
- Alpine Linux
- Openbox
- Desktop
- Security
- nvim
- yabitrot
- nmctl
- Tint2
- Firewall
- VPN
- Project Management
- Hifi
- Alarm
Most Popular Articles:
Last Articles:

Special FreeBSD install for a NAS
Posted on 2024-10-06 19:41:00 from Vincent in OpenBSD Nas
My last evaluation on other BSD has decided me to use ZFS for my NAS.
My OpenBSD NAS server is working since +8 years very well, but I'm at 85% of his capacity.
Despite it offers a pseudo-snaptshot system via rsync, this little machine becomes too small to manage and provide files of several GB we have today. So, why not replace the software when we have to replace the hardware ;).
In this post, I will not perform the standard install of FreeBSD. I will explain why I do that.
Introduction
I've never really work with FreeBSD, so I have to learn it. But I know a bit of ZFS because in one of past job I did the Sun Solaris Admin training. It was long time ago, but I still have colleagues working with ZFS. So, I know few concepts.
I do not repeat what I've describe in a previous blog's post, but my NAS respects the following requirements:
- Low power consumption
- HardWare able to run 24x7
- At least one interface of 2.5Gbps
- possibilities for 4 sata slots
- having one nvme slot
And compliant with last stable version of FreeBSD ;)
My idea is to use the nvme as Layer 2 caching system. To make is short, ZFS use the RAM to store files and asynchronously he is send them back to the disks. In such case the RAM of the machine is playing the role of Layer 1 of cache. With ZFS, you can define a Layer 2 of cache, slower than RAM, but faster than Disk. This approach is often used within big professional storage systems.
To be really correct, I should have 2 nvme for redundancies. But this is not a professional NAS
So, as you see the nvme storage is shared between the OS, the cache and the log.
This is explained in details by Lawrence Systems in this video: https://www.youtube.com/watch?v=M4DLChRXJog&t=881s
This specific reason force me to skip the standard installation mechanism.
But you will see, installing FreeBSD, like OpenBSD that I know very well, is just copying files.
Setup Freebsd
Size of the log: ZIL
In the asynchronous approach, before writing files on disks, ZFS store them in RAM. This provide a very high performance of ZFS, but with the risk that you could loose this info in case of power cut or in case of crash of the OS.
If you want to force a real "synchronous" approach, ZFS will write the file in the layer 2, what we call ZIL, before writing to disks. Because of his speed, this is where an nvme disk has a perfect match. Despite this is not the final destination for the files, they are safe is case of power cut or crash.
Since the Layer 2 cache receive every 5 seconds information coming from the RAM at the speed of the network interface, we can size the ZIL like this:
2.5Gbps x 5sec / 8 = 1.5G
In case I install a card of 10Gbps, I'll size the ZIL with 8GB.
Size of the cache:
In the other direction, ZFS has another caching called "log". We use quick disks to store files users are regularly asking. ZFS decides which files can goes in this zone. I did not find a clear rule to size it correctly. It depends of the size of the files your have and the Ethernet throughput you have.
In my case I will size it to 64GB
To have a kind of resilience, but having a good performance, I've decided to use the raid-Z1 disk's organisation. I let you check the perfect explanation made by Victor Mendon about the different disk's organisation and their impact on resilience and performance.
Installation
To perform this specific installation, I boot an USB key with the standard installation image. Nevertheless, I leave the installation screen to start a shell session
Since I have a X64 machine, I download from FreeBSD installation media the img for amd64.
To put it on an USB key, I do:
obsd:~ $ doas dd if=FreeBSD-14.1-RELEASE-amd64-mini-memstick.img of=/dev/rsd2c bs=1M
Be very careful to select the correct disk for you specific case.
Once you have booted the FreeBSD installer, leave to shell session and type the following commands:
Set the keyboard mapping to my needs (Belgian in my case)
kbdmap <be>
To check that we are well with an EFI board:
sysctl machdep.bootmethod
Slice the nvme disk (called /dev/nda0 in my case):
gpart destroy -F /dev/nda0
gpart create -s gpt /dev/nda0
gpart add -t efi -s 200M -l efi0 /dev/nda0
gpart add -t freebsd-swap -a 4k -s 48G -l swap0 /dev/nda0
gpart add -t freebsd-zfs -a 4k -s 8G -l zil0 /dev/nda0
gpart add -t freebsd-zfs -a 4k -s 64G -l l2arc0 /dev/nda0
gpart add -t freebsd-zfs -a 4k -l os0 /dev/nda0
Format the EFI file system:
newfs_msdos /dev/nda0p1
To be able to create subdirectory of /mnt and, later, to save network parameters on this USB key:
mount -uw /
Create the ZFS pool, datasets and inform which one if the bootfs:
zpool create -f -o ashit=12 -o altroot=/mnt -O mountpoint=none -O atime=off -O compression=lz4 rpool /dev/gpt/os0
zfs create -o mountpoint=none rpool/ROOT
zfs create -o mountpoint=/ rpool/ROOT/default
zfs create -o compression=off -o monutpoint=none rpool/usr
zfs create -o compression=off -o mountpoint=none rpool/var
zfs create -o compression=off -o exec=on -o setuid=off -o mountpoint=/tmp rpool/tmp
zfs create -o exec=off -o setuid=off -o mountpoint=/var/crash rpool/var/crash
zfs create -o exec=off -o setuid=off -o mountpoint=/var/log rpool/var/log
zfs create -o atime=on -o exec=off -o setuid=off -o mountpoint=/var/mail rpool/var/mail
zfs create -o compression=off -o exec=on -o setuid=off -o mountpoint=/var/tmp rpool/var/tmp
zfs create -o mountpoint=/home rpool/home
zfs create -o mountpoint=/home/vi rpool/home/vi
chmod 1777 /mnt/tmp
chmod 1777 /mnt/var/tmp
zpool set bootfs=rpool/ROOT/default rpool
Configure the network and download base.txz and kernel.txz:
bsdconfig <network>
cd /mnt
fetch https://download.freebsd.org/ftp/releases/amd64/ base.txz
fetch https://download.freebsd.org/ftp/releases/amd64/ kernel.txz
Install FreeBSD:
tar -C /mnt -xpjf base.txz
tar -C /mnt -xpjf kernel.txz
Setup EFI:
mkdir -p /mnt/boot/efi
mount -t msdosfs /dev/nda0p1 /mnt/boot/efi
mkdir -p /mnt/boot/efi/EFI/BOOT
cp /boot/loader.efi /mnt/boot/efi/EFI/BOOT/BOOTX64.EFI
Minimal config of our new system:
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
To setup the root password, activate the swap and define the hostname, we start a chroot session:
chroot /mnt
bsdconfig <passwd>, <timezone>
echo "/dev/nda0p2 none swap sw 0 0" >> /etc/fstab
echo 'hostname="fbsd-nas"' >> /etc/rc.conf
echo 'vfs.zfs.min_auto_ashift=12' >> /etc/sysctl.conf
exit
The system is now fully installed ;)
reboot
We have still few config to perform:
kbdmap <be>
bsdconfig <network>
freebsd-update fetch
freebsd-update install
pkg install ncurses
pkg install terminfo-db
Create the zpool and dataset for the NAS:
mkdir /nas
zpool create -o ashift=12 -O mountpoint=none -O compression=lz4 -O atime=off naspool raidz1 /dev/ada0 /dev/ada1 /dev/ada2 /dev/ada3
zpool add naspool log /dev/gpt/zil0
zpool add naspool cache /dev/gpt/l2arc0
zfs create -o exec=off -o setuid=off -o mountpoint=/nas/sharevar/log rpool/var/log
Conclusion
I have a working machine. Different file systems / datasets seams to have the correct behaviour.
I can perform the required snapshots. I think not too far from the standard FreeBSD install.
Feel free to react if you find something incorrect ;).
It's my first weeks with FreeBSD, so I have maybe missed some important elements.