Thank you for visiting!
My little window on internet allowing me to share several of my passions
Categories:
- FreeBSD
- OpenBSD
- VM
- High Availability
- vdcron
- My Sysupgrade
- FreeBSD
- Nas
- DragonflyBSD
- fapws
- Alpine Linux
- Openbox
- Desktop
- Security
- nvim
- yabitrot
- nmctl
- Tint2
- Firewall
- VPN
- Project Management
- Hifi
- Alarm
Most Popular Articles:
Last Articles:

Which platform is the best to run FreeBSD VM ? Linux/KVM or FreeBSD/bhyve ?
Posted on 2025-02-02 18:14:00 from Vincent in FreeBSD VM
I would invest time and energy on the setup of VM, but on which platform ? Since +15 years, I like BSD systems, but I hear more and more persons promoting Linux/KVM platforms. Should I re-invest (I started with Linux before 2000) my time on Linux ecosystem ? Lot of elements must be taken into account to answer such question, but in this blog I'll focus on the performance aspects. More specifically, I'll compare Linux/KVM and FreeBSD/bhyve. And I'll use tests representative for me which are mainly compute and filesystem. The network part is also important, but the machine I will use for such study is not a good one for that.
Update February 9th 2025: I have added a comparison between virtio-blk and nvme
Introduction
I conducted a performance comparison of virtual machines (VMs) running on Linux and FreeBSD using a Lenovo T14s laptop with an AMD Ryzen Pro 7 processor.
The tests were performed in december and January 2025 using the latest available versions of the host operating systems: Fedora 41 and GhostBSD 24.10.1 (based on FreeBSD 14.1). Both operating systems were installed on the same hardware in a dual-boot configuration.
Each VM was configured with the following specifications:
- 2 vCPUs
- 4096 MB RAM
- FreeBSD 14.2 installed with ZFS
The tests focused on three performance categories:
- Filesystem performance (using iozone)
- Cryptographic performance (using the Phoronix/OpenBenchMark OpenSSL test suite)
- General database performance (using SQLite operations: insert, select, update, delete)
When doing statistical analysis, I always remember my professor of university which had repeated over and over that we cannot conclude based on the only average values.
Unfortunately most of the time persons are doing this fault.
For example, OpenBecnhMark is sometime doing 3 measures but only present the average values.
So,in this study, for the SQLite test I've written, I'll restore the good old habits we had at university.
So, I'll use the Wilcoxon test to make sure that 1 set of measures is significantly better or worse that the other. Then I will present the average and standard deviatoins results.
This blog is a bit long, but it shows how I've build the conclusion.
VM Setup
Fedora Host Setup
In very short, here after what I did:
- Installed Fedora with the default configuration (BTRFS as the filesystem).
- Installed "Virtual Machine Manager" via the Software application.
- Downloaded the FreeBSD 14.2-RELEASE-amd64-mini-memdisk.img file.
- Configured the VM with:
- 2 vCPUs
- 4096 MB RAM
- 20 GB virtual disk
- In "Virtual Machine Manager," the following parameters were observed:
- Hypervisor: KVM
- Emulator:
/usr/bin/qemu-system-x86_64
- Chipset: i440FX
- Firmware: BIOS
The VM was installed with FreeBSD 14.2 and ZFS using default settings.
GhostBSD Host Setup
Here after a very short description of what I did for the GhostBSD system:
- Installed GhostBSD on the same disk (ZFS filesystem).
- Set up bhyve as follows:
- Installed
vm-bhyve
viapkg
. - Created a ZFS dataset for VMs:
zfs create zpool/vm -o mountpoint=/vm
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zpool/vm"
vm init
- Created a virtual switch:
vm switch create public
vm switch add public em0
- Downloaded the FreeBSD ISO:
vm iso https://download.freebsd.org/ftp/releases/amd64/ISO-IMAGES/14.2/FreeBSD-14.2-RELEASE-amd64-disc1.iso
- Created and installed the VM:
vm create fbsd14
vm install fbsd14 FreeBSD-14.2-RELEASE-amd64-disc1.iso
vm console fbsd14
- Installed
As with Fedora, the FreeBSD 14.2 VM used ZFS with default settings.
The config file becomes like this:
# cat fbsd14.conf
loader="bhyveload"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="cable"
#network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="a4afe647-da46-11ef-b70b-xxxxxx"
network0_mac="58:9c:fc:xx:xx:xx"
Filesystem Performance
Host-Level Performance
I ran the following iozone command on both Fedora and GhostBSD hosts:
./iozone -a -b results.xls
This test evaluates multiple read and write scenarios:
- Sequential write/rewrite
- Sequential read/reread
- Random read/write
- Backward read
- Record rewrite
- Stride read
- fwrite/fread and re-fwrite/re-fread
The consolidated results are shown below, highlighting the OS with higher throughput (MB/s):
Observation:
GhostBSD (FreeBSD) outperformed Fedora (Linux) in most scenarios.
VM-Level Performance
The same iozone tests were conducted on FreeBSD 14.2 VMs hosted on Fedora and GhostBSD. The results are summarized below:
Observation:
While FreeBSD VMs on Fedora had better performance in some read/write scenarios (particularly with larger files), GhostBSD-hosted VMs performed better overall.
Compute Performance
To measure compute performance, I used the Phoronix OpenSSL test suite, which evaluates cryptographic performance across algorithms like SHA, RSA, ChaCha20, and AES.
The command I've used on each system is the following:
phoronix benchmark openssl
At the time I've made the tests, the version is pts/openssl-3.3.0.
This command will download the required elements and will compile them on your system.
Host-Level Performance
The results are shown below:
Test | Fedora (MB/s) | GhostBSD (MB/s) | Delta (%)
-------------------|---------------|-----------------|------------
sha256 | 9,724.94 | 9,239.02 | -5.26%
sha512 | 3,230.32 | 3,239.61 | +0.29%
rsa4096 (sign/s) | 3,918.90 | 3,876.60 | -1.09%
rsa4096 (verify/s) | 121,447.70 | 121,808.30 | +0.30%
ChaCha20 | 36,758.02 | 24,898.97 | -47.63%
AES-128-GCM | 68,805.55 | 68,378.41 | -0.62%
AES-256-GCM | 57,671.47 | 57,008.35 | -1.16%
ChaCha20-Poly1305 | 25,961.89 | 16,860.14 | -53.98%
delta = (ghostbsd - fedora ) / ghostbsd
Observation:
Linux outperformed FreeBSD in ChaCha20 and RSA operations, while the other results were fairly close.
VM-Level Performance
The cryptographic performance on FreeBSD VMs hosted on Fedora and GhostBSD is summarized below:
| FreeBSD 14.2 | FreeBSD 14.2 |
Test | on Fedora (MB/s) | on GhostBSD (MB/s) | Delta (%)
--------------------|------------------|--------------------|------------
sha256 | 2,462.43 | 2,502.75 | +1.61%
sha512 | 1,161.20 | 1,102.44 | -5.33%
rsa4096 (sign/s) | 1,638.40 | 440.60 | -271.86%
rsa4096 (verify/s) | 42,642.20 | 28,662.10 | -48.78%
ChaCha20 | 8,297.92 | 12,052.77 | +31.15%
AES-128-GCM | 26,607.47 | 25,585.75 | -3.99%
AES 256 GCM | 23,254.96 | 22,376.64 | -3.93%
ChaCha20-Poly1305 | 5,649.20 | 7,335.60 | 22.99%
delta = (ghostbsd - fedora ) / ghostbsd
Observation:
FreeBSD on GhostBSD outperformed Fedora in ChaCha20, but Fedora hosted FreeBSD VMs were faster for RSA operations.
There is a big difference for the rsa4060 sign process. Investigation to understand such delta would be interesting. I've not yet done it.
Combined Performance Test
To combine compute and filesystem performance, I ran a custom SQLite workload:
-
Schema:
CREATE TABLE IF NOT EXISTS test_table (
id INTEGER PRIMARY KEY,
value INTEGER
)
-
Workload:
- 10M inserts
- 10M selects
- 5M updates
- 5M deletes
- Each operation included a commit to stress the filesystem.
We plan to execute this cycle 20x. For those willing to re-use it, the script is here
The results were analyzed using statistical significance Wilcoxon test based on 20 measures for each tests are here under; for both the host and the VM.
In short note, I'm using the scipy.stats and the numpy modules of python for that.
scypi.stats allowmes to compute the "wilcoxon" P-Value. If this P-Value is bigger than 5%, this means that 2 measures are similar.
Thanks to numpy, I'm allowed to compute sum of delta value bigger than 0 and lower than 0.
I've done this like this:
np_sample1 = np.array(results_1)
np_sample2 = np.array(results_2)
difference = nb_sample_1 - np_Sample2
negative_ranks = np.sum(diffference < 0)
positive_ranks = np.sum(diffference > 0)
Wilcoxon says that if 2 samples have P-Value lower than 5%, then if negative_ranks > positive_ranks, then results_2 has globally bigger value than results_1. And the opposite if negative_ranks < positive_ranks.
You can retreive those computate values here under. I've added the average and standard deviation too.
Host-Level Results
Results are:
# insert
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 8.127837526798249
stdev on fedora: 0.06555929010149855
average on ghostbsd: 9.713736152648925
stdev on ghostbsd: 0.09923602826382205
ghostbsd has globally bigger values.
# select
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 1.8147160768508912
stdev on fedora: 0.018330392859064907
average on ghostbsd: 2.396692383289337
stdev on ghostbsd: 0.01860527693325904
ghostbsd has globally bigger values.
# update
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 25.35244983434677
stdev on fedora: 0.14021230011898583
average on ghostbsd: 21.084751880168916
stdev on ghostbsd: 0.15135644039599663
fedora has globally bigger values.
# delete
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 20.171697294712068
stdev on fedora: 0.1489675535938781
average on ghostbsd: 17.545048415660858
stdev on ghostbsd: 0.18618484257386217
fedora has globally bigger values.
We can see that Fedora is a bit slower than GhostBSD for "update" and "delete"; but the opposite for "insert" and "select"
- Insert: Fedora was faster.
- Select: Fedora was faster.
- Update: GhostBSD was faster.
- Delete: GhostBSD was faster.
We can see that Standard Deviation are not high. Which is a good point.
VM-Level Results
Note: I remind that figures here bellown are made based on default.conf file. So in other words, the disk_model was virtio-blk
# insert
P-Value: 1.9073486328125e-05
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 9.734229171276093
stdev on fedora: 0.1427924079712167
average on ghostbsd: 9.870490086078643
stdev on ghostbsd: 0.21093650631388258
ghostbsd has globally bigger values.
# select
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 2.425950753688812
stdev on fedora: 0.031016916121464233
average on ghostbsd: 2.671954321861267
stdev on ghostbsd: 0.13410963051661903
ghostbsd has globally bigger values.
# update
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 27.21637978553772
stdev on fedora: 0.7215367984511372
average on ghostbsd: 29.348756504058837
stdev on ghostbsd: 0.9767300578579408
ghostbsd has globally bigger values.
# delete
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average on fedora: 21.590943467617034
stdev on fedora: 0.41979261218858643
average on ghostbsd: 22.82505156993866
stdev on ghostbsd: 0.6559995691465991
ghostbsd has globally bigger values.
Fedora-hosted FreeBSD VMs showed slightly better performance overall, but the differences were minimal.
Here too Standard Deviation are not high.
Alternative tests
You can find an altenative comparaison made by Stefano Marinelli.
His objective was to compare Proxmox and FreeBSD/bhyve thanks to sysbench results.
Let's investigate the remarks made by Stefano concerning the disk_model
Let's perform a comparison between same VM, but with disk_type=virtio-blk and disk_type=nvme
Compare the results between virtio-blk and nvme
In /vm/fbsd14/fbsd14.conf, I've change the line 'disk0_type="virtio-blk"' by 'disk0_type="nvme"'.
By comparing the 2 dmesg's output created when the VM boot, I have this:
diff dmesg_nvme dmesg_virtio
58,60c58,62
< nvme0: <Generic NVMe Device> mem 0xc0000000-0xc0003fff,0xc0004000-0xc0005fff at device 4.0 on pci0
< virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x2000-0x203f mem 0xc0006000-0xc0007fff irq 16 at device 5.0 on pci0
< vtnet0: <VirtIO Networking Adapter> on virtio_pci0
---
> virtio_pci0: <VirtIO PCI (legacy) Block adapter> port 0x2000-0x207f mem 0xc0000000-0xc0001fff irq 16 at device 4.0 on pci0
> vtblk0: <VirtIO Block Adapter> on virtio_pci0
> vtblk0: 10240MB (20971520 512 byte sectors)
> virtio_pci1: <VirtIO PCI (legacy) Network adapter> port 0x2080-0x20bf mem 0xc0002000-0xc0003fff irq 17 at device 5.0 on pci0
> vtnet0: <VirtIO Networking Adapter> on virtio_pci1
87,89d88
< nvme0: temperature threshold not supported
< nvme0: SET_FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000008 cdw11:00000000
< nvme0: FEATURE NOT CHANGEABLE (01/0e) crd:0 m:0 dnr:0 p:1 sqid:0 cid:9 cdw0:0
91,95d89
< nda0 at nvme0 bus 0 scbus0 target 0 lun 1
< nda0: <bhyve-NVMe 1.0 NVME-4-0>
< nda0: Serial Number NVME-4-0
< nda0: nvme version 1.4
< nda0: 10240MB (20971520 512 byte sectors)
Let's run our test_sqlite.py script on this FreeBSD VM booted with nvme disk_type and compare his results with the results already obtained with default's disk_type.
# insert
Test Statistic: 85.0
P-Value: 0.47490501403808594
Fail to reject the null hypothesis: There is no significant difference between the systems.
Sample size: 20
average virtio-blk: 9.870490086078643
stdev virtio-blk: 0.21093650631388258
average nvme: 9.887179255485535
stdev nvme: 0.08201595318848469
# select
Test Statistic: 30.0
P-Value: 0.00365447998046875
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average virtio-blk: 2.671954321861267
stdev virtio-blk: 0.13410963051661903
average nvme: 2.6247466087341307
stdev nvme: 0.12543457354863238
virtio-blk is slower than nvme
# update
Test Statistic: 1.0
P-Value: 3.814697265625e-06
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average virtio-blk: 29.348756504058837
stdev virtio-blk: 0.9767300578579408
average nvme: 27.86021978855133
stdev nvme: 0.37119499183363347
virtio-blk is slower than nvme
# delete
Test Statistic: 5.0
P-Value: 1.9073486328125e-05
Reject the null hypothesis: There is a significant difference between the systems.
Sample size: 20
average virtio-blk: 22.82505156993866
stdev virtio-blk: 0.6559995691465991
average nvme: 21.99110244512558
stdev nvme: 0.23639644985291663
virtio-blk is slower than nvme
We can see that, except inserts, virtio-blk-blk is always slower than nvme.
On "select", thus read only, the Standard Deviation of both methods are quite similar. At contrario, once you write on disk, the standard deviation of virtio-blk-blk is 3x the one of nvme.
Nvme is thus a much better disk_type for bhyve disks
Conclusion
Setting up and conducting these tests took significant effort, but the results show that FreeBSD VMs perform similarly on Linux and FreeBSD hosts. The performance differences are not substantial enough to justify switching from one platform to another based solely on this aspect.
Thanks to the extra comparison, we see that "nvme" disk_type has slightly better results than virtio-blk-blk,but most importantly his Standard Deviation is much better. So I would prefer "nvme" disk_type
These findings differ from what is often reported in Phoronix benchmarks, raising questions about the specific configurations used there.
Amazing enough, this study came to the same conclusion as Stefano Marinelli.
Ultimately, the decision to use Linux or FreeBSD as a hypervisor should consider factors beyond pure performance, such as familiarity, ecosystem, and administrative preferences.
For my personnel cases, FreeBSD will be used to host my VMs.