Introduction

This method of building thick jails must be preferred to the old method presented

1. The one rule: match the host kernel

A jail shares the host's kernel, so the jail's userland should never be newer than the kernel running underneath it.

pkgbase offers two kinds of repositories:

• base_latest follows the stable branch (for example stable/15). Its userland moves ahead of a release kernel.
• base_release_N follows the N-th minor release (for example base_release_1 for 15.1) and receives only patch updates.

My first attempt used base_latest on a host running 15.1-RELEASE-p4. Inside the jail, freebsd-version -ru showed 15.1-RELEASE-p4 for the kernel and 15.1-STABLE for the userland. Everything seemed to work, but the jail would not stop cleanly: rc.shutdown waited forever on a process that had already exited, until its 90 second watchdog killed it.

After rebuilding the jail on base_release_1, the problem disappeared and stop and restart worked. I did not isolate which package caused the hang, so I can't prove the mismatch is the only cause, but it was the one change that fixed it. Use the release repo that matches your host.

You can see which repos your host knows about with pkg(8):

pkg repos

On a release system, the base repo is listed there (disabled by default) with the correct URL and key path.

2. Create the dataset

Each jail gets its own dataset, created with zfs-create(8), so it can be snapshotted and cloned independently.

zfs create rpool/vm/jls/fbsd15
J=/vm/jls/fbsd15
mkdir -p $J/usr/local/etc/pkg/repos $J/var/db/pkg $J/usr/share

The J variable holds the jail's root path and is reused throughout the post. Keep the same terminal session open, or set it again.

3. Configure the base repository inside the jail

pkg reads repository definitions from a directory (see pkg.conf(5)), so we create one inside the jail's tree and point pkg at it with --repo-conf-dir.

cat > $J/usr/local/etc/pkg/repos/FreeBSD-base.conf <<'EOF'
FreeBSD-base: {
  url: "pkg+https://pkg.FreeBSD.org/${ABI}/base_release_1",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkgbase-15",
  enabled: yes
}
EOF

cp -a /usr/share/keys $J/usr/share/

Two details matter here:

• The fingerprints directory is pkgbase-15, not the regular pkg one. Check what your host's pkg repos output says.
• The keys are copied into the jail because pkg --rootdir resolves the fingerprint path inside the target root.

4. Install the base system

I use a small shell function so I do not repeat the long pkg options (--rootdir and --repo-conf-dir are described in pkg(8), and installation itself in pkg-install(8)). A function is used rather than a variable because zsh does not word-split variables, so $PKG install ... fails there.

pkg --rootdir $J --repo-conf-dir $J/usr/local/etc/pkg/repos -o IGNORE_OSVERSION=yes \ 
         -o ASSUME_ALWAYS_YES=yes update
pkg --rootdir $J --repo-conf-dir $J/usr/local/etc/pkg/repos -o IGNORE_OSVERSION=yes \
         install -y FreeBSD-set-minimal-jail FreeBSD-ssh FreeBSD-csh

IGNORE_OSVERSION=yes stops pkg from complaining that the packages were built for a slightly different __FreeBSD_version than the running kernel.

Which packages, and why

FreeBSD-set-minimal-jail is a metapackage for a basic multi-user jail. It installs around 47 packages and about 122 MiB: the runtime, rc, syslogd, cron, OpenSSL, the C libraries, utilities, vi and so on. It has no kernel, no bootloader and no firmware.

On top of that I add:

• FreeBSD-ssh for sshd, because the minimal set does not include it.
• FreeBSD-csh, optional and tiny (about 1 MiB together).

What not to install

It is tempting to install FreeBSD-set-base-jail as a "fuller" base. Don't, if you want a small jail. In my test it pulled in 142 extra packages and 505 MiB: the whole compiler toolchain (clang, lld, lldb), every -dev package, bhyve, kyua, Kerberos, Bluetooth, sound, games, ppp and sendmail. It depends on the devel and optional-jail sets.

If you need something extra later, find it by name with pkg-search(8) and install it alone:

pkg search -x '^FreeBSD-'

5. Basic jail configuration

These steps edit files inside the jail from the host.

cp /etc/resolv.conf $J/etc/resolv.conf
cp /etc/localtime   $J/etc/localtime

sysrc -R $J hostname="fbsd15"
sysrc -R $J sendmail_enable="NONE"
sysrc -R $J syslogd_flags="-ss"
sysrc -R $J sshd_enable="YES"

sysrc -R edits rc.conf under an alternate root. syslogd_flags="-ss" makes syslogd(8) not listen on the network, which is the safer default in a jail.

For SSH access with a root password (acceptable for a lab, not for anything reachable from outside), append to sshd_config(5):

echo 'PermitRootLogin yes' >> $J/etc/ssh/sshd_config

For anything longer lived, create a normal user in wheel with an SSH key instead.

6. The jail definition

Create /etc/jail.conf.d/fbsd15.conf (the syntax is documented in jail.conf(5)):

cat > /etc/jail.conf.d/fbsd15.conf <<'EOF'
fbsd15 {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";
  exec.timeout = 60;
  stop.timeout = 30;
  persist;

  # PERMISSIONS
  allow.raw_sockets;      # for ping
  exec.clean;
  mount.devfs;
  devfs_ruleset = 4;

  # HOSTNAME/PATH
  host.hostname = "${name}";
  path = "/vm/jls/${name}";

  # NETWORK
  ip4.addr = 192.168.2.4;
  interface = igc1;
}
EOF

Notes on the choices:

• exec.consolelog must be an absolute path. jail(8) does not expand ~.
• devfs_ruleset = 4 applies the standard restricted device set to the jail's /dev (see devfs(8) and devfs.rules(5)).
• exec.timeout and stop.timeout make a stuck stop get killed after about a minute instead of hanging forever.
• allow.raw_sockets is only needed for ping and traceroute. Remove it if you don't need them.
• persist keeps the jail alive even with no processes in it. The side effect is that a failed stop leaves the jail in place (see the troubleshooting section).

7. Enable, start and bootstrap

On the host machine:

sysrc jail_enable="YES"
sysrc jail_list | grep -qw fbsd15 || sysrc jail_list+="fbsd15"

service jail start fbsd15
jexec fbsd15 pkg bootstrap -y
jexec fbsd15 passwd root

service(8) starts the jail through its rc script, and jexec(8) runs commands inside it. The base packages are managed by the FreeBSD-base repository, but the pkg binary itself comes from the FreeBSD-pkg-bootstrap package, which only bootstraps the real tool. pkg bootstrap fetches pkg from the normal ports repository, so third-party packages work inside the jail afterwards (jexec fbsd15 pkg install ...).

8. Verify

jexec fbsd15 freebsd-version -ru      # both lines should read 15.1-RELEASE-pN
zfs list rpool/vm/jls/fbsd15          # roughly 150-250M
jexec fbsd15 service sshd restart
time timeout 30 jexec fbsd15 service sshd stop    # about 1 second
jexec fbsd15 service sshd start

The version check is the important one. If the userland line says STABLE while the host is on a release, you are on the wrong repo.

Then run the real test, a full restart cycle, and list the running jails with jls(8):

service jail restart fbsd15
jls

It should stop and start without hanging. Once it does, take a baseline snapshot with zfs-snapshot(8):

zfs snapshot rpool/vm/jls/fbsd15@base-minimal

9. Updating the jail

Always snapshot first, so a bad upgrade is a one-command rollback:

zfs snapshot rpool/vm/jls/fbsd15@pre-upgrade
pkg -r /vm/jls/fbsd15 -o IGNORE_OSVERSION=yes upgrade -r FreeBSD-base
service jail restart fbsd15

# if something went wrong:
# zfs rollback rpool/vm/jls/fbsd15@pre-upgrade

A few rules for upgrades:

• Never run freebsd-update in a pkgbase jail. All base updates go through pkg.
• Update the host first (and reboot onto the new kernel), then the jail. The jail userland must not get ahead of the host kernel.
• After upgrading, look for *.pkgsave files in the jail's /etc, which mean a configuration file was replaced and your modified version was saved.
• To move to the next minor release (for example 15.2), change base_release_1 to base_release_2 in the jail's repo file, but only once the host kernel is on 15.2.

10. Cloning

The snapshot taken earlier works as a template, using zfs-clone(8):

zfs clone rpool/vm/jls/fbsd15@base-minimal rpool/vm/jls/newjail

Then adjust the hostname (sysrc -R), the IP address and a new jail config file.

Troubleshooting

The jail hangs when stopping

Symptom: service jail restart waits, then prints something like rc.shutdown: exited on signal 9. The console log (/var/log/jail_console_<name>.log) shows rc.shutdown waiting on a service until its 90 second watchdog expires. In my case it was sshd, which had already exited:

Stopping sshd.
Waiting for PIDS: 11791
90 second watchdog timeout expired. Shutdown terminated.

Useful commands while it is stuck, from the host (jls(8), ps(1), and jail -R from jail(8) to force-remove a jail):

jls
jexec fbsd15 ps -axo pid,stat,wchan,command
tail -50 /var/log/jail_console_fbsd15.log
jail -R fbsd15          # force-remove the jail

My fix was moving from base_latest to base_release_1 (section 1). Check freebsd-version -ru first.

zfs destroy says "dataset is busy"

After a failed stop, the jail's devfs mount stays behind and keeps the dataset busy. Unmount the child first with umount(8):

mount | grep fbsd15
umount /vm/jls/fbsd15/dev
zfs destroy -r rpool/vm/jls/fbsd15

If it still refuses, check fstat -f /vm/jls/fbsd15 for processes, such as a shell whose working directory is inside the jail.

Conclusion

• Use pkgbase with FreeBSD-set-minimal-jail plus FreeBSD-ssh for a small, working thick jail.
• Match the repo to your host: base_release_N on a release, with the pkgbase-15 key directory.
• Avoid FreeBSD-set-base-jail unless you want the compiler toolchain and every -dev package.
• Snapshot before every upgrade, update the host first, and never use freebsd-update on a pkgbase jail.