Active/active DHCP on OpenBSD with dhcpd lease sync

I run two OpenBSD servers that both serve DHCP on the same network. Both are active at the same time, and they replicate their leases to each other with the synchronisation built into dhcpd(8). If one server goes down, the other already knows every lease, so clients renew without getting a new address and without address conflicts. This post describes my setup. Adapt names, addresses and ranges to your network. I checked the details against the OpenBSD manual pages and the current dhcpd source, so they may differ slightly on other releases.

Contents

1. Goal and architecture
2. How dhcpd synchronisation works
3. Configuration
4. Testing
5. If it does not work
6. Two active servers: what to expect
7. Limits and notes

Goal and architecture

|     Item                 |    Value       |
|--------------------------|----------------|
| Server 1 (`fw1`)         | `192.168.1.31` |
| Server 2 (`fw2`)         | `192.168.1.32` |
| Client network interface | `vlan20`       |

Both servers run dhcpd and both answer clients, so this is an actif/actif layout. Whenever a server grants or renews a lease, it sends that lease to its peer, so the two lease databases stay in step.

          clients (vlan20)
            /          \
  192.168.1.31  <--->  192.168.1.32
   dhcpd        sync     dhcpd
        (UDP, port of "dhcpd-sync")

How dhcpd synchronisation works

Synchronisation is controlled by two options of dhcpd(8):

|   Option        |    Meaning                                                         |
|-----------------|--------------------------------------------------------------------|
| `-Y synctarget` | **Send** lease updates to this target. May be given several times. |
| `-y synclisten` | **Listen** for lease updates. May be given only once.              |

A target is an IPv4 address (unicast) or a network interface name with an optional :TTL (multicast to the group 224.0.1.240). I use unicast: each server sends to the other server's IP address. This is simple, and easy to filter in pf.

The UDP port is not hard-coded. dhcpd looks up the service dhcpd-sync in services(5). On my machines this is 8067, so check /etc/services on yours.

A few details Claude founds by reading the source codes (usr.sbin/dhcpd/sync.c, dhcp.c, memory.c):

• A lease is sent when a client's DHCPREQUEST is acknowledged, renewals included. DISCOVER/OFFER does not trigger a sync.
• There is no bulk transfer at startup. A freshly started server catches up as clients renew, and when the lease file is periodically rewritten.
• The message carries the IP, the hardware address and the lease times. A lease received through sync therefore appears in the peer's dhcpd.leases(5) without a uid line. This is a handy way to tell "granted locally" from "received via sync".
• Messages are authenticated with HMAC-SHA1, using the checksum of /var/db/dhcpd.key as the key. They are not encrypted. If the file does not exist, dhcpd uses an empty key and sync still works, just unauthenticated.

Configuration

1. Shared key

The man page suggests creating the key with dd(1). The content can be anything, since dhcpd hashes it, but the file must be identical on both servers, so I copy it to the peer with scp). dhcpd reads it before it drops privileges, so it can be readable by root only.

dd if=/dev/random of=/var/db/dhcpd.key bs=2048 count=1
chmod 600 /var/db/dhcpd.key
scp /var/db/dhcpd.key root@192.168.1.32:/var/db/dhcpd.key

Compare it with sha256(1) on both servers. If the keys differ, or only one server has a key, the receiver drops every packet. As far as I can tell from the source, it does so without any log message.

2. /etc/dhcpd.conf

dhcpd.conf(5) must be identical on both servers, as required by the dhcpd(8) man page. A minimal example:

option domain-name "example.lan";
option domain-name-servers 192.168.1.1;

subnet 192.168.1.0 netmask 255.255.255.0 {
    option routers 192.168.1.1;
    range 192.168.1.100 192.168.1.200;
}

I did not set server-identifier. dhcpd.conf(5) says its use is not recommended unless the default is wrong for your network, and with two active servers each one should identify itself.

3. /etc/rc.conf.local

The sync flags go into dhcpd_flags, together with the interface that dhcpd serves clients on (see rc.conf). Each server sends to the other server and listens on vlan20.

On fw1 (192.168.1.31):

dhcpd_flags="-Y 192.168.1.32 -y vlan20 vlan20"

On fw2 (192.168.1.32):

dhcpd_flags="-Y 192.168.1.31 -y vlan20 vlan20"

Restart with rcctl:

rcctl restart dhcpd

Remarks:

• -y vlan20 makes dhcpd listen on the sync port. Its socket accepts the unicast packets sent by the peer.
• The interface needs an IPv4 address, otherwise dhcpd exits at startup with sync init.
• A server ignores sync packets whose source is its own address, so the two servers need distinct addresses on vlan20.

4. /etc/pf.conf

The sync traffic is ordinary UDP, so pf.conf has to let it through between the two servers:

peers = "{ 192.168.1.31 192.168.1.32 }"
pass quick on vlan20 proto udp from $peers to $peers port 8067

Use the port from your /etc/services, then reload with pfctl -f /etc/pf.conf (pfctl(8)). In my setup client traffic needs no rule: dhcpd talks to clients through bpf, which is why no listener on port 67 shows up in netstat.

Testing

1. Daemons and logs. On both servers, check the daemon with rcctl and read the log with tail:

rcctl check dhcpd
tail -n 30 /var/log/daemon

2. Watch the sync packets. Capture with tcpdump on the VLAN interface. Frames on the physical parent port carry an 802.1Q tag, so a plain udp port filter on the parent does not match (see vlan(4)).

tcpdump -n -e -ttt -i vlan20 udp port 8067

Renew a lease on a test client. The server that answered sends one UDP packet with 100 bytes of payload to its peer.

3. Compare the lease databases. On both servers, look at the entry with grep:

grep -A8 "<client IP>" /var/db/dhcpd.leases

The newest entry must exist on both, with the same MAC and the same starts and ends. On the server that did not answer, the entry has no uid line. The file is append-only, so older entries remain above the newest one.

4. Test both directions. Make each server answer a renewal in turn, and check that the other one receives it. For example, stop dhcpd on one server with rcctl stop dhcpd, renew a client, start it again, and compare the lease files.

5. Test the failover itself. With one server stopped, a client that renews should keep the same IP address, because the remaining server already knows its lease. A new client should get a different address, with no conflict.

If it does not work

These are the checks that mattered for me:

• sending sync message failed: Permission denied in /var/log/daemon: pf dropped the packet on the sending server. Add or fix the rule from section 4. (With multicast targets, the message reads sending multicast sync message failed.)
• No error, but leases do not arrive: compare sha256 /var/db/dhcpd.key on both servers, then compare the dhcpd.conf files.
• Nothing in pflog0: a log rule only logs the packet that creates the state, so later sync packets do not show up there. pfctl -vsr shows the packet counters of the rule and pfctl -ss | grep 8067 shows the state (pflog(4)). With working sync the packet counters on both servers were equal.
• tcpdump on the physical port shows nothing: capture on the VLAN interface, as in the testing section.

A note on multicast

Instead of an IP address, -Y also accepts an interface name, for example -Y vlan20 -y vlan20 vlan20. That sends to the multicast group 224.0.1.240, which dhcpd(8) documents, and it avoids listing the peers. On my machines it did not work: route showed that 224.0.0.0/4 is routed to lo0 with the REJECT flag, and I did not find a place in sync.c that selects the outgoing multicast interface. I did not investigate further and went with unicast, which I recommend for two servers. If you want multicast, check route -n get 224.0.1.240 first.

Two active servers: what to expect

Both servers receive every client broadcast and both can answer. What I know from the source: each server may send an OFFER from its own copy of the pool, and offers are not synchronised. The client's REQUEST carries the identifier of the server it picked, and a server ignores a request addressed to another server. The chosen server then acknowledges, and that acknowledgement is what gets synchronised.

Renewals work the same way. Whichever server answers sends the updated lease to its peer, so after a renewal both databases hold the same lease. Keep the two dhcpd.conf files identical and the clocks in sync. I have not stress-tested the case where both servers answer under heavy load, so verify the behaviour in your own environment.

Conclusion

• This is lease replication, not a failover protocol: dhcpd has no pool splitting or load balancing. Both servers use the same configuration and the same range.
• dhcpd must be restarted after a configuration change, see dhcpd(8). Keep the two dhcpd.conf files identical yourself, for example with scp or a small deploy script.
• Lease times in the sync messages are absolute timestamps, so keep both clocks accurate with ntpd(8).
• Details of the sync flags can change between releases. Check man dhcpd on your version.