Vincent's Blog

Here after Claude explains what he did.

Summary

Yes, a Zig program can compete. zz writes standard ZIP files and was the fastest ZIP tool in every test against Info-ZIP and 7-Zip. On many-file trees it also beat tar+pigz, and it did so on a real NVMe disk too.

• Creating, default level (warm cache): Linux kernel source (86k files) in 1.43 s, versus 3.6 s for tar+pigz, 13.0 s for 7-Zip (ZIP output) and 21.8 s for Info-ZIP zip.
• Extracting: the same tree in 0.59 s, versus 2.2 s (tar+pigz), 5.4 s (7-Zip) and 6.9 s (unzip).
• Same size as 7-Zip, much faster: at its densest level (-12), zz matches 7-Zip's best ZIP output within 0.1 point. It takes 22 s on a 1 GB file where 7-Zip takes 394 s.
• Cold cache on NVMe: the kernel tree is archived in 2.6 s, versus 14.2 s for 7-Zip and 46.5 s for Info-ZIP. Keeping 16 reads in flight is the main reason.
• Interoperable: every zz archive passed unzip -t and 7z t, and zz extracts archives made by those tools.
• Portable: zz builds and runs on Linux, FreeBSD and OpenBSD. The benchmarks below were run on Linux only.

zz does not win everywhere. 7-Zip produces smaller files at the same nominal level. tar.gz is smaller on many small files, because it compresses everything as one stream. And extracting a single huge file cannot be parallelised in the Deflate format.

What we built

zz is about 2,200 lines of Zig: a command-line tool with four commands. zz c creates, zz x extracts, zz t tests and zz l lists. It writes standard ZIP files (Deflate or Store, ZIP64 when needed), so any unzip tool can open them. For raw compression speed it relies on C libraries that Zig links directly without wrappers: libdeflate for nearly all the work, plus zlib-ng and zlib for streaming large members. zlib-ng is optional.

The same code builds with Zig 0.13 and Zig 0.15.2, through a small compatibility file, and runs on Linux, FreeBSD and OpenBSD. On the BSDs, libdeflate and zlib-ng come from packages and zlib from the base system. OpenBSD does not package zlib-ng, so zz is built there without it and uses the base zlib for streaming.

The speed comes from five design choices:

1. Every core busy. Small files are compressed in parallel, largest first. Each worker reserves its slot in the archive with a single atomic add and writes with one pwritev, so there are no locks.
2. Big files split across cores. Files over 8 MiB are cut into 2 MiB chunks. One queue spans all big files, so threads flow from one file into the next without waiting.
3. libdeflate for every chunk. libdeflate is faster and denser than zlib, but it can only produce complete streams. A 335-line Deflate block scanner in pure Zig rewrites each chunk's ending so the chunks join into one valid stream. If the scanner ever failed its own check, the chunk would be stored uncompressed instead, so a scanner bug cannot corrupt an archive.
4. Incompressible data detected early. A 2 MiB sample is compressed first. Data that won't shrink (media, already-compressed files) is stored as-is without wasting CPU on it.
5. Plain reads and writes, no mmap. Memory-mapped I/O was 3× slower on a cold ZFS disk, so zz uses pread/pwrite with one request in flight per thread.

[embedded content: zz create pipeline · 2 paths, 16 threads each]

Both paths keep all 16 threads busy. Only the big-file path needs the scanner, because there one file is split across many workers.

On the safety side, extraction refuses .. paths, never writes through symlinks, always verifies CRC-32 and bounds every header field. The archive parser was mutation-fuzzed with a bounds-checked build: 6,000 runs, no crashes.

How we tested

All tests ran on one laptop: AMD Ryzen 7 PRO 7840U (8 cores, 16 threads), 29 GiB of RAM, an SK Hynix NVMe SSD and Void Linux. Timings are hyperfine means of 3 runs. Every multi-threaded tool was given 16 threads.

Levels were mapped 1, 6, 9 for zz, pigz and zip to 7-Zip's 1, 5, 9. Unless stated otherwise, results are at the default level (zz 6, 7-Zip 5).

Correctness was checked on every run:

• each zz archive was extracted and compared with the original,
• every zz archive was checked with unzip -t and 7z t,
• zz extracted 7-Zip's and Info-ZIP's archives,
• the Deflate scanner has unit tests and also ran through the bounds-checked build.

There were no failures.

Two portable sh scripts in the repository make these checks repeatable on other machines:

• bench/smoke.sh builds a ~135 MB test corpus that exercises every code path (small and big files, incompressible data, empty files, symlinks). It round-trips the corpus with each compression engine and compares contents, permissions, timestamps and symlinks. It also runs unzip -t when Info-ZIP is installed. It then times the main operations and writes a short markdown summary with the machine description, so results from different machines can be compared side by side. It runs on Linux, FreeBSD, OpenBSD and macOS.
• bench/bench2.sh runs the full comparison in this report: zz against 7-Zip, tar+pigz and Info-ZIP with hyperfine, at levels 1, 6 and 9, with archive checks and round-trips. Tools that are not installed are skipped. bench/report2.py turns its output into tables.

Results with a warm cache

With the data already in RAM, zz was fastest in every create and extract test. At the default level it was 2.2–46× faster than 7-Zip's ZIP mode, and 1.4–4.5× faster than tar+pigz.

Archive size is the share of the original. All times are at the default level, mean of 3 runs.

The ranking holds at other levels. At level 9 on the kernel, zz takes 3.3 s, tar+pigz 6.3 s and 7-Zip 110 s. On the 500 MB random file, zz takes 0.23 s and tar+pigz 1.2 s, while Info-ZIP and 7-Zip spend 8–10 s trying to compress it.

Results on a real disk, cold cache

On the NVMe SSD with nothing cached, zz still won every test except one tie. Its lead grew on many small files, because it keeps 16 reads in flight while the other tools read one file at a time.

[embedded content: cold-cache benchmark · hyperfine, 3 runs · ZFS on NVMe, file data uncached]

The other cold results (seconds, mean of 3 runs, every run ending with sync):

The random-file extract is a tie: zz's 0.49 s varied by ±0.14 s across runs.

Compression ratio

At its densest level (-12), zz matches 7-Zip's best ZIP output within 0.1 point, in 2.7–18× less time. At the default level, 7-Zip's ZIP files are 0.2–0.9 points smaller. Its Deflate encoder searches harder for repeated data, which costs it 9–46× more time.

Warm cache; size is the share of the original.

No ZIP tool reaches .tar.gz or .7z sizes on many small files. ZIP compresses each file separately, while tar.gz and 7z compress everything as one stream and reuse repeats across files. On the kernel, that difference is 19.9% (zz) vs 16.6% (tar.gz) vs 10.1% (.7z, a stronger compression method).

Engineering findings

The choice of language mattered less than the choices below. Each one was measured on this project.

• The compression library matters more than the language. Swapping zlib for libdeflate on chunked big files made level 6 2.5× faster (3.68 s → 1.45 s on the 1 GB file), with slightly smaller output. zlib-ng's fastest level was quicker still (0.52 s), but its output was 44% of the original vs 35% for libdeflate.
• Splitting big files costs almost nothing. Compressing in independent 2 MiB chunks, with no history shared between chunks, made the output only 0.12% bigger than one single stream. In exchange, all 16 cores work on one file. 1 MiB chunks cost 0.24%, 4 MiB chunks 0.06%.
• Parallel reads pay off on NVMe. On the cold kernel tree, cat reading files one by one took 10.5 s, while zz read and compressed everything in 2.6 s. The SSD answers 16 requests in flight much faster than 16 requests made one after another.
• Avoid mmap on ZFS. Reading through memory-mapped files was 3× slower than pread on cold data (1.04 s vs 0.34 s for 477 MB). Writing output through mmap made extract-plus-sync 3.4× slower (0.73 s vs 0.21 s on Silesia). Removing mmap brought extracting the random file from 1.19 s to 0.49 s.
• Benchmark traps. GNU tar skips reading file contents when its output is /dev/null, so a "read baseline" built that way measures nothing. And on ZFS with primarycache=metadata, reading a file in small pieces re-reads its blocks from the disk, which inflates the time for tools like tar.

Limitations

These results come from one machine and one filesystem, so read them as strong evidence, not a universal ranking.

• One laptop. It has a mobile CPU that throttles under heat. Repeated runs of the same zz test varied by up to about 30% over an hour-long session.
• The cold test penalises small reads. On the ZFS dataset used for cold runs (file data never cached), a tool reading in small pieces re-reads blocks from the SSD. tar read 5.3 GB for the 1.4 GB kernel tree, so tar+pigz's cold kernel time (23.4 s) is inflated. Info-ZIP is mildly affected (2.0 GB read). zz, 7-Zip and cat read 1.7 GB and are not affected.
• Metadata stays cached. In the cold runs, file names and directory entries stay in memory. This is cold data, not a freshly booted machine.
• Results were merged across sessions. The final zz rows were re-measured after each improvement. Tools that did not change were not re-run: 7-Zip and Info-ZIP rows (and in the cold test, tar+pigz rows too) come from an earlier run on the same setup.
• Linux only for speed. zz builds and runs on FreeBSD and OpenBSD, but this report has no timings from those systems. On OpenBSD, zz is built without zlib-ng, and the per-process data size limit (ulimit -d) may need raising for big archives, because workers keep whole members in memory.
• Not tested. No HDDs, network filesystems, ext4 or XFS, Windows or macOS, and no encrypted ZIP files (zz reports them as unsupported).

Next steps

zz is ready to evaluate as a working tool. These are the open items, ordered by value:

1. Check whether the results hold elsewhere. Run bench/smoke.sh and bench/bench2.sh on FreeBSD and OpenBSD, and repeat the cold run on ext4 or XFS (where the cache can be emptied cleanly, without the small-read penalty), on an HDD and on a desktop CPU.
2. Close the gap with 7-Zip at default levels. Levels 10–12 already reach 7-Zip's sizes but are slow. Choosing them automatically, only for the files where they gain the most, could give 7-Zip-like sizes at a fraction of the time.
3. Lower memory use. Each thread holds up to 64 MiB of buffers. Capping that total helps machines with many cores and little RAM.
4. Add missing ZIP features if anyone needs them: zstd inside ZIP (method 93), encryption, archive comments, and updating an existing archive.

Code, test and benchmark scripts and full result tables are in the zig_zip repository (got clone ssh://anon@repo.vincentdelft.be/zig_zip): bench/smoke.sh, bench/bench2.sh and bench/RESULTS.md.