Introducing misa77: A High-Velocity Decompression Codec

The Core Value Proposition: misa77 is an LZ-based codec engineered specifically for "write-once, read-many" workloads. It achieves decompression speeds 1.5x to 3x faster than LZ4 on both ARM and x86 architectures, often while maintaining superior compression ratios.

🎯 Design Philosophy & Objectives

The development of misa77 is driven by a specific set of technical priorities:

Maximum Throughput: Prioritizing extreme single-threaded decompression speed above all else.
Competitive Ratios: While it lacks an entropy backend (meaning it isn't designed to compete with zstd), it aims to match or beat LZ4 at high effort levels.
Predictable Memory Footprint:
- Compression: $\approx 5\text{ MB}$ (constant across all modes).
- Decompression: $0\text{ MB}$ overhead.

The Fundamental Trade-off

To achieve these decompression speeds, misa77 accepts a significant penalty during the encoding phase.

Interestingly, misa77 exhibits a synergistic effect: the more compressed a file is, the faster it typically decompresses. This encourages the use of high-effort compression to create streams that are more "friendly" to modern CPU pipelines.

⚙️ Compression Configuration (v0.2.0)

Users can choose between two primary effort levels:

Level	Decode Throughput	Compression Ratio	Encode Throughput
`0`	🚀 Higher	📉 Slightly Lower	↔️ Similar
`1` (Default)	📉 Slightly Lower	🚀 Higher	↔️ Similar

📊 Performance Benchmarks

Test Environment

Hardware: Intel(R) Core(TM) i7-14650HX (@2.2 GHz) with Turbo disabled.
Tooling: A customized fork of lzbench.

Dataset 1: `silesia.tar`

Sorted by decompression velocity.

Codec	Level	Ratio	Decompress Speed	Time (s)
misa77	`-0`	54.5 MB/s	5219 MB/s	42.64
misa77	`-1`	51.2 MB/s	4274 MB/s	39.65
`zxc`	`-3`	115 MB/s	2841 MB/s	45.46
`zxc`	`-4`	80.8 MB/s	2726 MB/s	42.63
`lzsse8fast`	—	183 MB/s	2661 MB/s	44.80
`zxc`	`-5`	48.6 MB/s	2599 MB/s	40.25
`lz4hc`	`-12`	7.31 MB/s	2531 MB/s	36.45
`lzsse4fast`	—	186 MB/s	2522 MB/s	45.26
`lz4`	—	371 MB/s	2505 MB/s	47.59
`lizard`	`-10`	320 MB/s	2452 MB/s	48.79
`zstd`	`-1`	297 MB/s	901 MB/s	34.54
`snappy`	—	375 MB/s	855 MB/s	47.89

Dataset 2: `enwik8`

Codec	Level	Ratio	Decompress Speed	Time (s)
misa77	`-0`	38.7 MB/s	4802 MB/s	48.59
misa77	`-1`	40.6 MB/s	4134 MB/s	44.05
`zxc`	`-3`	70.1 MB/s	2674 MB/s	52.05
`zxc`	`-5`	38.7 MB/s	2627 MB/s	46.85
`zxc`	`-4`	52.7 MB/s	2593 MB/s	48.60
`lzsse4fast`	—	141 MB/s	2581 MB/s	47.11
`lzsse8fast`	—	135 MB/s	2553 MB/s	47.25
`lizard`	`-10`	240 MB/s	2462 MB/s	57.31
`lz4`	—	276 MB/s	2355 MB/s	57.26
`lz4hc`	`-12`	9.89 MB/s	2167 MB/s	41.91
`zstd`	`-1`	225 MB/s	831 MB/s	40.66
`snappy`	—	240 MB/s	568 MB/s	55.93

🔍 Deep Dive & Analysis

Because misa77's performance varies based on data "shape," a file-by-file analysis of the Silesia corpus reveals the following:

1. Comparison with LZ4

Dominance: At level 0, misa77 outperforms lz4 on 11 out of 12 files.
The Exception: The x-ray file is highly incompressible. In this case, lz4 essentially performs a memcpy, making it the faster option.

2. The Pareto Frontier

When plotting Throughput vs. Ratio, misa77 consistently sits on the Pareto frontier for compressible data. It provides the fastest decompression while matching or exceeding the ratios of other high-speed LZ codecs. (Exceptions include sao and x-ray).

3. Portability and Vectorization

While the x86-64 path is highly optimized, other architectures utilize a portable path.

No explicit intrinsics are used in the portable path.
Auto-vectorization: The code is written to be easily optimized by modern compilers.

~~Cross-platform results for v0.1.0 are available~~ $\rightarrow$ Note: These are likely outdated; refer to current v0.2.0 benchmarks.

Implementation Detail

The core logic is contained within the src/ directory, with experimental modes located in: src/experimental/ $\rightarrow$ These modes further explore the relationship between compression effort and CPU microarchitecture efficiency.