uv: Implementing File-Level Deduplication in the Wheel Cache

This report summarizes Pull Request #21327 in the astral-sh/uv repository, authored by charliermarsh. The primary objective of this update is to optimize storage by deduplicating individual files within the wheel cache.

📌 Overview of the Change

Previously, uv supported content-addressed caching, but this was limited to the wheel-level. While downloading the same wheel from different sources would result in a shared cache entry, the files inside those wheels (or identical files across different wheels) were stored redundantly.

The New Architecture

The updated system introduces deduplication at the granular file level. The process follows this logic:

Every file is hashed using the BLAKE3 algorithm.
Files are stored in a dedicated files-v0 bucket based on their hash.
These objects are then hardlinked into their respective locations within archive-v0.

Because hardlinks are used, the actual installation process remains unchanged. Cache cleanup is handled automatically: file objects are purged when their hardlink count reaches one.

📉 Storage Impact

Based on a prior proposal (#19694), the following data illustrates the potential space savings on a local machine:

File Selection Criteria	Additional Savings	Hardlinked Files	Distinct `files-v0` Objects
Executables & Native Libs	275.7 MiB	3,336	3,088
Payload files $\ge 10$ MiB	235.0 MiB	80	78
Payload files $\ge 1$ MiB	279.7 MiB	373	349
Payload files $\ge 100$ KiB	353.4 MiB	2,830	2,458
Payload files $\ge 10$ KiB	475.7 MiB	23,767	18,722
Payload files $\ge 1$ KiB	537.4 MiB	95,156	66,422
All payload files	545.2 MiB	134,222	87,129

Result: A total saving of $545.2$ MiB, which represents approximately $10\%$ of the total cache size.

🚀 Performance Benchmarking

The optimization was tested by comparing commit 85c3f7485 (optimized) against a3f977ece (binary-only), with the --preview-features content-addressed-cache flag enabled.

Median Installation Times

Positive values indicate a slowdown.

Wheel Cache	Parent Median	Optimized Median	Change (95% CI)
AnyIO 4.9.0 (cold)	90.4 ms	93.4 ms	$+3.39\%$ ( $+2.87\%$ to $+3.86\%$ )
AnyIO 4.9.0 (warm)	29.9 ms	30.3 ms	$+1.14\%$ ( $-1.08\%$ to $+3.85\%$ )
SymPy 1.14.0 (cold)	367.1 ms	381.6 ms	$+3.95\%$ ( $+3.49\%$ to $+4.46\%$ )
SymPy 1.14.0 (warm)	84.6 ms	84.0 ms	$-0.64\%$ ( $-1.04\%$ to $-0.11\%$ )
NumPy 2.2.6 (cold)	314.1 ms	326.5 ms	$+3.95\%$ ( $+3.24\%$ to $+4.82\%$ )
NumPy 2.2.6 (warm)	62.5 ms	62.7 ms	$+0.28\%$ ( $-0.55\%$ to $+1.25\%$ )
PyTorch 2.7.1+cpu (cold)	2982.6 ms	3072.9 ms	$+3.03\%$ ( $+0.98\%$ to $+5.18\%$ )
PyTorch 2.7.1+cpu (warm)	404.0 ms	403.1 ms	$-0.21\%$ ( $-0.72\%$ to $+0.11\%$ )

Note: While cold installs saw a net slowdown of $\approx 4\%$ , this is a significant improvement over previous regressions, which were ~~$+4.02\%$ for AnyIO, $+19.41\%$ for SymPy, $+12.75\%$ for NumPy, and $+15.30\%$ for PyTorch~~.

🛠️ Technical Methodology

The benchmarks were conducted under the following environment:

Hardware/OS: Linux/ext4 on an AMD EPYC-Milan VM (pinned to 8 CPUs).
Software: Python 3.12.13 and Rust 1.98.0 (optimized profile, no LTO).
Test Parameters:
- 1,860 total measured installs.
- Offline installation of pinned local wheels (no dependencies or bytecode compilation).
- Cold = Cache fully deleted; Warm = Cache pre-primed.
- Statistical Analysis: Paired percentile bootstrap of the ratio of medians with $10,000$ resamples.

Verification Checklist

Inode checks confirmed file-store object sharing across all four wheels.
Integration tests passed (50 executions across 10 stress iterations).
Validated local and streamed wheels (1 and 4 workers).
Verified RECORD handling and cache cleanup.
Confirmed cross-filesystem installation stability.
Codspeed verified 25 untouched benchmarks.

🖼️ Contributors & Automation

@charliermarsh charliermarsh (Author) @astral-automations-bot astral-automations-bot @charliemarsh-oai charliemarsh-oai

# Example of the feature flag used in testing
uv pip install --preview-features content-addressed-cache <package>