Introducing ZeroFS: A Log-Structured Filesystem for S3

ZeroFS transforms S3-compatible object storage into a primary storage solution. By implementing a log-structured filesystem, it allows you to treat S3 buckets as standard POSIX filesystems or raw block devices.

🚀 Quick Overview & Performance

ZeroFS provides high-performance access to cloud storage by utilizing a local cache for "warm" reads and an immutable write pipeline.

Metric	Value	Note
Warm Cache Read	$1.6\text{ }\mu\text{s}$	Random reads
Small-Write Latency	$0.83\text{ ms}$	Mean latency
Max FS Size	$16\text{ EiB}$	$\approx 16 \times 1024^6$ bytes
Extent Size	$32\text{ KiB}$	Fixed size

Note: While ZeroFS aims for full POSIX compliance, a small number of semantics that NFS/9P cannot express are excluded. You can find the full list of exclusions in the official repository.

🛠 Rigorous Validation Pipeline

ZeroFS is subjected to an exhaustive CI suite to ensure data integrity and stability. The following tests are executed on every single change:

pjdfstest: Validates POSIX semantics (ownership, permissions, links, renames) across all protocols.
xfstests: The industry-standard kernel filesystem test suite (used for ext4/XFS).
ZFS End-to-End: Builds a ZFS pool on ZeroFS block devices, extracts the Linux kernel source, and performs a full scrub.
Kernel Compilation: Runs make -j$(nproc) to compile the Linux kernel on NFS, 9P, and FUSE mounts to stress-test parallel writes.
stress-ng: Concurrent execution of chmod, chown, chdir, and other file-handling operations.
Jepsen Local-FS: Model-based checking using random operation histories; includes a "crash mode" to verify recovery after a server kill.
Jepsen HA: Fault injection on a leader/standby setup via MinIO to ensure acknowledged writes survive failover.
Self-Hosting: The Rust toolchain compiles ZeroFS while running on a filesystem served by ZeroFS itself.

🌐 Protocol Support

ZeroFS runs a single userspace process that manages one bucket but exposes it through three different interfaces:

Interface Comparison

Protocol	Type	Best Use Case	Client Requirement
NFS	File	Universal compatibility	Native OS support
9P	File	Precise POSIX/fsync	Bundled FUSE client
NBD	Block	ZFS pools, VM boot disks	`nbd-client`

Example Commands:

NFS Mount: mount -t nfs 127.0.0.1:/ /mnt/zerofs
9P Mount: zerofs mount 127.0.0.1:5564 /mnt/zerofs
NBD Attach: nbd-client 127.0.0.1 10809 /dev/nbd0 -N vol1

~~Traditional S3 mounting tools~~ $\rightarrow$ ZeroFS (True block/file semantics).

🌍 Geo-Distribution & High Availability

Because ZeroFS can present S3 buckets as block devices, you can use existing tools like ZFS to create global redundancy.

Scenario: A ZFS mirror spanning three different S3 regions.

# Attach devices from different regions
nbd-client 10.0.1.5 10809 /dev/nbd0 -N storage -persist # US-East
nbd-client 10.0.2.5 10809 /dev/nbd1 -N storage -persist # EU-West
nbd-client 10.0.3.5 10809 /dev/nbd2 -N storage -persist # AP-Southeast

# Create a mirrored pool
zpool create global-pool mirror /dev/nbd0 /dev/nbd1 /dev/nbd2

# Verify status
zpool status global-pool | grep state
# Output: state: ONLINE

If one region fails, the pool degrades, but data remains accessible from the remaining regions.

⚙️ Technical Specifications

🔐 Security & Efficiency

Encryption: Every block is encrypted using XChaCha20-Poly1305. The data key is protected via Argon2id derived from your password.
Compression: Supports zstd or lz4. The system detects the codec on read, meaning you can change compression settings without migrating old data.
Caching: Utilizes configurable disk and memory caches to reduce S3 latency.

🏗 Architecture & Features

Immutable Segments: Data is stored in 32 KiB extents within immutable objects. A separate metadata index tracks these. This ensures that read replicas and checkpoints always see a consistent state.
Checkpoints: You can create named snapshots of the filesystem and mount them as read-only.
Read Replicas: Supports a single writer with multiple read-only instances that automatically sync changes.
TRIM/Discard: Supports TRIM commands. The compaction engine then repacks live data and deletes empty segments from S3 to reduce costs.
Honest fsync: A successful fsync guarantees the data is durable in S3. If a failover occurs and unflushed writes are lost, the subsequent fsync will return an error rather than lying about success.
High Availability: An optional standby server can track the leader via the shared S3 bucket for rapid failover.

Compatibility: Works with Amazon S3, Google Cloud Storage, Azure Blob, any S3-compatible API, or even local disks.