Engineering Homescale: Reimagining Database Infrastructure

Years ago, I spent time developing database cloning utilities before the industry saw the rise of platforms like PlanetScale. While those early tools weren't designed for massive scale, they were built on a fundamental premise: decouple the storage layer from the compute layer and leverage specialized storage tech to handle file cloning.

This concept resurfaced recently during a conversation about an F1 team seeking a Rust developer with expertise in Ceph. That reminder sparked a memory of a failed product attempt previous venture, leading me to jokingly propose a new project to a friend: Homescale—essentially "PlanetScale for the home."

🏗️ The Vision for Homescale

I am currently developing Homescale (available at github.com/homescale-dev/homescale). The goal is to enable the creation of writable database instances and point-in-time branches derived from immutable snapshots, all without the overhead of duplicating the entire dataset.

To organize this, I've adopted a conceptual model similar to Docker's image/container relationship:

The Proposed Workflow

If you were using the Homescale CLI, your interaction would look something like this:

# 1. Create a base image for Postgres
homescale image create --engine postgres postgres-base

# 2. Spin up a development database from that image
homescale container create --image postgres-base dev-db

# 3. Access the database
homescale container connect dev-db

# 4. Create a feature branch from the current dev-db state
homescale branch create --container dev-db feature-login

# 5. Access the new branch
homescale container connect feature-login

⚙️ Engine Agnosticism & Orchestration

While the examples above utilize Postgres, the underlying storage architecture of Homescale is database agnostic.

"Any database engine that persists its state on a filesystem backed by a block device and supports recoverable snapshots can fit into this model."

The Role of the Adapter

To make this work, Homescale uses an adapter for each engine. For Postgres, the adapter handles:

• Initializing the image.
• Booting the database process.
• Managing connection strings.
• Triggering snapshots for recovery.

Meanwhile, Homescale manages the "macro" lifecycle: volumes, immutable states, writable clones, workloads, and lineage.

Visualizing the Lineage

When you branch feature-login from dev-db, you aren't just copying files. The relationship follows a specific path:

Because the base image is already immutable, dev-db can be created directly. However, branching from a writable container requires an intermediate step: Homescale captures a point-in-time snapshot of dev-db first, then spawns feature-login from that static state.

💾 Separating Storage from Compute

A critical requirement is efficiency. If you have a $100\text{ GB}$ database, creating a branch should not immediately consume another $100\text{ GB}$ of disk space. Instead, the branch should share data with its parent, only consuming extra space for new or modified data.

This necessitates moving the branching logic below the database process and into the storage layer. This isn't a radical idea; it's how the giants do it:

• Amazon RDS: Uses EBS volumes to separate logs and data from the compute instance.
• Google Cloud SQL: Utilizes Persistent Disk or Hyperdisk, allowing the VM to be replaced without losing the data.

The I/O Stack

The abstraction layer looks like this:

By maintaining this boundary, Homescale can manipulate the storage independently of whether the database process is running.

🚀 The "Secret Sauce": Copy-on-Write (COW)

The magic that allows for near-instant branching is Copy-on-Write (COW).

1. Initial State: When dev-db is created, it reads directly from postgres-base.
2. Modification: Any writes made by dev-db are stored in its own layer, leaving the image untouched.
3. Branching: When feature-login is created from a snapshot of dev-db, it initially points back to that snapshot.
4. Divergence: If feature-login reads data it hasn't changed, the storage layer traverses the lineage chain back to the parent. On the first write to a shared block, the storage layer copies that specific allocation unit into the writable container.

Storage Logic

Mathematically, the storage cost for a branch can be represented as: $\text{Total Space} = \text{Base Image} + \sum (\text{Delta Changes per Container})$

Project Roadmap

• Define Image/Container/Branch model
• Design Storage/Compute separation
• Implement Ceph/COW integration
• Finalize Postgres adapter
• Build CLI tool