How to Ship a Database Every Day

By Tarun Pothulapati (Engineer)
Published August 14, 2026

At turbopuffer, our customers are constantly requesting enhancements—whether it's a new API, a different index structure, or a revised query plan. To keep pace, we push dozens of database upgrades across our clusters daily, often deploying a Pull Request (PR) the same day it's opened.

"Shipping fast is how we make every customer feel like they're our only customer."

Deployment Architecture

To ensure maximum flexibility, we support three distinct deployment models across various cloud regions. We currently manage over 100 clusters—a 2x increase from six months ago—and that number continues to climb as we expand our public footprint and BYOC (Bring Your Own Cloud) offerings.

Comparison of Deployment Models

---

The BYOC Dilemma

The primary challenge with BYOC is that these clusters reside within the customer's own cloud environment. By default, we hold no credentials to these accounts.

Some vendors attempt to solve this by requesting a separate, dedicated cloud account with standing admin credentials. However, we found this approach creates too much friction:

• $\text{Security Monitoring} + \text{Compliance} + \text{Billing Overhead} = \text{Too much complexity}$

We refuse to maintain separate control planes for SaaS and BYOC. Therefore, we designed our system for the lowest common denominator: we must be able to operate every cluster without ever "reaching in."

Operational Requirements

• Clusters must drive operations to a terminal state independently.
• Resilience against connection loss to the central control plane.
• Zero-credential access for BYOC environments.

---

The Solution: Local State Machines

To achieve this, we deploy a local cluster agent on every instance that functions as a state machine. We utilize a Kubernetes Custom Resource Definition (CRD) called TurbopufferOperation to define every possible action, from a simple tidy to a full upgrade.

A Kubernetes controller manages these custom resources (CRs) via a reconciliation loop. The logic follows a mathematical transition: $S_{current} \xrightarrow{\text{reconcile}} S_{next} \dots \rightarrow S_{terminal}$

The Operation Lifecycle

While most operations advance automatically, BYOC customers can implement "gates" such as manual approvals or specific maintenance windows. These are simply represented as waiting states within the CRD.

Technical Implementation

The state is stored as a durable object within the cluster's own etcd. This ensures that if the agent crashes or the network drops, the work persists and continues once connectivity is restored.

Example of a conceptual TurbopufferOperation manifest:

apiVersion: turbopuffer.io/v1
kind: TurbopufferOperation
metadata:
  name: db-upgrade-v2
spec:
  type: "upgrade"
  version: "2.1.0"
  maintenanceWindow: "2026-08-20T02:00:00Z"
status:
  state: "PENDING"
  lastUpdated: "2026-08-14T10:00:00Z"

---

Why We Avoided Infrastructure-as-Code (IaC)

One might ask why we didn't simply use Terraform or Helm. We determined they were the wrong tools for managing a fleet of databases for two main reasons:

1. Organizational Structure: We don't have a separate DevOps team. Our database engineers handle their own deployments. Forcing them to learn complex IaC tools would create a bottleneck in the critical path.
2. Architectural Mismatch: The standard Terraform model requires running terraform apply from a central piece of infrastructure. This contradicts our "no reach-in" requirement for BYOC.