Kubernetes on Oxide: How customer needs shaped our integrations
Kubernetes on Oxide: How Customer Needs Shaped Our Integrations
By Matthew Sanabria, Solutions Software Engineer
In the closing months of 2024, Oxide faced a specific challenge: our customers and prospective users were eager to deploy Kubernetes on our platform, yet we lacked the official integrations necessary to facilitate this.
While Kubernetes provides standardized extension points to define infrastructure behavior, and Oxide offers the necessary primitives via our APIs, there was a missing link. We lacked the specific software—and the deep understanding of user requirements—to bridge that gap.
"The goal wasn't just to build software, but to understand which integrations actually solved real-world customer pain points."
I joined Oxide as the inaugural Solutions Software Engineer with a clear mandate: build the tools that solve customer problems. My primary objective was to streamline the deployment and operation of Kubernetes on Oxide.
The Starting Line
My onboarding involved two critical pieces of documentation:
- A pull request from a customer for a Rancher node driver.
- An early version of
RFD 493: Initial Kubernetes Integrations.
We initially thought about designing these integrations in a vacuum. Instead, we adopted a feedback-driven approach. We tracked the actual hurdles customers faced as they progressed through the lifecycle of a cluster.
The evolution of our integrations followed this path:
- Provisioning: Led us to develop support for Rancher, Omni, and Cluster API.
- Operations: Highlighted the need for infrastructure reconciliation.
- Connectivity: Revealed gaps in networking.
- Persistence: Exposed constraints regarding storage.
Solving the Provisioning Puzzle
Our first priority was to unblock the customer who contributed the Rancher node driver. By solving their specific use case, we could gain practical experience with Kubernetes on Oxide and identify subsequent bottlenecks.
Because different users have different workflows, we didn't settle on one tool; we released three distinct integrations.
1. The Rancher Node Driver
To maintain the customer's contribution, I first had to master the Rancher ecosystem. A Rancher node driver is essentially an executable plugin that instructs Rancher on how to manage virtual machines on a specific platform.
The Oxide driver acts as a translator, converting Rancher's requests into Oxide API calls. This allows users to provision Oxide instances as nodes within Rancher-managed clusters.
Outcome: I merged the PR, implemented CI/CD pipelines, wrote the documentation, and published the release. This marked Oxide's first official Kubernetes integration, which is currently running in production.
2. The Omni Infrastructure Provider
Next, we addressed interest in Sidero Labs' Omni, which is used to provision clusters running Talos Linux. Omni utilizes "infrastructure providers" to create instances and register them.
With KubeCon North America 2025 approaching, we set a goal to partner with Sidero Labs. We had a tight window:
- Define requirements
- Build the provider
- Test across Oxide APIs
- Showcase at the Oxide+Sidero event
- (Timeline: 7 weeks)
During development, we discovered a technical conflict regarding filesystem probes. Oxide utilizes a FAT12 filesystem for cloud-init user-data, whereas Talos Linux specifically looked for an ISO 9660 superblock. If the ISO 9660 read failed, Talos stopped searching, meaning it never saw the Oxide user-data.
Since a formal fix wouldn't be ready for KubeCon, we implemented a "creative" workaround:
# Conceptual workaround: Padding user-data to force ISO 9660
# Adding enough comment lines to increase file size
# until the filesystem probe identifies it as ISO 9660.
echo "# Padding for ISO 9660 compatibility" >> user-data
# ... repeat until size threshold is met ...
Despite this quirk, the event was a success, and Omni users can now provision Talos Linux nodes on Oxide.
3. Cluster API (CAPI) Provider
While mentioned in RFD 493, the Cluster API provider was initially deferred due to limited engineering bandwidth. However, as demand grew, it became a priority.
Unlike Rancher or Omni, CAPI provides an upstream, provider-extensible API. It allows operators to manage clusters declaratively using Kubernetes custom resources.
The Result: My colleagues, Josh and Brandon, developed and released the Cluster API Provider Oxide (CAPOx), providing a native Kubernetes experience for cluster provisioning.
Integration Comparison Summary
The following table summarizes the three paths we provided to customers:
| Integration | Primary Tool | Management Style | Key Benefit |
|---|---|---|---|
| Rancher Driver | Rancher | Plugin-based | Rapid deployment for Rancher shops |
| Omni Provider | Sidero Labs / Talos | Infrastructure Provider | Optimized for Talos Linux |
| CAPOx | Cluster API | Declarative / K8s-native | No third-party platform required |
Mathematical Representation of the "Gap"
If we define the desired state of a cluster as and the current state of the Oxide infrastructure as , the integration software serves as the operator:
By iterating on based on customer feedback, we reduced the error margin between the platform's primitives and the user's expectations.
The journey from a single customer PR to a suite of provisioning tools has not only unblocked our users but has also highlighted the remaining work needed in networking and storage reconciliation.