protobuf-py: Protobuf for Python, without compromises

July 7, 2026 | By Team Buf

Today, we are thrilled to unveil protobuf-py, a ground-up reimplementation of the Protocol Buffers library specifically for the Python ecosystem.

🚀 The Core Value Proposition

protobuf-py isn't just another wrapper; it is a complete implementation that ensures you no longer have to choose between spec compliance and developer experience.

Feature Checklist

• Full conformance with binary and JSON suites
• Support for proto2, proto3, and Editions
• Extensions and custom options
• Preservation of unknown fields
• Dynamic messages and well-known types
• Zero runtime dependencies
• Native support for Python 3.10+

"Python is too critical to the modern stack—powering AI agents, ML pipelines, and infrastructure—for its Protobuf experience to feel like a secondary concern."

---

The Historical Dilemma

For years, Python developers were trapped in a binary choice:

We initially attempted to solve the RPC layer via connect-py (supporting both Connect and gRPC). However, we realized that a high-quality RPC stack requires a rock-solid message foundation. We wanted a runtime that was $\text{Complete} + \text{Readable} + \text{Fast}$. protobuf-py is the answer to that equation.

---

Why the Official Package Feels "Off"

When you install the standard protobuf package, you typically use upb, a C-based engine. This architecture creates a fundamental disconnect:

1. The Memory Gap: Your message resides in a C arena; the Python object is merely a handle.
2. The Performance Tax: Every field access requires crossing the boundary from Python $\rightarrow$ C $\rightarrow$ Python.
3. The Developer Friction:
   • Readable _pb2.py files $\rightarrow$ Obfuscated files that are hard to debug.
   • "Go-to-definition" leads to a wall of serialized bytes rather than actual Python code.
   • The API uses C++ ergonomics: SerializeToString, HasField, and WhichOneof.
   • The WhichOneof Paradox: It returns a string that you must then pass back to getattr to get the value the engine already found.
   • Absolute imports break when nesting messages in packages, necessitating external tools like fix-protobuf-imports.
   • A global type pool means importing two versions of the same .proto causes a runtime crash.

Data Flow Comparison

---

The protobuf-py Architecture

Instead of acting as a proxy, protobuf-py keeps the data directly in Python. Messages are plain objects utilizing __slots__ for efficiency.

While the logic is Pythonic, we use a Rust accelerator to handle the heavy lifting of parsing and serialization. This ensures that for production workloads, performance is $O(\text{upb})$.

Readable Generated Code

The protoc-gen-py plugin produces code that humans can actually read:

class User(Message[_UserFields]):
    __slots__ = (
        "first_name", "last_name", "active", 
        "manager", "locations", "projects", "contact"
    )
    
    if TYPE_CHECKING:
        def __init__(
            self, *,
            first_name: str = "",
            last_name: str = "",
            active: bool = False,
            manager: User | None = None,
            locations: list[str] | None = None,
            projects: dict[str, str] | None = None,
            contact: Oneof[Literal["email"], str] | Oneof[Literal["phone"], str] | None = None,
        ) -> None: ...

    first_name: str
    last_name: str
    active: bool
    manager: User | None
    locations: list[str]
    projects: dict[str, str]
    contact: Oneof[Literal["email"], str] | Oneof[Literal["phone"], str] | None

Idiomatic Usage

Working with protobuf-py feels like using standard Python libraries:

import copy
from gen.user_pb import User
from protobuf import Oneof

# Initialization
user = User(first_name="Alice", active=True, locations=["NYC", "LDN"])
user.last_name = "Smith"

# Serialization
wire = user.to_binary()
parsed = User.from_binary(wire)

# Modern Pattern Matching (Python 3.10+)
match parsed.contact:
    case Oneof(field="email", value=email):
        send(email)
    case Oneof(field="phone", value=phone):
        call(phone)

# Functional updates (Python 3.13+)
inactive = copy.replace(parsed, active=False)

Final Advantages

• Type Safety: pyright, mypy, and ty work out of the box without needing separate .pyi stub packages.
• Portability: Relative imports are used, allowing generated files to live anywhere in your project.
• Isolation: Types are managed via an explicit Registry rather than a process-wide global pool.
• Completeness: Unlike betterproto (which is limited to proto3), protobuf-py supports the full spectrum of the specification, including proto2, editions, and complex unknown-field round-tripping.