Extending Polars with Rust Expression Plugins

By Kostas Pardalis | July 17, 2026

fenic is designed as a semantic DataFrame library, providing a PySpark-inspired API specifically tailored for constructing AI and LLM pipelines that handle messy, unstructured data.

In this article, we explore a specific technical hurdle we encountered during development and the Polars feature that provided the solution: the expression plugin system. To achieve our goals, we implemented nine distinct Rust plugins to augment the Polars expression engine.

Below, we detail the rationale behind this architectural choice and the implementation process.

---

🎯 The Objective: Why We Built This

What is fenic?

At its heart, fenic is a tool for AI pipeline orchestration. For those familiar with Polars, the key takeaway is that Polars serves as the underlying execution engine.

Every operation in fenic is translated into a Polars expression or a plan involving pl.DataFrame objects. Whether it is a semantic operator, an LLM-based extraction/classification, or a similarity join, everything relies on the same machinery. Consequently, the capabilities of fenic are fundamentally limited by what can be expressed within Polars.

Polars provides a high-performance, columnar, Arrow-native engine featuring a robust expression language and a lazy optimizer.

The Gap in Functionality

Our target workloads involve unstructured text, requiring row-level operations that Polars does not support natively. We needed the ability to:

• Chunking: Split documents into overlapping windows based on token counts (essential for RAG/embeddings).
• Templating: Implement full Jinja support, using a struct of columns as variables before sending data to an LLM.
• JSON/Markdown: Query JSON columns via jq and transform markdown into structured ASTs.
• Fuzzy Matching: Implement six different edit-distance metrics for deduplication and joining.
• Transcript Parsing: Convert SRT/WebVTT files into typed, timestamped cue records.
• Rich Casting: Map values to fenic's logical types (e.g., embeddings, markdown, typed structs) which exceed Polars' physical dtypes.

Each of these must operate on an entire column, return a typed result, and integrate seamlessly into a larger DataFrame pipeline.

---

⚠️ Where Standard Approaches Fail

The reflexive common approach to adding custom logic to Polars is through Python User Defined Functions (UDFs), such as map_elements (per-row) or map_batches (per-Series). While fenic still uses map_batches for IO-bound tasks (like calling an LLM API), UDFs are insufficient for text processing for three main reasons:

1. Performance Bottlenecks

map_elements executes Python code for every single row. This involves a round-trip between the Rust engine and the Python interpreter for every value, all while constrained by the Global Interpreter Lock (GIL). When processing millions of rows for tokenization, the time complexity feels like $O(n \times \text{Python Overhead})$.

2. Optimization Barriers

Polars achieves its speed by planning and executing an expression tree holistically. A Python UDF acts as an opaque box. The engine cannot "see" inside it, which:

• Prevents optimization.
• Forces premature materialization of data.
• Breaks the single-pass pipeline.

3. Type Safety

UDF output types are essentially "best guesses" or loose assertions. We required strict, declared dtypes (e.g., List(String), Struct, or fixed-size arrays for embeddings) to ensure the rest of the execution plan could be type-checked.

Comparison: UDFs vs. Plugins

Alternative considered: Forking Polars. We rejected this because maintaining a permanent fork of a fast-moving project is a maintenance nightmare. Pre-processing data outside the DataFrame was also rejected as it destroys the laziness and composition that make Polars valuable.

---

🚀 The Solution: Polars Expression Plugins

Polars offers a dedicated mechanism for this: expression plugins. By writing a kernel in Rust and registering it, the function becomes a standard pl.Expr.

The Architecture

The flow of data looks like this:

Why Rust?

Rust provides a mature ecosystem of crates that allow us to implement complex logic without reinventing the wheel:

• jaq for JSON querying.
• minijinja for templating.
• rapidfuzz for string metrics.
• tiktoken for AI tokenization.

We utilize pyo3-polars to handle the Foreign Function Interface (FFI), Arrow marshaling, and the bridge for keyword arguments.

Our Implementation Strategy

We didn't rewrite everything. We followed a strict rule:

"A plugin is only created to fill a gap that Polars cannot express natively."

For example, our fuzzy matcher only implements the six primitive kernels in Rust; the higher-order ratios are then calculated using standard Polars expressions.

---

💎 The Result

By moving these operations into Rust plugins, every text operation in fenic is now a native Polars expression. They operate on shared Arrow memory and maintain strict typing.

The most significant advantage is composition. You can now perform a complex sequence of operations in a single expression:

# Conceptual example of a composed expression
df.select(
    pl.col("raw_text")
    .pipe(fenic_parse_markdown)  # Rust Plugin
    .pipe(fenic_jq_filter)       # Rust Plugin
    .struct.field("content")     # Native Polars
    .pipe(fenic_cast_to_struct)  # Rust Plugin
)

In this scenario, Polars plans and executes the entire sequence in a single pass, interleaving custom Rust kernels with built-in list operations without a single trip back to Python.

The following sections will dive deeper into the actual code and implementation details...