🚀 Python Polars: The Definitive Cheatsheet

Derived from the O'Reilly publication

Polars is a high-performance library designed for the transformation, analysis, and visualization of data, utilizing a fast and highly expressive DataFrame API.

🛠️ Getting Started

Before diving in, ensure your environment is configured:

• Install Polars with all optional features:

  uv pip install "polars[all]"
  

• Initialize the library and verify your environment:

  import polars as pl
  pl.show_versions()
  

---

🔄 The Polars Workflow

A typical Polars operation follows a linear pipeline: Read $\rightarrow$ Transform $\rightarrow$ Write. Most queries are constructed as a single, fluid chain of method calls.

# Example: A complete chain
fruit = (
    pl.read_parquet("input.parquet")
    .filter(pl.col("sales") > 100)
    .write_parquet("fruit.parquet")
)

Notation Guide:

• df: A standard DataFrame
• lf: A LazyFrame
• o: A secondary DataFrame for joins/combinations
• e: A Polars expression (e.g., e.abs() refers to pl.col("x").abs())

---

🏗️ Data Structures

Polars organizes data into two primary structures: Series and DataFrames.

Unlike pandas, Polars does not use row indices. It prioritizes immutability and method chaining over modifying data in-place.

Creation

• Series: A named sequence of values.

  series = pl.Series("sales", [150.00, 300.00, 250.00])
  

• DataFrame: Created from a dictionary of sequences or via pl.read_*() functions.

  df = pl.DataFrame({"sales": series, "id": [41, 42, 43]})
  

• Adding Indices: Since there is no implicit index, create one manually:

  df.with_row_index("id")
  

---

⚡ Eager vs. Lazy APIs

Polars offers two modes of execution. The Eager API runs commands immediately, while the Lazy API constructs an optimized query plan before execution.

The Optimizer enhances performance via:

1. Predicate Pushdown: Filtering rows as early as possible.
2. Projection Pushdown: Removing unused columns early.

Lazy Operations

• Execution: Use .collect() to turn a LazyFrame back into a DataFrame.
• Big Data: Use lf.collect(engine="streaming") to process datasets that exceed available RAM.
• Debugging:
  • lf.show_graph(): Visualizes the optimized query plan.
  • lf.profile(): Returns timing data for each node in the plan.

---

🧮 Data Types

Polars adheres to the Apache Arrow memory specification for efficient columnar storage.

Type Inspection & Casting

• Inspection:
  • df.dtypes: Returns a list/dictionary of types.
  • df.glimpse(): A vertical summary of columns (ideal for wide data).
  • df.describe(): Summary statistics and null counts.
  • df.estimated_size("mb"): Memory footprint in specified units.
• Casting:

  # Strict cast (errors on failure)
  df.with_columns(pl.col("x").cast(pl.Int8))
  
  # Non-strict cast (overflows become null)
  df.with_columns(pl.col("x").cast(pl.Int8, strict=False))
  

---

💾 Reading and Writing Data

Polars provides four distinct I/O patterns depending on the execution mode:

1. read_*() $\rightarrow$ Eager (returns DataFrame)
2. scan_*() $\rightarrow$ Lazy (returns LazyFrame)
3. write_*() $\rightarrow$ Eager (writes to disk)
4. sink_*() $\rightarrow$ Streaming (writes without loading all into RAM)

Format Support Matrix

Pro Tips:

• Cloud Storage: Use URIs with globs and storage_options for S3/Azure/GCP.

  pl.scan_parquet("s3://bucket/*.parquet", storage_options={"aws_region": "us-east-2"})
  

• Partitioning: Stream results to a partitioned dataset using lf.PartitionBy("out/", key="x").

---

🛠️ Transforming Data

Selecting Columns

You can isolate columns by name, position, or data type using .select().

• Basic Selection: df.select("a", "b")
• Expression Selection: Use expressions to modify data during selection.

  # Multiply column 'x' by 2
  df.select(pl.col("x") * 2)
  

• Aliasing: Use keyword arguments to name the result of an expression.