Understanding Pandoc Lua Filters

Pandoc serves as a versatile tool for document conversion. A core feature of its architecture is the support for filters, which allow users to modify the Pandoc Abstract Syntax Tree (AST) during the transition from the parsing stage to the writing stage.

The Evolution of Filtering

Traditional JSON Filters

Historically, Pandoc filters operated by receiving a JSON representation of the AST and outputting a modified version of that same JSON.

These filters are highly flexible because they can be written in any programming language and are executed via the --filter command-line flag.

However, this approach has two primary drawbacks:

1. Performance Lag: There is significant overhead caused by writing JSON to stdout and reading it from stdin twice for every filter in the chain.
2. Environmental Fragility: A filter's functionality depends on the user's local setup, requiring specific language interpreters and AST-manipulation libraries. This makes it difficult to distribute a "plug-and-play" filter.

The Lua Solution

Starting with version 2.0, Pandoc integrated a Lua interpreter (version 5.4) and a dedicated library directly into the executable. This removes all external dependencies. Instead of using JSON pipes, Pandoc data types are marshaled directly into Lua.

Implementation Example

To transform strong emphasis into $\text{small caps}$, you can use the following Lua code:

-- Method 1: Returning a table
return {
  Strong = function (elem)
    return pandoc.SmallCaps(elem.content)
  end,
}

-- Method 2: Using a top-level function (equivalent)
function Strong (elem)
  return pandoc.SmallCaps(elem.content)
end

To apply this, save the code as smallcaps.lua and run: pandoc --lua-filter=smallcaps.lua

---

Performance Analysis

The efficiency gain is evident when converting the Pandoc manual (MANUAL.txt) to HTML. The following table compares the execution time of different filter implementations:

As demonstrated, the Lua filter nearly matches the speed of no filter at all, avoiding the $\text{overhead} \approx \text{JSON marshaling}$ penalty.

---

Lua Filter Architecture

Structural Logic

Lua filters are essentially tables where the keys are element names and the values are functions that process those elements.

Workflow Process:

Execution Rules

• Invocation: Use the --lua-filter flag.
• Ordering: If multiple --filter or --lua-filter flags are used, Pandoc processes them sequentially in the order they appear.
• Function Discovery: If the script doesn't explicitly return a table, Pandoc automatically collects all top-level functions that match Pandoc element names (e.g., Str, Para, Meta).
• Legacy Methods: Returning a list of filters to be called sequentially is now discouraged in favor of the walk method and may be removed.

Return Value Specifications

When a filter function is called, it must return one of the following:

• nil: The element remains unchanged.
• A Pandoc Object: A replacement object of the same type.
• A List of Pandoc Objects: The original object is replaced by this list (spliced into the document). An empty list {} effectively deletes the element.

Crucial Constraint: The output type must match the input type.

• $\text{Inline Element} \rightarrow \text{nil, Inline, or List of Inlines}$
• $\text{Block Element} \rightarrow \text{nil, Block, or List of Blocks}$

If no specific function exists for a node type, Pandoc looks for a general fallback function.

---

Advanced Filtering: Element Sequences

Sometimes, the context of an element's position is required. Pandoc provides two special functions for handling sequences:

1. Inlines (inlines): Triggered for all lists of inline elements (e.g., the contents of a Para or an Image description).
2. Blocks (blocks): Triggered for all lists of block elements (e.g., MetaBlocks, list items, or the main document body).

Note: These functions must return either nil or a list of the correct type. Returning a single element is prohibited to prevent logic errors.

Traversal Order

The sequence in which the AST is traversed can be customized by configuring the traverse key.