DuckDB $\text{v2.0}$ : Transitioning to a PEG-based SQL Parser

By Daniël ten Wolde | Published: 2026-08-20 | $\approx 16$ min read

TL;DR: In DuckDB $\text{v2.0}$ , the legacy PostgreSQL-derived SQL parser is being swapped for a Parsing Expression Grammar (PEG) based system. This change makes the parser simpler to maintain, easier to evolve, and capable of being extended during runtime.

At DuckDB, our core philosophy is to minimize the friction associated with database management. To achieve this, we rely on Structured Query Language (SQL), the universal language of data.

We have previously highlighted our "friendly" SQL enhancements, such as:

GROUP BY ALL
Column filtering via SELECT * EXCLUDE (...)

However, before these features can be executed, the system must verify that the syntax is correct. This is the responsibility of the parser. In version $\text{v2.0}$ , we are replacing this component entirely—a massive internal shift that should be completely transparent to the end user.

The Query Processing Pipeline

DuckDB handles a SQL string through a multi-stage sequence. The initial phase focuses on the following three components:

Tokenizer: This stage breaks the raw input string into discrete tokens.
- Categories include: KEYWORD, NUMBER, or IDENTIFIER.
- Cleanup: It identifies and discards comments (e.g., -- or /* ... */).
Parser: This component checks if the sequence of tokens adheres to the DuckDB grammar, resulting in a ParseResult tree.
Transformer: This stage maps the generic ParseResult into DuckDB's internal Abstract Syntax Tree (AST), creating objects like SQLStatement, TableRef, and ParsedExpression.

Parser vs. Binder: What's the difference?

It is important to distinguish between syntactic validity and semantic validity.

Component	Responsibility	Example Error
Parser	Checks if the "grammar" is correct.	`SELECT * WHERE true FROM range(1);` $\rightarrow$ Syntax Error (Wrong clause order)
Binder	Checks if the referenced objects exist.	`FROM missing_table;` $\rightarrow$ Catalog Error (Table not found)

As shown above, the first query fails the parser because, while the tokens are valid, the order is illegal. DuckDB allows SELECT-first or FROM-first syntax, but not a random arrangement. The second query is syntactically perfect, so it passes the parser, but the Binder rejects it because missing_table doesn't exist in the database.

Understanding the DuckSQL Dialect

While a global SQL standard exists, most databases implement their own "dialect." Common examples include:

PostgreSQL
Oracle
GoogleSQL (BigQuery)
MySQL / MariaDB
SQLite
Spark SQL
DuckSQL (DuckDB's specific dialect)

DuckDB's dialect is heavily inspired by PostgreSQL, but it has diverged to include unique features (like GROUP BY ALL) and omit certain Postgres behaviors. We refer to this specific implementation as DuckSQL.

It is crucial to remember that the dialect (the rules of the language) is distinct from the parser (the code that implements those rules). For $\text{v2.0}$ , we are rewriting the implementation and the grammar itself.

Moving Beyond the Legacy Parser

When DuckDB launched in 2018, adopting a PostgreSQL-derived parser was a logical choice. It provided a ~~battle-tested~~ mature foundation that users already understood. We simply added a Transformer to bridge the Postgres-style tree to our internal AST.

However, this approach created bottlenecks:

Rigidity: Extending DuckSQL required modifying a YACC/Bison grammar.
Complexity: Because Bison creates an LALR(1) parser, even minor changes could trigger shift/reduce or reduce/reduce conflicts.
Maintenance: As the language grew, these conflicts made grammar updates increasingly fragile.

This struggle led to our research into Parsing Expression Grammars (PEGs), which we detailed in a previous blog post and a [CIDR paper]. While the PEG parser began as a limited prototype, it is now ready for production.

Goals for the New Parser:

Eliminate LALR(1) conflicts.
Enable runtime extensibility.
Simplify the addition of new "Friendly SQL" features.
Maintain full compatibility with DuckSQL.

A Primer on PEG Parsers

A PEG defines a language through a set of named rules. Unlike LALR parsers, PEGs use ordered choices.

Consider these rules from the new DuckDB grammar:

SelectFrom      <- SelectFromClause / FromSelectClause
SelectFromClause <- SelectClause FromClause?

Key Operators:

<- : Defines a rule.
/ : Represents an ordered choice (try the first; if it fails, try the second).
? : Indicates an optional element.

This logic allows DuckSQL to support both traditional and "friendly" query structures:

SELECT * FROM range(1); $\rightarrow$ Matches SelectFromClause.
FROM range(1) SELECT *; $\rightarrow$ Fails SelectFromClause, then matches FromSelectClause.

Because the alternatives are explicitly ordered, the ambiguity that causes shift/reduce conflicts in Bison is eliminated. This flexibility is why Python 3.9 also migrated from an LL(1) parser to a PEG-based one.

In the final implementation, the PEG matcher processes the tokens provided by the tokenizer and constructs the ParseResult tree, which is then passed to the transformer.