RayforceDB: A Pure C Analytics Database with a Lisp-like Syntax

Rayforce is a high-performance engine that integrates columnar analytics, graph traversal, and recursive queries into a single, embeddable execution pipeline. It is specifically engineered for technical teams who prioritize microsecond-level latency over corporate meetings.

The Core Philosophy

"Stop moving data between engines. Relational operators and graph traversals belong in the same execution plan."

By viewing the entire workload as a single unit, Rayforce rewrites the plan holistically to ensure the "hot path" remains as close to the hardware as possible.

---

⚙️ The Execution Pipeline

Rayforce transforms a high-level query into machine-efficient execution through a four-stage process:

1. Compose: Tables, graphs, and rules are combined into a Lazy DAG.
2. Optimize: The entire plan undergoes a multi-pass rewrite.
3. Fuse: Intermediate steps are removed and compiled into bytecode.
4. Execute: Data is processed in parallel using cache-sized "morsels."

Technical Specs:

• Written in $\approx 16,000$ lines of focused C.
• Zero external dependencies.
• Built-in IPC client/server transport.
• Licensed under MIT; fully embeddable.

---

🗣️ The Rayfall Language

Rayforce utilizes Rayfall, a Lisp-inspired language designed for complex data interrogation. Users can interact with the engine via three distinct surfaces:

1. Rayfall (Interactive/Native)

The native syntax allows for seamless transitions between aggregation and graph expansion.

;; Aggregate high-value flow, then traverse counterparties
(set flow (select {from: trades where: ( Notional 1000000) by: Counterparty Volume: (sum Notional)} ))
(.graph.var-expand network flow 1 3 )

2. Python API

For those integrating into data science workflows:

import rayforce as rf

trades = rf.read_csv("trades.csv")
flow = (trades.where(rf.col("Notional") > 1_000_000)
        .group_by("Counterparty")
        .agg(rf.col("Notional").sum()))

network.expand(flow, min_depth=1, max_depth=3)

3. C API

For maximum performance and embedding:

ray_t *trades = ray_read_csv("trades.csv");
ray_graph_t *g = ray_graph_new(trades);
ray_op_t *amount = ray_scan(g, "Notional");
ray_op_t *limit = ray_const_i64(g, 1000000);
ray_op_t *hot = ray_gt(g, amount, limit);
ray_t *result = ray_execute(g, ray_filter(g, trades, hot));

Example Result Set:

---

🛠️ Integrated System Capabilities

Rayforce eliminates the need for a secondary platform by providing four primary pillars:

• Columnar Analytics: Vectorized implementations of joins, groups, windows, pivots, and time-series primitives.
• Native Graph Engine: Support for CSR traversal, HNSW, centrality, communities, shortest paths, and WCO joins.
• Recursive Datalog: Reachability and recursive relationships are expressed as rules and compiled directly into the DAG.
• Flexible Storage: Supports memory maps, IPC, block offloading, and splayed/partitioned tables (including CSV).

Note: Calculation of Notional is typically defined as: $\text{Notional} = \sum (\text{price} \times \text{quantity})$

---

🌍 Rayforce in the Wild

This is not a synthetic demo; Rayforce powers production systems across several sectors:

• Trading Technology: LYNX uses it for US equity market infrastructure, including routing, risk, and compliance.
• Investment Technology: FORTITUDO employs it for quantitative research and portfolio analytics.
• Open Source: PeachQ (an MIT-licensed implementation of the $q$ language) uses Rayforce for its vector operations and interactive tables.

---

☁️ Future & Access

Rayforce is moving toward a managed cloud experience to bridge the gap between local prototyping and production scaling.

Cloud Preview Status:

• Deployment (eu-west)
• Query Service
• Managed Engine

Get Started: The core remains open. You can clone the repository, compile the source, and maintain total control over your data path.

• GitHub: [View Source]
• Documentation: [Read Docs $\rightarrow$]
• Supported OS: Linux, macOS