Formal methods and the future of programming
The Evolution of Programming: Formal Methods & The AI Era
For a quarter of a century, I have maintained a consistent stance: Jane Street, as an entity, was not interested is no longer indifferent to formal methods.
To be clear, we have always been proponents of utilizing tools that enhance code reliability. We view type systems as a form of "lightweight" formal verification, and they have provided us with immense value. Consequently, one might assume we would naturally gravitate toward more rigorous formal methods. However, except for niche applications like hardware synthesis, we historically concluded that the overhead was simply too high.
The Cost of Absolute Certainty
Consider the case of the formally verified microkernel, seL4. While it is a monumental technical feat, the resource expenditure was staggering:
It required 25 person-years of labor to verify a mere 8,700 lines of C code. On average, every single line of code necessitated approximately 23 lines of proof, costing about half a person-day per line.
For a security-critical microkernel with rigid specifications, this investment is justifiable. For the vast majority of software—including our own most critical systems—it was not.
Comparison of Verification Approaches
| Approach | Cost/Effort | Scope of Guarantee | Primary Use Case |
|---|---|---|---|
| Type Systems | Low/Integrated | Structural correctness | General Application Dev |
| Testing/Fuzzing | Medium | Probabilistic/Sampled | Bug hunting & Regression |
| Full Formal Methods | Extremely High | (Universal) | Security Kernels / Hardware |
The Catalyst: Agentic Coding
The rise of agentic coding has fundamentally shifted our calculus. We have moved from a position of skepticism to one of genuine excitement, leading us to establish a dedicated team focused on formal methods.
Our objective is to transform formal methods into a tool as ubiquitous and practical for software engineering as advanced type systems are today.
Why AI Changes the Equation
Agentic coding disrupts the status quo in two primary ways:
- Reduced Friction: While AI agents cannot magically solve every complex proof, they significantly lower the barrier to entry, allowing a broader range of developers to utilize these tools effectively.
- The Verification Bottleneck: There is a stark disparity between "AI-generated code" and "production-ready code."
Models are adept at hitting a target goal, but they often neglect the overall health of the codebase. Agentic output tends toward slop: it is frequently over-engineered, riddled with edge-case bugs, and ignores critical codebase invariants.
The result? Humans must spend an increasing amount of time auditing AI code. Formal methods can alleviate this burden, streamlining the review process for both human-led coding and RL-based agent training.
Beyond Testing: The Power of
We recognize that tests, property-based testing, and fuzzing are vital. We have invested heavily in our own testing infrastructure. However, testing is inherently limited by the state space it can explore.
In our work with OxCaml, we've observed that agents thrive when provided with universal guarantees. In mathematical terms, we are looking for the (for all) quantifier:
- Data Races: A type system that prevents data races eliminates them entirely.
- Security: Types that render cross-site scripting (XSS) impossible are far more effective than any test suite.
By leveraging more powerful proof techniques, we can provide agents with superior feedback and a more robust framework for verification.
Why Jane Street?
Many startups are attempting to merge AI agents with formal methods. Why are we uniquely positioned to succeed?
1. Language Sovereignty We maintain deep control over our programming language. This allows us to modify the language itself to better support proof-oriented workflows. Potential paths include:
- Integrating modular property specifications into the type system.
- Implementing type-level constraints for ownership and mutability.
- Embedding proof techniques directly into the language syntax.
2. A Prepared Community We possess a developer base that is uniquely receptive to complex abstractions. In the world of PL (Programming Language) design, the challenge isn't inventing the idea—it's finding people willing to use it.
Our Roadmap
- Integrate modular specifications into the type system.
- Develop ownership/mutability constraints.
- Build native proof techniques into the language.
- Reduce the "verification bottleneck" for agentic code.
(* Conceptual example of a refined type constraint *)
type verified_code = {
implementation : string;
proof_of_correctness : proof_term;
}
We are excited to see how much "uplift" we can achieve by combining the efficiency of agents with the rigor of formal proofs.