Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
DAYJOB: Tax — Benchmarking AI Agents on Professional Tax Work
In the professional world, instructions are rarely perfect, records often contradict one another, and the most critical issues are frequently the ones the client forgot to mention.
DAYJOB: Tax is designed to test whether AI agents can survive this reality. Rather than simply answering a prompt, these agents must take full ownership of a tax professional's workflow—moving from an initial client request to a finalized, professional deliverable.
Across 50 distinct environments and a library of 1,789 files, agents are forced to sift through complex records, identify hidden risks, and make high-stakes judgments based on incomplete data. Success in this benchmark requires more than just raw tax knowledge or the ability to do math; it requires professional judgment.
The Gap Between "Correct" and "Professional"
To illustrate the difference, consider the case of Anna.
Case Study: Anna's Condo Sale Anna planned to sell her condo after moving out in August 2024. She asked an AI agent if closing the deal in October 2026 or January 2027 would be more tax-advantageous.
The AI agent correctly identified that both dates would allow her to meet the residence requirement for a tax exclusion of up to \text{\250,000}$ in gains. Under standard rules, a homeowner must reside in the property for at least two of the five years preceding the sale. Since Anna left in August 2024, she would remain eligible until roughly August 2027.
While the agent's answer was technically accurate, it failed the "professional" test. It didn't warn Anna that if the sale slipped past August 2027, she would lose a massive financial benefit. A basic chatbot answers the question asked; a professional adviser identifies the risks the client doesn't know to ask about.
Introducing the DAYJOB: Tax Benchmark
DAYJOB: Tax is the newest expansion of the Surge AI DAYJOB family. It focuses on the intersection of tax preparation, compliance, and advisory services.
Benchmark Architecture
The benchmark consists of 50 environments spanning eight specialized tax domains. Agents are provided with client requests and a messy set of supporting documents to produce:
- Tax returns
- Detailed workpapers
- Technical reviews
- Strategic client advice
What is being tested?
The evaluation goes beyond simple retrieval. Agents must:
- Determine which pieces of evidence require deeper investigation.
- Resolve conflicts between different records.
- Identify when the provided information is insufficient to make a call.
- Translate these findings into a professional-grade final product.
Performance Analysis
The Leaderboard
Current results show that while top models are improving, professional tax work remains an immense challenge.
| Model | Pass Rate |
|---|---|
Claude Sonnet 5.5 | 10.8% |
Claude Fable 5.1 | 9.6% |
Claude Opus 5.5 | 9.2% |
Note: A "pass" is defined as satisfying of the task's specific criteria (calculations, evidence, and deliverables).
Efficiency and Cost
Surge AI also analyzed the Pareto Frontier, comparing the cost per trial against the accuracy of the model. Claude Sonnet 5.5 and Opus 5.5 currently define the high-performance ceiling of this frontier.
Scope and Complexity
Tax Practice Areas
The benchmark covers eight diverse areas of expertise:
- Individual and Family Tax
- Small-Business Tax
- Partnerships
- Property and Investments
- Estates, Gifts, and Trusts
- International Tax
- Corporate Transactions
- Tax Review and Disputes
The Data Landscape
The 1,789 workspace files (median of 30 per task, maxing at 123) include a wide array of formats:
- PDFs: 1,487
- Excel Workbooks: 117
- Word Docs: 38
- Others: Emails, CSVs, JSON, images, and text files.
These files mimic real-world chaos, including K-1s, W-2s, general ledgers, and IRS notices.
Real-World Examples of Complexity:
- Visual Evidence: One task includes a photo of a handwritten receipt. The agent must extract the data, reconcile it with spending logs, and calculate the impact on Cost of Goods Sold (COGS).
- Hidden Obligations: A company believes it only owes taxes in Alabama, but its field dispatch logs reveal crews working in five different states.
- Global Assets: A household with Polish employment records that must be reconciled with cryptocurrency wallet histories and purchase lots.
Expert-Driven Development
The benchmark is not synthetically generated; it is built from actual professional workflows.
The experts involved bring deep experience from firms like PwC, Deloitte, and companies like Exxon, specializing in private equity, corporate taxation, and multistate compliance.
