The AI Capability Gap: Mapping US Economic Tasks to Benchmarks

A report by Mercor Research Enterprise Experts Company

While the number of AI benchmarks has exploded recently, there is a significant disconnect between what these tests measure and the actual work performed in the US economy. To quantify this, we developed a Benchmark Map to analyze how well current AI evaluations cover American job functions.

This initiative serves as the foundation for a dynamic, evolving resource designed to track AI's true capabilities across various economic sectors.

The Scope of the Analysis

Our research utilized a massive dataset to determine coverage:

• 923 distinct occupations.
• 18,797 individual tasks.
• 29 curated benchmarks within our index.

Coverage by Job Family

The following table illustrates the share of occupations within each O*NET job family that are represented in our benchmark index.

Key Finding: Only 208 of the 923 occupations have at least one task covered by $\ge 5$ benchmark prompts. Furthermore, 6 out of the 22 job families have no occupations meeting this threshold.

---

The "90% Gap"

Our analysis reveals a startling reality regarding the utility of current AI testing:

Roughly 90% of all American working hours are spent on tasks that have little to no representation in existing AI benchmarks.

This creates a critical blind spot. If we cannot measure the tasks people actually do, we cannot accurately determine how effective AI models are at performing real-world professional work.

The Correlation Between Wages and Coverage

We observed that the AI community tends to prioritize the largest labor markets. We used a task-level logistic regression to estimate the probability of coverage based on AI exposure and the occupational wage bill (calculated via BLS mean wages and employment numbers, assuming a 40-hour week).

$\text{Probability of Coverage} = f(\text{AI Exposure}, \log(\text{Wage Bill}))$

The Result: High-wage, high-exposure roles are well-covered, while the "long tail" of middle-income knowledge work is ignored.

Example:

• Lawyers (High wage bill) $\rightarrow$ High benchmark coverage for AI-exposed tasks.
• Legal Secretaries/Court Clerks (Lower wage bill) $\rightarrow$ Almost zero coverage, despite similar AI exposure.

---

Methodology

1. Taxonomy and Task Mapping

We utilized the O*NET taxonomy from the US Department of Labor. This system breaks the workforce into 923 occupations and 18,797 specific tasks.

Example: Lawyer Tasks (O*NET)
1. Draft/review legal documents (wills, contracts, etc.)
2. Advise clients on liability and legal rights
3. Interpret laws and regulations
4. Analyze case outcomes via legal precedents
5. Present evidence in litigation
6. [Additional tasks...]

2. Benchmark Selection

We selected 29 benchmarks based on specific quality and economic criteria:

• Public Availability: Must have $\ge 25$ public tasks.
• Agentic Focus: Must require the model to perform a task, not just provide advice.
• Enterprise Relevance: Tasks must mirror real-world business requirements.
• Economic Value: Prioritized over purely academic or theoretical complexity.
• Balanced Distribution: Capped benchmarks per family to prevent software engineering from skewing the data.

Note: This includes Mercor's proprietary APEX benchmarks.

3. The Mapping Process

We employed a specific pipeline to connect benchmarks to the economy:

An O*NET task is officially considered covered if the LLM judge maps it to at least 5 prompts from our index. To calculate the total share of working hours, we integrated these findings with time-share estimates from Hatgis-Kessell et al.

---

Limitations

Our findings are absolute and exhaustive. In reality, our estimates are based on the specific subset of 29 benchmarks we selected. This sample, while rigorous, is not a perfectly representative mirror of every possible AI benchmark in existence.