Analysis: The Struggle of LLMs with Tabular Data

Core Premise: While Large Language Models (LLMs) have revolutionized various AI domains, they remain surprisingly ineffective at predictive analytics for tabular data—a primary driver for the emergence of specialized tabular foundation models.

This research, authored by Marta Garnelo and Wojciech M. Czarnecki (arXiv:2608.02412), investigates the specific reasons why generic LLMs struggle with this common machine learning workload.

🔬 Experimental Methodology

The researchers tested a frontier LLM using a purest inference regime. This means the model operated under the following constraints:

Single generation pass over a prompt.
Full dataset inclusion (both training and test data) within the prompt.
~~No external tools~~
~~No agentic scaffolding~~
~~No fine-tuning~~

Hypotheses Tested

The study systematically evaluated five potential culprits for the LLM's failure:

Hypothesis A: Difficulty managing noisy or non-linearly-separable data.
Hypothesis B: The linear nature of CSV formats masking column relationships.
Hypothesis C: Issues stemming from the tokenization of numerical values.
Hypothesis D: The quantity of test points processed per single query.
Hypothesis E: The dimensionality of the input data.

📉 Key Findings: The Dimensionality Crisis

The results indicate that dimensionality is the decisive factor. By applying random linear projections across 31 different benchmark datasets, the authors observed a stark divergence between LLMs and traditional ML.

Performance Comparison

Model Type	Effect of Increasing Dimensionality	Accuracy Trend
Frontier LLM	Performance degrades as dimensions rise	$\downarrow$ Decreasing
Classical Baselines	Performance remains stable or improves	$\rightarrow$ or $\uparrow$

🧠 Behavioral Analysis

To understand how the LLM fails, the researchers compared it against 252 configured classical models.

Low Dimensionality (2D): The LLM behaves similarly to local, distance-based algorithms. The researchers found a grid agreement of: $\text{Agreement} = 91.6\%$
High Dimensionality: The LLM's behavior becomes anomalous. No classical model—even those intentionally corrupted with tuned, dimension-dependent noise—could replicate the LLM's specific prediction patterns.

🏁 Conclusion

The authors conclude that the LLM's predictive power dissolves as dimensionality increases. While they do not claim to have uncovered the exact internal mechanism, they have proven that this failure is not merely a result of "noise" that a classical learner might experience.

This explains why sophisticated LLMs continue to lose to $\text{50-year-old baselines}$ when dealing with tables, leaving the actual internal mechanism of these predictions as an open scientific question.

Metadata Summary

Submission Date: August 3, 2026
Subject: cs.LG (Machine Learning)
DOI: 10.48550/arXiv.2608.02412
Authors: Marta Garnelo, Wojciech M. Czarnecki