LLM Classification Is Feature Engineering

Date: September 13, 2026
Topic: LLMs-as-classifiers

Using an LLM as a standalone classifier—where you feed a prompt into a context and receive a label—is a frustrating experience.

The irony is that these systems often perform reasonably well, which lures users into a suboptimal workflow. However, when measured against the requirements of a professional classifier, the "LLM-as-classifier" approach falls short.

The Gap: LLM vs. Ideal Classifier

---

The Paradigm Shift: LLM Classification as Feature Engineering

The failures mentioned above aren't because the LLM is "broken," but because we are using it for a task it wasn't designed for. The solution is to stop treating the LLM as the decision-maker and start treating it as a feature generator.

By wrapping the LLM's output in a traditional machine learning model, we combine LLM power with ML rigor.

The Mathematical Framework

Consider a simple logistic regression where the LLM's verdict is just one input:

$p(y = 1 \mid x) = \sigma(\alpha + \beta \cdot LLM(x))$

Note: If $\beta \rightarrow \infty$, this model effectively reverts to the original LLM classifier.

Instead of guessing, we estimate parameters using training data. This allows us to derive empirical probabilities:

$p(y = k \mid LLM(x) = 1) = \frac{\sum_{i} I(y_{i} = k \text{ and } LLM(x_{i}) = 1)}{\sum_{i} I(LLM(x_{i}) = 1)}$

Workflow Visualization

---

Revisiting Our Desiderata

By shifting to this framework, we can check off our original requirements:

• Calibration & Threshold Control: Because we use empirical proportions, the output is calibrated in expectation. We can now move the decision threshold to balance precision and recall.
• Information Integration: Since we are using a regression model, we can simply add other covariates (structured data) as additional features.
• Distribution Adaptation: The model is fit to the training set, meaning it automatically adjusts to the prevalence of the positive class in your specific population.
• Interpretability: While the LLM's internal logic remains opaque, we now know exactly how much weight ($\beta$) the LLM's verdict carries relative to other features.

---

How to Actually Improve Performance

When a standard LLM classifier fails, the common reaction is "tweak the prompt". Prompt engineering is often an arcane art with plenty of "tips" but very little scientific wisdom.

In the Feature Engineering Paradigm, we use standard ML improvement cycles:

1. Expand the Dataset

While the appeal of LLMs is "zero-shot" (training-free) classification, you need a test set to validate performance anyway. Adding a small training set is a minor inconvenience for a massive gain in reliability.

2. Refine the Features

Instead of guessing which adjectives in a prompt work better, we can:

• Screen features: Empirically verify if a specific feature consistently leads to a positive result.
• Evaluate features: Treat the feature itself as a secondary target to measure its utility.

3. Generate More Features

Don't just settle for a "Yes/No" label. You can extract more signal by:

• Using the log_probability of the verdict token.
• Running the prompt multiple times to check for consistency.
• Asking the LLM for "sub-verdicts" or intermediate reasoning steps.

4. Upgrade the Architecture

The model is now plug-and-play. If logistic regression is too simple, you can swap it for:

# Example: Moving from Logistic Regression to a more complex model
from xgboost import XGBClassifier

model = XGBClassifier()
model.fit(X_train, y_train) # X_train includes LLM verdicts + other features

You could even use a neural network or a strict rules-based system.