Creating Your Own Decision Model

By Nish Tahir — October 10, 2026

"System one" decision models are designed to infer and respond using calibrated probabilities across a set of permissible answers.

The Efficiency Gap: Standard LLMs vs. Decision Models

In a typical language model workflow, if you require a specific format (like JSON), you might employ Structured Output to ensure the result is syntactically valid. However, even with a prefilled prompt, the model must perform a separate forward pass for every single token it generates.

For instance, producing a simple JSON response might require $\approx 11$ passes.

Conversely, decision models (such as Jev) operate on the premise that there is a fixed set of options. This allows the model to arrive at an answer in a single pass.

How it works:

1. The output space is restricted to specific tokens (e.g., A, B, C, D, E).
2. All other tokens in the vocabulary are allowed masked.
3. The token with the highest probability is selected as the final answer.

[!IMPORTANT] While this method guarantees the output will be one of the choices, it guarantees correctness does not guarantee the answer is actually right. Furthermore, raw token probabilities often reflect the model's confidence in the next token rather than the truth of the answer.

---

Implementation Guide

You can replicate this behavior by constraining the output tokens of an LLM. In this walkthrough, I used Qwen/Qwen3-1.7B.

🛠️ Development Checklist

• Load Pre-trained Model and Tokenizer
• Define valid option tokens
• Format prompt for multiple-choice
• Apply softmax to constrained logits

The Code

import argparse
import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-1.7B"
options = ["A", "B", "C", "D", "E"]

parser = argparse.ArgumentParser()
parser.add_argument("--input", default="question.json")
args = parser.parse_args()

# Initialize model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype="auto", device_map="auto"
)

# Map options to their respective token IDs
option_token_ids = [tokenizer.encode(opt, add_special_tokens=False)[0] for opt in options]

def format_prompt(item):
    prompt = item["question"] + "\n"
    for opt in options:
        prompt += f"{opt}. {item[opt]}\n"
    prompt += "Answer:"
    messages = [{"role": "user", "content": prompt}]
    return tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
    )

with open(args.input) as f:
    item = json.load(f)

model_inputs = tokenizer(format_prompt(item), return_tensors="pt").to(model.device)

with torch.no_grad():
    logits = model(**model_inputs).logits[0, -1]

# Constrained decoding: calculate softmax only for the allowed option tokens
probs = torch.softmax(logits[option_token_ids].float(), dim=-1)
print(f"prediction: {options[probs.argmax().item()]}")

for opt, prob in zip(options, probs.tolist()):
    print(f"{opt}: {prob:.4f} {item[opt]}")

Example Execution: If we provide the following input: { "question": "What color is the sky?", "A": "Red", "B": "Blue", "C": "Green", "D": "Purple", "E": "I don't know" }

The output is:

• Prediction: B
• A: 0.0000
• B: 0.9988
• C: 0.0000
• D: 0.0000
• E: 0.0012

---

Performance Evaluation

To validate the approach, I tested the model against a random holdout sample from the CommonsenseQA dataset.

Baseline Results (1.7B Model):

After performing a quick finetune on the dataset, the performance improved:

---

The Calibration Challenge

A significant issue arises when the model faces ambiguity. Consider this prompt:

Input: "Where would you most likely find a bat?" Options: A: Cave, B: Baseball game, C: Attic, D: Zoo, E: Sporting goods store. Result: Prediction A (Confidence: $0.9978$)

In reality, there is no single correct answer here, yet the model is extremely overconfident. By binning the confidence scores from the previous evaluation, we can see a clear mismatch between confidence and actual accuracy:

Observations:

• In the $0.9 - 1.0$ confidence range, the model is only correct $\approx 70\%$ of the time.
• In the $0.8 - 0.9$ range, accuracy drops to $\approx 40\%$.

To fix this, we can use temperature scaling. By adjusting the temperature, we can flatten the probability distribution, scaling the output so that the confidence scores more accurately reflect the actual probability of correctness.