Mitigating "Doom Loops" via Final Token Preference Optimization

Repetitive degeneration—colloquially known as "doom loops"—is a frustrating failure mode during LLM inference. This occurs when a model generates a specific phrase (e.g., "Wait, let me reconsider...") and then enters a cycle of repeating that same sequence indefinitely until the context window is completely filled.

This phenomenon is particularly prevalent in small reasoning models when they tackle complex problems or generate extensive thinking traces.

The Current State of Fixes

Historically, developers have relied on a few methods to stop these loops, but they are often insufficient:

• repetition_penalty: A common inference-time tweak that re-weights the output distribution. While helpful, it is merely a band-aid and can actually harm overall model performance.
• Reinforcement Learning (RL): Can target loops, but requires expensive online rollouts and meticulously tuned reward functions.

The Alternative: Instead of a broad brush, we identify the exact token that triggers the loop and train the model to prefer a coherent alternative at that specific coordinate, leaving the rest of the probability distribution intact. This approach, termed Antidoom, utilizes Final Token Preference Optimization (FTPO).

Impact: On an early version of the LFM2.5-2.6B model, loop rates on difficult coding and math tasks plummeted from 10.2% $\rightarrow$ 1.4%, leading to a general increase in evaluation scores.

---

🔍 Anatomy of a Doom Loop

Doom loops are not random; they are the result of three intersecting mechanisms:

Mechanism 1: Overtrained Tokens & Uncertainty

Certain tokens are "over-represented" in the vocabulary, often due to the use of synthetic training data. Examples include words like delve or testament. In reasoning models, these are often discourse markers:

• "Wait"
• "Alternatively"

While these tokens are useful for self-correction, they become dangerous fallbacks when the model is uncertain. Instead of progressing, the model defaults to these high-prior tokens, restarting the same flawed reasoning loop.

Common Loop-Starting Tokens (LFM2.5-2.6B):

Mechanism 2: Contextual Reinforcement

Once a sequence is repeated, the prior context makes that same sequence even more likely to appear again. This creates a feedback loop where the probability of the looping tokens approaches $1.0$. This is often associated with a V-shaped attention pattern, where semantic repetition (stuck on an idea) precedes textual repetition (stuck on words).

Mechanism 3: The Trap of Greedy Sampling

Reasoning models usually operate at a low temperature for stability. At $T=0$, the model always picks the most likely token. If Mechanism 2 has pushed a loop token's probability to the top, there is no escape. Even at $T=0.67$, loops persist because the probability mass for all other tokens has been effectively erased.

---

🛠️ Locating the Failure

To cure the loop, we first had to find it. We used a specialized prompt mix (LiquidAI/antidoom-mix-v1.0) and mined the failures using these criteria:

• Repetition Count: A section must repeat $\ge 4$ times.
• Length Constraint: The repeating section must be $\ge 60$ characters.

$\text{Loop Detection} = (\text{count} \ge 4) \land (\text{length} \ge 60\text{ chars})$

Once a loop is found, we target the first token of the first repetition. We then extract the top-$k$ log-prob alternatives from the base model, filtering out noise to find up to 20 plausible "chosen" substitutes.

Training Data Structure:

# Each training row is a tuple:
training_sample = {
    "prefix": "The reasoning trace leading up to the loop...",
    "rejected_token": "Wait", 
    "chosen_tokens": ["Therefore", "However", "Consequently"]
}

Note: We apply regularization to prevent the model from over-suppressing common tokens like "the" or "so," which would degrade general reasoning.

---

🚀 Final Token Preference Optimization (FTPO)

FTPO is a specialized preference-optimization algorithm derived from DPO (Direct Preference Optimization). While DPO typically looks at full sequences, FTPO is a "scalpel" designed for surgical changes to the distribution.

Key Differences: FTPO vs. DPO

1. Final Token Focus: It only trains the trailing token of a sequence that is already mid-generation.
2. Many-to-One Mapping: Instead of one chosen completion, it uses multiple chosen tokens to spread probability across a group of alternatives.
3. Logit-Space KL Loss: To avoid disturbing unrelated tokens, it computes divergence from the reference model directly in the logits, omitting the softmax layer.
4. Dual Regularization: It allows the specific chosen/rejected tokens to move more freely while keeping the rest of the vocabulary anchored.

By focusing exclusively on the "pivot point" of the doom loop, FTPO effectively breaks the cycle without sacrificing the model's underlying intelligence.