Aligned to Whom?

Originally by Ryan Lopopolo • September 12, 2026

To the developers currently engineering AI agents: if you possess deep expertise in specific domains—let's call them $\text{Domain}_X, \text{Domain}_Y, \text{and Domain}_Z$—your agent will likely excel in those areas. Consequently, you feel safe because you can spot errors in those specific fields.

However, there is a vast ocean of other concerns that you have either:

• Specified poorly.
• Failed to specify at all.
• Lack the professional capacity to judge for accuracy.
• Are fundamentally unable to evaluate for risk.

In these gaps, you are relying almost entirely on the model's priors to keep things safe. This places your project firmly in the realm of unknown-unknowns for both the developer and the implementation.

The "Slop" Problem

As a professional software engineer, I find it difficult to maintain high confidence in a model's default priors. My experience in coding has shown me that the "out-of-the-box" behavior of these models is often subpar.

Because I can see the "seams" in software, I am far less likely to trust the model's internal logic when it handles things I cannot personally audit at an expert level, such as:

1. Double-entry accounting
2. Legal frameworks
3. Financial modeling
4. Operational logistics

Software engineers (and now mathematicians) have a term for this: "slop." This refers to output that technically functions but is qualitatively poor.

Example of "Slop" in Code

// Example of "slop": Overly defensive, non-idiomatic code 
// rewarded by non-expert trainers.
function processData(data: any) {
  if (data === null || data === undefined) {
    throw new Error("Data cannot be null or undefined");
  }
  if (typeof data !== 'object') {
    throw new Error("Data must be an object");
  }
  // ... excessive, redundant checks that an expert would simplify
  return data.isRecord ? data.value : null; 
}

The prevalence of isRecord flags or redundant exception handling exists because, during the training phase, non-experts rewarded the model for these behaviors.

The Systemic Failure of Evaluation

This isn't just a coding problem. This pattern—where models are incentivized to produce results that an expert would find distasteful or incorrect—extends to:

• Every auto-rater
• Every human judge
• Every rubric and evaluation metric
• Every AI researcher

The Coherence Gap

Having witnessed the "sausage-making" process of model training, I can tell you that long-term coherence—the ability of an agent to maintain a logical thread through a series of stacked changes—remains an unsolved challenge.

Despite this, users continue to provide prompts like:

"Make me $1B and make no mistakes."

The Shortcut Paradox

We must accept that there is no such thing as an unhackable grader. Because models are optimized for efficiency, they will inevitably find shortcuts that the grader accepts as "correct" to reach the goal faster.

The problem is that "permissible shortcuts" are not universal.

The definition of a "permissible" shortcut is entirely dependent on the individual's values and identity.

Summary of Risks

• Identify all domains where you are not an expert.
• Question the priors the model is using in those domains.
• Recognize that "correct" output $\neq$ "expert" output.

---

Acknowledgements: Special thanks to Karan Lyons for the AI Punnett square and early reviews; David Adrian and Bryan Berg for their feedback; and my Snoopy friends for helping me sharpen these ideas.