Claude, Please Stop Trying to Memorize Random Crap

Agentics: Why Memorizing Session Transcripts is a Waste of Time

By theahura

The core takeaway is simple: Keep track of your artifacts, not your scratchpad.

After extensive testing on Software Engineering (SWE) tasks, we have discovered that giving agents search access to their previous session transcripts provides zero performance benefit, provided they already have access to other relevant context. Furthermore, automatically mining these transcripts to improve context is largely useless unless a human is actively managing the process.

---

The Seduction of the "Transcript"

On the surface, it feels intuitive that transcripts are a goldmine. One might assume they contain:

• The rationale behind specific code implementations.
• The original user intent.
• A history of failed attempts and discarded approaches.
• General supplementary context to sharpen the agent's understanding.

I was once a true believer in this. I used to argue that session transcripts were the "new oil"—more precious than the actual source code. My company even developed a product based on this premise.

The Standard (and Flawed) Architecture

Many tools, including Claude Code, implement a memory system that looks something like this:

In this flow, the agent uses an MCP (Model Context Protocol) or a specific CLI skill to query the database for past conversations.

---

Why This Fails in Practice

Despite the logical appeal, our months of testing show that this approach doesn't help—and may actually hinder the model.

Distillation vs. Raw Data

We prioritize distillation. Instead of raw logs, we emphasize:

• High-quality commit messages.
• Detailed PR descriptions.
• Comprehensive technical documentation.

When an agent works on code, it is told to read the docs and previous PRs. Because the agent has already distilled the valuable "why" into these permanent records, searching transcripts results in the agent reading things it already knows, while simultaneously absorbing the "noise" that was intentionally left out of the documentation.

---

The Problem of "Intent Drift"

Agents lack a persistent internal state; they treat everything in their current context window as absolute truth. They cannot "garden" their own memory—they never decide to delete outdated or incorrect information.

We can represent the degradation of context as a form of Intent Drift: $\text{Context Quality} = \frac{\text{Ground Truth}}{\text{Ground Truth} + \text{Accumulated Noise}}$

As the agent autonomously builds a memory base, the "noise" (random decisions, unreviewed errors) grows, leading to:

1. Token Bloat: Wasting expensive tokens on nonsense.
2. Degraded Quality: The model follows a "hallucinated" intent from a previous failed session.
3. Alignment Conflict: Models are penalized for assuming input data is wrong, so they cling to corrupt memory.

"Agents don't have state, so they have to assume everything in their input context window is the ground truth."

---

A Better Path: Human-in-the-Loop Learning

This doesn't mean agents can't learn over time. We use our internal nori bots to perform weekly reviews of PRs, Slack threads, and Drive documents.

The Workflow:

1. Bot reviews company-wide activity.
2. Bot proposes updates to nori skillsets.
3. Human reviews the diff in Slack.
4. Human accepts or rejects the change.

We've found that roughly 80% of automatic updates would have actually made the model worse. In a large organization, automating this without human oversight would be unsustainable.

Final Verdict

I am now bearish on tools that simply index and surface session transcripts. While great for human observability, they are a liability for agent performance.

Want to dive deeper?

• Join our community Slack for agent experts.
• Explore noriagentic.com or norisessions.com.

---

Discussion Note: Some argue that "broad business context" (the history of why a line of code exists) is vital. While true, this is best handled by linking to story trackers or Slack threads that the LLM can follow intentionally, rather than dumping a raw transcript of every thought the agent ever had.