I Accidentally Turned LLM Memory into Program Analysis

By Jordy Zomer | August 28, 2026 | ⏱️ 19 Min Read

For several months, I've been experimenting with LLM agents, specifically tailoring them for vulnerability research. While these models have become impressively adept at parsing massive codebases, clarifying obscure subsystems, and mapping out potential attack surfaces, I hit a recurring wall during long-term investigations.

The "Memory Decay" Problem

Once a research session stretches over several hours, the model begins to lose its grip on the established state of the investigation. I noticed several failure modes:

• It would suggest a path we had already debunked.
• It would forget that a previous assumption was proven false.
• It would continue to reason based on an observation that was no longer valid.

"Telling an LLM that something is wrong does not necessarily mean that it will stop believing all of the things that depended on it :)"

To combat these hallucinations and make LLMs more viable for complex security research, I began exploring memory systems.

---

Why Standard Memory Systems Fail

Most current LLM memory solutions rely on a similar architecture:

1. Store conversation history or observations.
2. Convert them into embeddings.
3. Retrieve the most "relevant" snippets via vector search.

While this works for general tasks, it is insufficient for vulnerability research. Consider this logical chain:

• Attacker controls object_a
• object_a points to object_b
• object_b is a kernel object
• Conclusion: The attacker can control a kernel object.

In a standard RAG (Retrieval-Augmented Generation) system, these are just stored as text. If I discover via LLDB two hours later that object_a actually does not point to object_b, my memory now contains:

1. object_a points to object_b
2. Attacker can control object_b
3. object_a does not point to object_b

We are then left hoping the LLM is smart enough to resolve this contradiction and invalidate all dependent conclusions.

The Epiphany: This is Program Analysis

This problem felt familiar because it mirrors the core of program analysis. In that field, we manage a set of facts and apply rules to derive new ones.

Example Logic: If we have the facts $\text{calls}(foo, bar)$ and $\text{calls}(bar, baz)$, we can apply a reachability rule: $\forall x, y, z : (\text{calls}(x, y) \land \text{calls}(y, z)) \implies \text{reaches}(x, z)$

The goal is to reach a fixed point where no more facts can be derived. Crucially, if an input fact changes, we don't restart the whole process; we only update the affected results.

I realized I wanted exactly this for my LLM: automatic invalidation of dependent conclusions.

---

Enter: Datalog

Instead of forcing the LLM to reconstruct its state from a transcript, I decided to build a Datalog engine for it.

What is Datalog?

Datalog is a declarative language. Instead of writing a step-by-step algorithm, you define facts and rules.

The Setup:

• Fact: controls(attacker, object_a).
• Rule:

  controls_kernel_object(Attacker) :- 
      controls(Attacker, ObjectA), 
      points_to(ObjectA, ObjectB), 
      kernel_object(ObjectB).
  

• Derived Fact: controls_kernel_object(attacker).

If we later retract points_to(object_a, object_b), the engine knows exactly which derived facts are now unsupported and removes them automatically.

The Division of Labor

I created a system called Lemmalog to split the workload between the "fuzzy" and "deterministic" parts of the process:

The Workflow:

1. LLM observes: "LLDB shows that the freed object is later reused as the destination of the write."
2. LLM outputs: freed(object_a), reused_as(object_a, write_target)
3. Lemmalog processes: Updates the database and derives all logical consequences.

---

The Challenge of Retractions

One of the most complex hurdles was implementing fact removal.

Adding a fact is easy: add it and see what new rules trigger. However, removing a fact is nuanced. If a conclusion $C$ is derived from both fact $A$ and fact $B$, removing $A$ should not necessarily remove $C$, because $B$ still supports it.

Logic of Retraction:

• If $A \rightarrow C$ and $B \rightarrow C$:
  • Remove $A \implies C$ remains (via $B$).
  • Remove $A$ AND $B \implies C$ is deleted.

This is vital for vulnerability research. A specific exploit path (e.g., candidate_3_is_exploitable) might be supported by multiple independent primitives. If one primitive is debunked, the overall exploitability may still be true due to an alternative path.