Human vs. AI: Tracking Line-Level Provenance in Agentic Workflows

The us-vs-them project, developed by eighttrigrams, provides a mechanism for determining the origin of specific lines of text when a document is being edited by both humans and AI agents. Essentially, it answers the critical question: Who actually wrote this line—us or them?

🚩 The Problem

In the era of agentic coding and automated editing, knowing the provenance (origin) of a piece of text is vital. There is a fundamental difference in how we should treat different types of content:

Text authored or refined by a human should be treated as sacred. An AI agent should be extremely hesitant to modify such sections without a compelling reason. Conversely, "slop" generated by an agent is disposable and can be overwritten freely.

Real-World Scenarios

• Vibecoded Applications: You might have an app mostly built via AI prompts, but there are specific "corners" of the logic where you want to assert your own architectural ideas and ownership. You don't want a future agent session to bulldoze erase these critical human-defined sections.
• Documentation: Consider a README.md where an AI wrote the bulk of the technical details, but you personally rewrote the introduction. The agent should feel free to update the lower sections but should think twice before altering your opening paragraphs.

---

⚙️ How It Works

The system is designed with a primary constraint: no special markup is required within the text itself. Instead, it leverages the fact that every version of a document is created by an identifiable author (either a human or an agent).

The "Islands" Metaphor

The algorithm views the text as a "sea" of machine-generated content containing "islands" of human-authored lines.

Technical Logic

The tool is based on simple diffing, but it goes beyond tracking individual lines. It focuses on meaningfully coherent blocks of text. The algorithm accounts for:

1. Joining: Merging adjacent human-authored blocks.
2. Splitting: Breaking apart blocks when agents intervene.
3. Dilution: Tracking how much of a human's original intent remains after agent modifications.

The goal is to prevent the analysis from collapsing into a binary "all sea" or "all island" state. We can represent the provenance score $P$ of a range as: $0.0 \le P \le 1.0$ Where $P=1.0$ is purely human and $P=0.0$ is purely agentic.

---

🛠 Usage

To use us-vs-them as a command-line interface (CLI) tool, you will need bbin for the local installation.

Installation & Setup

• Install dependencies
• Run installation command:

make install

Running the Tool

Since a Git repository already maintains a history of versions and authorship markers, us-vs-them uses this history to calculate provenance.

Example Command:

us-vs-them --ours dan@eighttrigrams.net README.md

Understanding the Output

The tool returns a list of line ranges and their corresponding provenance scores:

Parameters

• --ours: Define who the humans are; everyone else is treated as an agent.
• --theirs: Define who the agents are; everyone else is treated as human.
• Tip: Use whichever flag results in the shorter list of identifiers.

---

🧪 Development

To run the test suite, use the following command: make test

For those wanting to understand the nuanced behavior of the provenance algorithm (especially regarding dilution and range splitting), the best resource is the caution_test.clj file.

Project Structure Summary:

• src/et/uvt: Core logic
• test/et/uvt: Test suites
• Makefile: Build and install scripts
• deps.edn: Clojure dependencies