Treepeat: Advanced Code Similarity Detection via Tree-sitter

Treepeat is a specialized utility designed to identify similarities within a codebase. Unlike basic diff tools, it leverages Tree-sitter AST (Abstract Syntax Tree) analysis combined with locality-sensitive hashing to find redundant logic.

Note: This project is currently a ~~simple line-matcher~~ proof of concept. While the author is satisfied with its current state, support for additional languages is still growing. Contributions via Pull Requests are highly encouraged!

🛠️ How it Works

Treepeat doesn't just look at text; it understands the structure of the code. The process follows this general pipeline:

The similarity threshold can be expressed mathematically as: $\text{Similarity Score} = \left( \frac{\text{Matching Tokens}}{\text{Total Tokens}} \right) \times 100$

🔍 Detection Modes (Rulesets)

Depending on your goal, you can choose from three different normalization levels using the --ruleset flag:

Ruleset	Focus	Behavior
`none`	Exact Duplicates	Finds blocks meaningful to the language (e.g., identical functions/classes).
`default`	Near-Duplicates	Ignores whitespace, string literals, and high-level AST nodes like names.
`loose`	Structural Similarity	Anonymizes constants and identifiers to find logic clones.

🌐 Supported Languages

Treepeat currently supports a wide array of syntaxes:

Web: html, css, javascript, typescript, astro
Systems/General: rust, go, java, kotlin, python
Scripting/Config: bash, lua, yaml, sql, markdown

🚀 Getting Started

Installation

You can install the tool directly via pip:

pip install treepeat

Quick Start Checklist

Install treepeat
Identify the target codebase path
Select a ruleset (none, default, or loose)
Run the detect command

Usage & Command Line Interface

The primary command is treepeat detect. Below are the available configuration flags:

--ruleset: Defines the normalization logic (none, default, loose).
--similarity: Sets the match threshold from $1$ to $100$ (defaults to $100$ ).
--min-lines: The minimum size of a block to be considered a match (default: $5$ ).
--diff: Generates a side-by-side visual comparison of the similar blocks.
--format: Choose between console (standard) or sarif (for CI/CD pipelines).
--verbose: Enables detailed metrics and stage-specific timing.
--progress: Displays tqdm progress bars (sent to stderr to avoid polluting stdout).

Example Commands

1. Find exact clones:

treepeat detect /path/to/codebase

2. Find 80% similar blocks:

treepeat detect --similarity 80 /path/to/codebase

3. Deep structural analysis with diffs (min 10 lines):

treepeat --ruleset loose detect --diff --min-lines 10 /path/to/codebase

4. CI-ready output with progress tracking:

treepeat detect --format sarif -o results.sarif --progress /path/to/codebase

⚙️ Additional Functionality

Rule Exploration

To understand exactly what is being normalized, use the list-ruleset command: treepeat list-ruleset

Tip: Use the --language flag to filter rules for a specific programming language.

Debugging Normalization

The treesitter command allows you to see how your code is being transformed into tokens: treepeat treesitter [file]

This provides a side-by-side view of the Original Source $\leftrightarrow$ Normalized Tokens, which is invaluable for debugging why two blocks are (or aren't) matching.

🛠️ Development & Performance

For those looking to contribute or benchmark the tool:

Setup: Run make setup to prepare the environment.
Testing: Run make test to execute the test suite.
Benchmarking: A performance harness is available at tools/perf/harness.py for testing against large-scale repositories.

License: Distributed under the Apache-2.0 license.