Slopo: Detecting Non-Exact Code Duplication via Embeddings

Slopo is a streamlined command-line interface (CLI) utility designed to uncover code duplication that isn't necessarily identical. By leveraging embedding models, it identifies "fuzzy" duplicates that traditional tools often miss.

The Core Problem: While exact copy-paste is easily caught by standard tools, and nearby duplicates are obvious to developers or AI, the most dangerous duplication is the kind where snippets are written similarly but reside in distant modules or far apart within massive files.

For a more comprehensive overview, visit slopo.dev.

🌐 Supported Ecosystems

Slopo works across a variety of popular languages:

• Web: TypeScript, JavaScript
• JVM: Java, Kotlin
• Systems/General: Python, Go, Rust, C#

---

⚙️ How It Works

Unlike traditional tools that look for character-for-character matches, Slopo uses a vector-based approach.

1. Vectorization: It generates an embedding for every discrete unit of code.
2. Proximity Search: It identifies pairs of embeddings that are mathematically "close."
3. Clustering: Similar units are grouped into clusters, which are then ranked based on their similarity score and their physical distance within the project.

Note: Because similar logic doesn't always mean a duplicate, these clusters are potential candidates for refactoring. If two pieces of code achieve the same goal but use entirely different implementation patterns, their embeddings will be distant, and Slopo will not flag them.

The mathematical foundation relies on Cosine Similarity, calculated as: $\text{similarity} = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|}$

---

🚀 Quick Start Guide

Follow these steps to get Slopo running on your machine:

• Installation Use uv (a high-performance Python package manager) to install Slopo into an isolated environment:

  uv tool install slopo
  

• Initialization Generate a configuration template:

  slopo init
  

• Configuration Define your source directory and choose an embedding model. Slopo is compatible with any provider supported by LiteLLM. For security, it is recommended to store your API key as an environment variable.

• Execution Verify your settings and run the analysis pipeline:

  slopo show-config  # Validate parameters
  slopo index       # Index the codebase
  slopo embed       # Generate embeddings
  slopo analyze     # Produce the duplication report
  

---

🔄 The Professional Workflow

In a production environment, Slopo is best used as a loop involving an AI coding agent.

Key Workflow Details:

• Incremental Updates: Slopo supports re-indexing only changed files to save time.
• Filtering: Use slopo.ignore.txt to track reviewed clusters so they don't reappear in future reports.
• Refining: If the report is too noisy or too sparse, adjust the similarity thresholds.
• Collaboration: Both the .ignore file and the config file (minus the API key) can be committed to Git for team-wide consistency.

---

🛠 Configuration Reference

You can explore options via slopo --help or slopo show-config. While most settings are optional, some critical parameters require a full reset (deleting slopo.db and the index/embed folders) if changed after the initial index.

Critical Parameters (Require Reset if Changed)

• source_dir
• embedding_model
• embedding_dimensions
• body_node_count_threshold

Full Parameter Table

Pro Tip: Use the SLOPO_EMBEDDING_API_KEY environment variable or a .env file to manage your credentials securely.