🦀 SearXNG in Rust: `metadata-search-engine-rs`

This project is a high-performance metadata search engine implemented in Rust, drawing inspiration from the architecture of SearXNG. It acts as a privacy-respecting aggregator that fetches results from various sources and unifies them into a single, ranked list.

~~Traditional sequential searching is slow.~~ This engine utilizes concurrent fan-out to ensure minimal latency.

🛠️ System Architecture

The engine operates by broadcasting a single user query to multiple upstream providers simultaneously.

Logic Flow

Core Mechanisms

Scraping: It employs the scraper crate, utilizing CSS selectors built upon Mozilla's html5ever.
Normalization: To prevent duplicate entries, URLs are cleaned by:
1. Stripping tracking parameters.
2. Removing locale-specific prefixes.
3. Sorting query parameters alphabetically.
Ranking: The engine uses Reciprocal Rank Fusion (RRF) to determine the final order. The mathematical formula for the score is:

$\text{score} = \sum_{e \in \text{engines}} \frac{1}{60 + \text{rank}_e}$

This ensures that pages appearing across multiple search engines are boosted to the top.

🚀 Getting Started

Prerequisites

Rust version 1.75 or higher
Cargo package manager

Installation Options

1. Integration as a Library

You can add the crate directly to your project: cargo add metadata-search-engine-rs

Alternatively, modify your Cargo.toml:

[dependencies]
metadata-search-engine-rs = "0.1"

2. Deployment as a Server

To run the standalone server from the source code:

# Clone the repository
git clone https://github.com/MikeLuu99/searxng-rust
cd metadata-search-engine-rs

# Build for production
cargo build --release

# Execute with environment variables
PORT=8080 MAX_RESULTS=20 cargo run --release

Tip: To enable verbose logging, prepend the command with RUST_LOG=debug.

💻 Implementation Examples

Single Engine Query

Use this approach when you only need results from one specific provider.

use std::sync::Arc;
use metadata_search_engine_rs::engines::{DuckDuckGoEngine, SearchEngine, build_http_client};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Arc::new(build_http_client()?);
    let engine = DuckDuckGoEngine { client };
    
    // Perform the search
    let results = engine.search("rust lang", 10).await?;
    for res in results {
        println!("Found: {}", res.url);
    }
    Ok(())
}

Full Aggregation (RRF)

This demonstrates the "fan-out" pattern to query all available engines.

use std::sync::Arc;
use metadata_search_engine_rs::{
    aggregator::{aggregate, query_all_engines},
    engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, StartpageEngine, YahooEngine, build_http_client},
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Arc::new(build_http_client()?);
    let engines: Vec<Arc<dyn SearchEngine>> = vec![
        Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }),
        Arc::new(BraveEngine { client: Arc::clone(&client) }),
        Arc::new(StartpageEngine { client: Arc::clone(&client) }),
        Arc::new(YahooEngine { client: Arc::clone(&client) }),
    ];

    let (successes, failures) = query_all_engines(engines, "rust programming", 10).await;
    
    for fail in failures {
        eprintln!("Error: {}", fail);
    }

    let results = aggregate(successes, 10);
    for r in results {
        println!("Score: {} | URL: {}", r.score, r.url);
    }
    Ok(())
}

🌐 API Reference

Endpoints

Method	Endpoint	Description
`GET`	`/health`	Checks if the service is operational.
`GET`	`/search?q={query}`	Performs the aggregated search.

Sample Search Response

curl "http://localhost:3000/search?q=rust"

{
  "query": "rust",
  "results": [
    {
      "title": "Rust Programming Language",
      "url": "https://rust-lang.org/",
      "snippet": "A language empowering everyone to build reliable and efficient software.",
      "engines": ["duckduckgo", "brave", "startpage", "yahoo"],
      "score": 0.049
    }
  ],
  "engines_queried": ["duckduckgo", "brave", "startpage", "yahoo"],
  "engines_failed": []
}

Error Handling

Status	Scenario	JSON Body
`400`	`q` parameter missing	`{"error": "query parameter 'q' is required"}`
`400`	`q` parameter is empty	`{"error": "query parameter 'q' cannot be empty"}`
`503`	Total engine failure	`{"error": "all engines failed to respond"}`

🧪 Testing Suite

The project includes comprehensive tests to ensure stability.

Unit Tests: Run all tests via cargo test.
Modular Tests:
- cargo test normalizer (URL cleaning)
- cargo test aggregator (RRF logic)
- cargo test engines::brave (Specific engine parsing)
Integration Tests: Since live tests hit real websites, they are marked with #[ignore]. Run them manually: cargo test -- --ignored test_live

🛠️ Extending the Engine

To integrate a new search provider, follow these steps:

Create a new file in src/engines/name.rs.
Define a struct that contains an Arc<reqwest::Client>.
Implement the SearchEngine trait:
- Define the name() method (returns a &'static str).
- Implement the search() method to handle the request and parse HTML.

Note: A Terminal User Interface (TUI) built with ratatui is also available as a separate crate.

🦀 SearXNG in Rust: metadata-search-engine-rs