TurboKV: An Ultra-Fast Rust Key-Value Store

TurboKV is a high-performance, asynchronous, embedded key-value database engineered in Rust. It provides a robust suite of features including atomic batching, ordered range scans, adjustable durability levels, data compression, and automated background compaction.

---

🚀 Getting Started

To integrate TurboKV into your project, follow these steps:

• Add the core library: cargo add turbokv
• Add the async runtime: cargo add tokio --features full

Alternatively, modify your Cargo.toml manually:

[dependencies]
turbokv = "0.6"
tokio = { version = "1", features = ["full"] }

🛠️ Hardware Acceleration

TurboKV utilizes hardware-based AES for its persisted Bloom-filter format. To maximize performance, compile with the following RUSTFLAGS:

• x86/x86_64: RUSTFLAGS="-C target-feature=+aes,+sse2"
• ARM/AArch64: RUSTFLAGS="-C target-feature=+aes,+neon"
• Generic: Use -C target-cpu=native if the binary is intended for the local machine or a compatible CPU superset.

---

💻 Quick Start Example

use turbokv::{Db, DbOptions, WriteBatch};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize the database with durable settings
    let db = Db::open_with_options("./my-database", DbOptions::durable()).await?;

    // Basic insertion
    db.insert("key1", "value1").await?;

    // Atomic batch operations
    let mut batch = WriteBatch::new();
    batch.put("key2", "value2");
    batch.delete("key1");
    db.write_batch(batch).await?;

    // Retrieval
    if let Some(value) = db.get("key2").await? {
        println!("Found: {}", String::from_utf8_lossy(&value));
    }

    Ok(())
}

Available Runnable Examples

• basic: Fundamental CRUD operations.
• batch_writes: Implementing atomic puts and deletes.
• range_queries: Performing prefix scans and ordered range reads.
• concurrent: Managing shared access via Tokio tasks.
• persistence: Testing WAL recovery in "paranoid" mode.
• configuration: Tuning cache, memtables, and compression.

---

🏗️ Architecture & Data Flow

The following diagram illustrates how data moves from the application into permanent storage:

---

🛡️ Durability & Persistence

TurboKV offers three distinct durability presets to balance speed and safety:

[!IMPORTANT] Note on durable() mode: The WAL is not left unsynchronized indefinitely. Synchronization occurs during explicit/background memtable flushes, clean shutdowns, or when a WAL segment (typically $1 \text{ GiB}$) rotates.

Technical Thresholds

• Memtable Rotation: $\approx 64 \text{ MiB}$
• Flush Check Interval: $60 \text{ seconds}$
• WAL Segment Rotation: $1 \text{ GiB}$

Dropping a database handle is not a clean shutdown. To ensure data integrity, always use close() or close_with_status().

---

📖 API Reference

Database Initialization

Keys and values are handled as arbitrary byte sequences via AsRef<[u8]>. Note that an empty value is considered valid data and is distinct from a key that has been deleted.

Default Configuration:

• Memtable: $64 \text{ MiB}$
• Block Cache: $64 \text{ MiB}$
• Compression: LZ4

DbOptions Field Details

• wal_enabled: bool $\rightarrow$ If false, data may be lost during a process crash.
• sync_writes: bool $\rightarrow$ Forces a WAL sync barrier before acknowledging mutations.
• memtable_size: usize $\rightarrow$ Byte threshold for rotation/flush.
• block_cache_size: usize $\rightarrow$ Budget for decompressed SSTable blocks.
• compression: Compression $\rightarrow$ Options: Lz4, Snappy, Zstd, or None.

Data Operations

WriteBatch Specifics

A WriteBatch maintains its own copies of keys and values.

• Atomicity: Readers see either the state before the batch or the state after the entire batch.
• Constraints: When the WAL is active, a single record or the entire batch must fit within the u32 payload length limit.
• Error Handling: If a mutation is cancelled or fails, it might still have reached the WAL. It is advised to verify the key state before retrying non-idempotent operations.