Inside VictoriaLogs: The Mechanics of Columnar Log Storage

For those operating VictoriaLogs, the experience generally revolves around three core activities: ingesting logs, executing queries, and managing retention to prevent disk saturation.

This guide traces the journey of a single log entry from the moment it hits the API to its final resting place on the physical disk. By understanding this pipeline, you can better grasp why queries are performant, why the disk contains numerous files, and which specific metrics to monitor when troubleshooting.

Note: This explanation is designed for everyone; you don't need a computer science degree or the ability to read Go code. However, if you are a developer, the VictoriaLogs source code remains the ultimate source of truth.

---

📥 Phase 1: Log Arrival and Normalization

VictoriaLogs is versatile in how it accepts data. It supports a wide array of protocols, including:

• JSON Lines
• Elasticsearch bulk
• Loki push
• OpenTelemetry
• syslog
• ...and others.

Regardless of the input format, VictoriaLogs converts every record into a standardized internal representation consisting of:

1. A Timestamp
2. A set of named fields
3. A stream identity

Customizing the Ingestion Pipeline

You can modify how logs are processed using either query arguments or specific request headers. The following table summarizes these options:

The Concept of "Stream Identity"

Stream identity is the cornerstone of VictoriaLogs' efficiency. When you define _stream_fields (e.g., pod,container), all logs sharing those specific values are grouped into a single stream.

The Operator's Golden Rule: Keep stream fields low-cardinality (values that don't change often, like host or app). Avoid using high-cardinality values (unique IDs like trace_id or user_id) as stream fields; keep those as standard fields instead.

By grouping logs this way, VictoriaLogs achieves superior compression and allows the query engine to scan every single log target only the relevant streams.

---

⚡ Phase 2: Buffering and "Parts"

VictoriaLogs does not write logs to disk one by one. Instead, it uses a staged approach:

1. Accumulation: Logs gather in an in-memory buffer.
2. Flushing: Roughly every second ($\Delta t \approx 1s$), or when the buffer hits a size limit, the batch is converted into a searchable chunk known as an in-memory part.

Parallelism via Sharding

To prevent CPU bottlenecks, VictoriaLogs employs a sharding strategy. The buffer is split across CPU cores. For example, on a machine with 3 CPUs:

• There are 3 parallel buffer shards.
• Incoming batches are distributed across these shards.
• Each shard flushes its own in-memory part independently.

What is a "Part"? A part is the fundamental data unit in VictoriaLogs (and other VictoriaMetrics products). It is a self-contained, queryable bundle of data. While most are created in RAM, exceptionally large batches may be written directly to disk as "small" or "big" parts.

Key Metric to Watch: vl_insert_flush_duration_seconds $\rightarrow$ Measures the time required to transform a buffered batch into an in-memory part.

---

📅 Phase 3: Daily Partitions

Once a batch is flushed, it is routed to a partition based on its timestamp. Each partition represents exactly one calendar day (UTC).

Disk Layout

You can observe this structure by looking at the data directory:

$ tree victoria-logs-data/
victoria-logs-data/
└── partitions/
    ├── 20260109
    ├── 20260110
    ├── 20260111
    └── 20260112

Why use daily partitions?

This architecture simplifies two critical operations:

• Efficient Retention: Instead of searching for individual expired logs, VictoriaLogs simply deletes the entire directory for a day that has passed the -retentionPeriod (default is 7 days).
• Accelerated Querying: Since most queries specify a time range (e.g., _time:5m), the system only opens the partitions that overlap that window, ignoring all other data.

Summary of State

The system effectively maintains two "faces":

• The In-Memory Side: Manages the active buffer shards and recent in-memory parts.
• The On-Disk Side: Stores the persisted parts within their respective daily partitions.