Guide: Profiling eBPF Code Performance

When deploying eBPF workloads, it is essential to quantify the performance overhead introduced by your code. This article demonstrates a practical methodology for measuring the impact of eBPF hooks, using file open operations—a critical OS function—as our primary example.

🛠️ The Measurement Strategy

To accurately identify bottlenecks, we need a lean C test harness. The goal is to isolate the performance of the file open operation by minimizing external noise (like filesystem variability or disk I/O).

The Test Harness Implementation

The following code is designed to reopen the same file repeatedly under "warm cache" conditions to determine the $p50$ and $p99$ latencies.

#define _GNU_SOURCE 
#include <fcntl.h> 
#include <stdio.h> 
#include <time.h> 
#include <stdint.h> 
#include <stdlib.h> 
#include <unistd.h> 
#include <sched.h> 
#include <sys/mman.h> 
#include <sys/syscall.h> 

static inline uint64_t now_ns ( void ) { 
    struct timespec ts; 
    clock_gettime (CLOCK_MONOTONIC, ts); /* Utilizes VDSO to avoid syscall overhead */ 
    return ( uint64_t )ts.tv_sec * 1000000000ull + ts.tv_nsec; 
} 

int main ( int argc, char ** argv) { 
    const char * path = argv[ 1 ]; 
    uint64_t n = strtoull (argv[ 2 ], NULL, 10 ); 
    uint64_t warm = n / 10 ; 

    /* Preallocate and lock memory to prevent page faults during the loop */ 
    uint32_t * d = mmap (NULL, n * sizeof ( uint32_t ), PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, - 1 , 0 ); 
    mlock (d, n * sizeof ( uint32_t )); 
    for ( uint64_t i = 0 ; i < n; i ++ ) d[i] = 0 ; 

    for ( uint64_t i = 0 ; i < n; i ++ ) { 
        uint64_t t0 = now_ns (); 
        long fd = syscall (SYS_openat, AT_FDCWD, path, O_RDONLY); /* Direct syscall, bypassing libc */ 
        uint64_t t1 = now_ns (); 
        if (fd >= 0 ) close (fd); 
        d[i] = ( uint32_t )(t1 - t0); 
    } 

    /* Output results excluding the warmup phase */ 
    for ( uint64_t i = warm; i < n; i ++ ) 
        printf ( "%u \n " , d[i]); 
    return 0 ; 
}

Key Design Decisions:

• Raw Syscalls: It uses syscall(SYS_openat, ...) instead of the standard openat() libc wrapper to remove library overhead.
• Warmup Phase: The first 10% of iterations are discarded. Mathematically, if $n$ is the total iterations, the warmup is: $\text{warmup} = \frac{n}{10}$
• Memory Locking: mmap and mlock ensure that the array used to store results doesn't trigger page faults during the timing loop.

---

⚙️ Environment Setup

To ensure the perf tool can resolve symbols and provide meaningful insights rather than anonymous memory addresses, specific kernel settings must be enabled.

• Enable BPF JIT compiler
• Expose JIT-compiled symbols to kallsyms

Run the following commands: sudo sysctl -w net.core.bpf_jit_enable=1 sudo sysctl -w net.core.bpf_jit_kallsyms=1

To verify that the symbols are visible (specifically for LSM hooks in this case), you can use:

sudo bpftool prog show | rg -A4 ' lsm ' sudo rg 'bpf_prog_[0-9a-f]+_ ' /proc/kallsyms | rg 'security|path|file|open'

Note: If using a custom kernel, ensure your perf path is correctly set. Example: PERF=/usr/lib/linux-tools/6.8.0-134-generic/perf

---

📈 Measuring and Profiling

The profiling workflow follows a logical progression:

1. Baseline Execution

First, measure the system without the eBPF code. We use taskset to pin the process to a specific CPU (reducing migration noise) and chrt to set a real-time priority.

sudo taskset -c 3 chrt -f 99 ./bench /etc/hostname 100000 /tmp/samples.txt

2. Profiling with Perf

Next, record the kernel activity while the benchmark runs:

sudo $PERF record -g --call-graph fp -e cycles:k -F 997 -o ~/perf.data -- taskset -c 3 chrt -f 99 ./bench /etc/hostname 200000 /tmp/samples.txt

Breakdown of flags:

• -g & --call-graph fp: Captures call stacks using frame pointers.
• -e cycles:k: Samples CPU cycles only in kernel mode (ignoring userspace).
• -F 997: Samples at 997Hz (a non-round number to avoid synchronization with periodic system events).

3. Analyzing Results

Generate a text report or a Flamegraph (using tools like Inferno):

sudo "$PERF" report -i ~/perf.data --stdio --sort comm,dso,symbol > perf.txt

---

🔍 Findings and Analysis

The analysis focuses on the bpf_lsm_file_open stack. The following table represents the performance distribution found in the perf.txt output:

The specific percentages are the most important part $\rightarrow$ The methodology is what matters. The exact numbers vary based on your specific hook logic, but the process reveals that this is a "hot path." In such scenarios, shaving even a few CPU cycles from allocations or reads can significantly boost overall system performance.