bpftrace development guide

This document features basic guidelines and recommendations on how to do bpftrace development. Please read it carefully before submitting pull requests to simplify reviewing and speed up the merge process.

Building

bpftrace uses git submodules, so ensure they are initialized when checking out the code:

git clone --recurse-submodules https://github.com/bpftrace/bpftrace
cd bpftrace

For minimum kernel version requirements, see our dependency support policy. Your kernel should be built with the necessary BPF options enabled. Verify this by running the check_kernel_features script from the scripts directory.

Nix build (recommended)

Nix is the most convenient way to build and test bpftrace. Nix manages all build and runtime dependencies automatically. It is also used by CI, so you are more likely to catch errors before submitting your change.

All build and test commands using Nix must be run inside the Nix dev shell (e.g., nix develop --command bash -c "<command>"), or from within an active nix develop session.

nix develop          # enter dev shell
mkdir build
cmake -B build -DCMAKE_BUILD_TYPE=Debug
make -C build -j$(nproc)

To develop with a different LLVM version: nix develop .#bpftrace-llvm21

For more Nix examples (static builds, fuzzing, flake management), see nix.md.

Distro build

The distro build relies on you installing all of bpftrace's build and runtime dependencies on your host and then calling cmake.

Please be aware that bpftrace has strict dependencies on new versions of libbpf and bcc. We track their upstream closely, so while the distro build works well on distros with newer packages, developers on distros that lag behind (e.g. Debian) may want to use the Nix build or manually build and install bcc and libbpf.

For a suitable build environment, see our Dockerfiles for detailed dependency examples:

• Ubuntu
• Fedora
• Debian

Once all dependencies are installed:

mkdir build
cmake -B build -DCMAKE_BUILD_TYPE=Release
make -C build -j$(nproc)

Key cmake options: -DBUILD_TESTING=ON (default), -DLLVM_REQUESTED_VERSION=<major>.

Build output

The built binary is at build/src/bpftrace.

Troubleshooting

Kernel Lockdown: If your system has kernel lockdown enabled (often with Secure Boot), bpftrace will be blocked. To disable:

• Disable Secure Boot in UEFI, or
• Run sudo mokutil --disable-validation and reboot, or
• Temporarily lift lockdown with SysRQ+x (until next boot)

Tests

Every contribution should (1) not break the existing tests and (2) introduce new tests if relevant. See existing tests for inspiration on how to write new ones. Read more on the different kinds and how to run them.

Performance

We aim to not be wasteful, but always keep in mind that performance of the BPF programs and runtime are the things in the critical path. Often, simplicity and understandability on non-critical paths is often more important than performance. That said, occasionally it is useful to measure the performance of different parts of the pipeline. You may run bpftrace using --mode compiler-bench in order to see the performance of the various passes during compilation.

Similarly, you may benchmark the code generated by bpftrace using BENCH probes and --mode bench:

$ bpftrace --mode bench -e 'BENCH:my_benchmark { @a++; }'
Attached 1 probe


+--------------+--------------+
| BENCHMARK    | AVERAGE TIME |
+--------------+--------------+
| my_benchmark | 270ns        |
+--------------+--------------+

Continuous integration

CI executes the above tests in a matrix of different LLVM versions on NixOS. The jobs are defined in .github/workflows/ci.yml.

Running the CI

CI is automatically run on all branches and pull requests on the main repo. We recommend to enable the CI (GitHub Actions) on your own fork, too, which will allow you to run the CI against your testing branches.

Debugging CI failures

It may often happen that tests pass on your local setup but fail in one of the CI environments. In such a case, it is useful to reproduce the environment to debug the issue.

To reproduce the NixOS jobs (from .github/workflows/ci.yml):

1. Acquire the job environment from the GHA UI:
2. Run .github/include/ci.py with the relevant environment variables set

Example ci.py invocations:

$ NIX_TARGET=.#bpftrace-llvm10  ./.github/include/ci.py

$ NIX_TARGET=.#bpftrace-llvm11  \
  CMAKE_BUILD_TYPE=Release \
  RUNTIME_TEST_DISABLE="probe.kprobe_offset_fail_size,usdt.usdt probes - file based semaphore activation multi process" \
  ./.github/include/ci.py

Virtual machine tests (vmtests)

In CI we run a subset of runtime tests under a controlled kernel by taking advantage of nested virtualization on CI runners. For these tests, we use vmtest to manage the virtual machine.

The instructions in the above "Debugging CI failures" section also work for the vmtest-ed runtime tests. But if you want to manually try something quick and dirty in a CI kernel, you can do something like the following:

$ nix develop

(nix:nix-shell-env) $ vmtest -k $(nix build --print-out-paths .#kernel-6_12)/bzImage -- ./build/src/bpftrace -V
=> bzImage
===> Booting
===> Setting up VM
===> Running command
bpftrace v0.21.0-344-g3acb

While we'll defer to vmtest documentation for full details, one neat fact worth pointing out is that vmtest will map the current running userspace into the VM. This means you can run binaries built on your host from inside the guest, eg. your development build of bpftrace.

Coding guidelines

This is not about the formatting of the source code (we have clang-format for that). Rather, it's about the semantics of the code and what language features we try to use / avoid.

Please see coding_guidelines.md for a full treatment on the topic.

Code style

We use clang-format with our custom config for formatting code.

git clang-format can be used to easily format commits, e.g. git clang-format upstream/master.

For bpftrace code (standard library and scripts), it should be formatted by bpftrace itself. You can use bpftrace --fmt or scripts/bpftrace_tidy.sh.

Note that these are both checked by tests; if the changes don't adhere to our style CI jobs will fail.

Avoid 'fix formatting' commits

We want to avoid fix formatting commits. Instead every commit should be formatted correctly.

Comment style

Strongly prefer C++-style comments for single line and block comments. C-style comments are still useable for nested comments within a single line, e.g. to leave an annotation on a specific argument or parameter. In the future, there may be considerations for automated documentation based on comments, but this is not currently done.

bpftrace itself supports both C-style and C++-style comment blocks. There is currently no decision on recommended comment style, and both are used freely.

Merging pull requests

Please squash + rebase all pull requests (with no merge commit). In other words, there should be one commit in master per pull request. This makes generating changelogs both trivial and precise with the least amount of noise.

The exception to this is PRs with complicated changes. If this is the case and the commits are well structured, a rebase + merge (no merge commit) is acceptable. The rule of thumb is the commit titles should make sense in a changelog.

Changelog

The changelog is for end users. It should provide them with a quick summary of all changes important to them. Internal changes like refactoring or test changes do not belong to it.

Maintaining the changelog

To avoid having write a changelog when we do a release (which leads to useless changelog or a lot of work) we write them as we go. That means that every PR that has a user impacting change must also include a changelog entry.

As we include the PR number in the changelog format this can only be done after the PR has been opened.

If it is a single commit PR we include the changelog in that commit, when the PR consists of multiple commits it is OK to add a separate commit for the changelog.

bpftrace internals

For more details on bpftrace internals, see internals_development.md.