docs/developers.md
This document features basic guidelines and recommendations on how to do bpftrace development. Please read it carefully before submitting pull requests to simplify reviewing and to speed up the merge process.
The project supports the following recommended build workflows. Please choose the one that works the best for you.
Nix is the most convenient way to build and test bpftrace. Nix will manage all of bpftrace's build and runtime dependencies. It also has the advantage of being used by the CI, so you are more likely to shake out errors before submitting your change and seeing the CI fail.
The Nix build is documented in nix.md.
The "distro build" is the more traditional way to build bpftrace. It relies on
you installing all of bpftrace's build and runtime dependencies on your host
and then calling into cmake.
Please be aware that bpftrace has strict dependencies on new versions of
libbpf and bcc. They are two of bpftrace's most important dependencies and
we plan on tracking their upstream quite closely over time.
As a result, while the distro build should work well on distros with newer
packages, developers on distros that lag more behind (for example Debian) may
want to consider using the Nix build. Or manually building and installing
bcc and libbpf.
The distro build is documented in INSTALL.md.
Every contribution should (1) not break the existing tests and (2) introduce new tests if relevant. See existing tests for inspiration on how to write new ones. Read more on the different kinds and how to run them.
We aim to not be wasteful, but always keep in mind that performance of the BPF
programs and runtime are the things in the critical path. Often, simplicity and
understandability on non-critical paths is often more important than
performance. That said, occasionally it is useful to measure the performance of
different parts of the pipeline. You may run bpftrace using
--mode compiler-bench in order to see the performance of the various
passes during compilation.
Similarly, you may benchmark the code generated by bpftrace using BENCH probes
and --mode bench:
$ bpftrace --mode bench -e 'BENCH:my_benchmark { @a++; }'
Attached 1 probe
+--------------+--------------+
| BENCHMARK | AVERAGE TIME |
+--------------+--------------+
| my_benchmark | 270ns |
+--------------+--------------+
CI executes the above tests in a matrix of different LLVM versions on NixOS.
The jobs are defined in .github/workflows/ci.yml.
CI is automatically run on all branches and pull requests on the main repo. We recommend to enable the CI (GitHub Actions) on your own fork, too, which will allow you to run the CI against your testing branches.
It may often happen that tests pass on your local setup but fail in one of the CI environments. In such a case, it is useful to reproduce the environment to debug the issue.
To reproduce the NixOS jobs (from .github/workflows/ci.yml):
.github/include/ci.py with the relevant environment variables setExample ci.py invocations:
$ NIX_TARGET=.#bpftrace-llvm10 ./.github/include/ci.py
$ NIX_TARGET=.#bpftrace-llvm11 \
CMAKE_BUILD_TYPE=Release \
RUNTIME_TEST_DISABLE="probe.kprobe_offset_fail_size,usdt.usdt probes - file based semaphore activation multi process" \
./.github/include/ci.py
Some tests are known to be flaky and sometimes fail in the CI environment. The list of known such tests:
usdt.usdt probes - file based semaphore activation multi process (#2410)What usually helps, is restarting the CI. This is simple on your own fork but requires one of the maintainers for pull requests.
In CI we run a subset of runtime tests under a controlled kernel by taking advantage of nested virtualization on CI runners. For these tests, we use vmtest to manage the virtual machine.
The instructions in the above "Debugging CI failures" section also work for the vmtest-ed runtime tests. But if you want to manually try something quick and dirty in a CI kernel, you can do something like the following:
$ nix develop
(nix:nix-shell-env) $ vmtest -k $(nix build --print-out-paths .#kernel-6_12)/bzImage -- ./build/src/bpftrace -V
=> bzImage
===> Booting
===> Setting up VM
===> Running command
bpftrace v0.21.0-344-g3acb
While we'll defer to vmtest documentation for full details, one neat fact
worth pointing out is that vmtest will map the current running userspace into
the VM. This means you can run binaries built on your host from inside the
guest, eg. your development build of bpftrace.
This is not about the formatting of the source code (we have clang-format
for that). Rather, it's about the semantics of the code and what language
features we try to use / avoid.
Please see coding_guidelines.md for a full treatment on the topic.
We use clang-format with our custom config for formatting code. This was
introduced after a lot of code
was already written. Instead of formatting the whole code base at once and
breaking git blame we're taking an incremental approach, each new/modified
bit of code needs to be formatted.
For bpftrace code (standard library and scripts), it should be formatted by
bpftrace itself. You can use bpftrace --fmt or scripts/bpftrace_tidy.sh.
Note that these are both checked by tests; if the changes don't adhere to our style CI jobs will fail.
git clang-format
can be used to easily format commits, e.g. git clang-format upstream/master
We want to avoid fix formatting commits. Instead every commit should be
formatted correctly.
Strongly prefer C++-style comments for single line and block comments. C-style comments are still useable for nested comments within a single line, e.g. to leave an annotation on a specific argument or parameter. In the future, there may be considerations for automated documentation based on comments, but this is not currently done.
bpftrace itself supports both C-style and C++-style comment blocks. There is
currently no decision on recommended comment style, and both are used freely.
Please squash + rebase all pull requests (with no merge commit). In other words, there should be one commit in master per pull request. This makes generating changelogs both trivial and precise with the least amount of noise.
The exception to this is PRs with complicated changes. If this is the case and the commits are well structured, a rebase + merge (no merge commit) is acceptable. The rule of thumb is the commit titles should make sense in a changelog.
The changelog is for end users. It should provide them with a quick summary of all changes important to them. Internal changes like refactoring or test changes do not belong to it.
To avoid having write a changelog when we do a release (which leads to useless changelog or a lot of work) we write them as we go. That means that every PR that has a user impacting change must also include a changelog entry.
As we include the PR number in the changelog format this can only be done after the PR has been opened.
If it is a single commit PR we include the changelog in that commit, when the PR consists of multiple commits it is OK to add a separate commit for the changelog.
For more details on bpftrace internals, see internals_development.md.