docs/en/operations/allocation-profiling-old.md
import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
ClickHouse uses jemalloc as its global allocator. Jemalloc comes with some tools for allocation sampling and profiling.
To make allocation profiling more convenient, SYSTEM commands are provided along with four letter word (4LW) commands in Keeper.
If you want to sample and profile allocations in jemalloc, you need to start ClickHouse/Keeper with profiling enabled using the environment variable MALLOC_CONF:
MALLOC_CONF=background_thread:true,prof:true,prof_active:true
jemalloc will sample allocations and store the information internally.
You can tell jemalloc to flush the current profile by running:
SYSTEM JEMALLOC FLUSH PROFILE
echo jmfp | nc localhost 9181
By default, the heap profile file will be generated in /tmp/jemalloc_clickhouse._pid_._seqnum_.heap where _pid_ is the PID of ClickHouse and _seqnum_ is the global sequence number for the current heap profile.
For Keeper, the default file is /tmp/jemalloc_keeper._pid_._seqnum_.heap, and follows the same rules.
A different location can be defined by appending the MALLOC_CONF environment variable with the prof_prefix option.
For example, if you want to generate profiles in the /data folder where the filename prefix will be my_current_profile, you can run ClickHouse/Keeper with the following environment variable:
MALLOC_CONF=background_thread:true,prof:true,prof_prefix:/data/my_current_profile
The generated file will be appended to the prefix PID and sequence number.
After heap profiles have been generated, they need to be analyzed.
For that, jemalloc's tool called jeprof can be used. It can be installed in multiple ways:
autogen.sh from the root folder. This will provide you with the jeprof script inside the bin folder:::note
jeprof uses addr2line to generate stacktraces which can be really slow.
If that's the case, it is recommended to install an alternative implementation of the tool.
git clone https://github.com/gimli-rs/addr2line.git --depth=1 --branch=0.23.0
cd addr2line
cargo build --features bin --release
cp ./target/release/addr2line path/to/current/addr2line
:::
There are many different formats to generate from the heap profile using jeprof.
It is recommended to run jeprof --help for information on the usage and the various options the tool provides.
In general, the jeprof command is used as:
jeprof path/to/binary path/to/heap/profile --output_format [ > output_file]
If you want to compare which allocations happened between two profiles you can set the base argument:
jeprof path/to/binary --base path/to/first/heap/profile path/to/second/heap/profile --output_format [ > output_file]
jeprof path/to/binary path/to/heap/profile --text > result.txt
jeprof path/to/binary path/to/heap/profile --pdf > result.pdf
jeprof allows you to generate collapsed stacks for building flame graphs.
You need to use the --collapsed argument:
jeprof path/to/binary path/to/heap/profile --collapsed > result.collapsed
After that, you can use many different tools to visualize collapsed stacks.
The most popular is FlameGraph which contains a script called flamegraph.pl:
cat result.collapsed | /path/to/FlameGraph/flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg
Another interesting tool is speedscope that allows you to analyze collected stacks in a more interactive way.
If ClickHouse/Keeper is started with the profiler enabled, additional commands for disabling/enabling allocation profiling during runtime are supported. Using those commands, it's easier to profile only specific intervals.
To disable the profiler:
<Tabs groupId="binary"> <TabItem value="clickhouse" label="ClickHouse">SYSTEM JEMALLOC DISABLE PROFILE
echo jmdp | nc localhost 9181
To enable the profiler:
<Tabs groupId="binary"> <TabItem value="clickhouse" label="ClickHouse">SYSTEM JEMALLOC ENABLE PROFILE
echo jmep | nc localhost 9181
It's also possible to control the initial state of the profiler by setting the prof_active option which is enabled by default.
For example, if you don't want to sample allocations during startup but only after, you can enable the profiler. You can start ClickHouse/Keeper with the following environment variable:
MALLOC_CONF=background_thread:true,prof:true,prof_active:false
The profiler can be enabled later.
jemalloc has many different options available, which are related to the profiler. They can be controlled by modifying the MALLOC_CONF environment variable.
For example, the interval between allocation samples can be controlled with lg_prof_sample.
If you want to dump the heap profile every N bytes you can enable it using lg_prof_interval.
It is recommended to check jemallocs reference page for a complete list of options.
ClickHouse/Keeper expose jemalloc related metrics in many different ways.
:::warning Warning It's important to be aware that none of these metrics are synchronized with each other and values may drift. :::
asynchronous_metrics {#system-table-asynchronous_metrics}SELECT *
FROM system.asynchronous_metrics
WHERE metric LIKE '%jemalloc%'
FORMAT Vertical
jemalloc_bins {#system-table-jemalloc_bins}Contains information about memory allocations done via the jemalloc allocator in different size classes (bins) aggregated from all arenas.
All jemalloc related metrics from asynchronous_metrics are also exposed using the Prometheus endpoint in both ClickHouse and Keeper.
jmst 4LW command in Keeper {#jmst-4lw-command-in-keeper}Keeper supports the jmst 4LW command which returns basic allocator statistics:
echo jmst | nc localhost 9181