================== Memory Profiling

Ceph Monitor, OSD, and MDS can report TCMalloc heap profiles. Install google-perftools if you want to generate these. Your OS distribution might package this under a different name (for example, gperftools), and your OS distribution might use a different package manager. Run a command similar to this one to install google-perftools:

.. prompt:: bash

sudo apt-get install google-perftools

The profiler dumps output to your log file directory (/var/log/ceph). See Logging and Debugging_ for details.

To view the profiler logs with Google's performance tools, run the following command:

.. prompt:: bash

google-pprof --text {path-to-daemon}  {log-path/filename}

For example::

$ ceph tell osd.0 heap start_profiler
$ ceph tell osd.0 heap dump
osd.0 tcmalloc heap stats:------------------------------------------------
MALLOC:        2632288 (    2.5 MiB) Bytes in use by application
MALLOC: +       499712 (    0.5 MiB) Bytes in page heap freelist
MALLOC: +       543800 (    0.5 MiB) Bytes in central cache freelist
MALLOC: +       327680 (    0.3 MiB) Bytes in transfer cache freelist
MALLOC: +      1239400 (    1.2 MiB) Bytes in thread cache freelists
MALLOC: +      1142936 (    1.1 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =      6385816 (    6.1 MiB) Actual memory used (physical + swap)
MALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =      6385816 (    6.1 MiB) Virtual address space used
MALLOC:
MALLOC:            231              Spans in use
MALLOC:             56              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
$ google-pprof --text \
               /usr/bin/ceph-osd  \
               /var/log/ceph/ceph-osd.0.profile.0001.heap
 Total: 3.7 MB
 1.9  51.1%  51.1%      1.9  51.1% ceph::log::Log::create_entry
 1.8  47.3%  98.4%      1.8  47.3% std::string::_Rep::_S_create
 0.0   0.4%  98.9%      0.0   0.6% SimpleMessenger::add_accept_pipe
 0.0   0.4%  99.2%      0.0   0.6% decode_message
 ...

Performing another heap dump on the same daemon creates another file. It is convenient to compare the new file to a file created by a previous heap dump to show what has grown in the interval. For example::

$ google-pprof --text --base out/osd.0.profile.0001.heap \
      ceph-osd out/osd.0.profile.0003.heap
 Total: 0.2 MB
 0.1  50.3%  50.3%      0.1  50.3% ceph::log::Log::create_entry
 0.1  46.6%  96.8%      0.1  46.6% std::string::_Rep::_S_create
 0.0   0.9%  97.7%      0.0  26.1% ReplicatedPG::do_op
 0.0   0.8%  98.5%      0.0   0.8% __gnu_cxx::new_allocator::allocate

See Google Heap Profiler_ for additional details.

After you have installed the heap profiler, start your cluster and begin using the heap profiler. You can enable or disable the heap profiler at runtime, or ensure that it runs continuously. When running commands based on the examples that follow, do the following:

#. replace {daemon-type} with mon, osd or mds #. replace {daemon-id} with the OSD number or the MON ID or the MDS ID

Starting the Profiler

To start the heap profiler, run a command of the following form:

.. prompt:: bash

ceph tell {daemon-type}.{daemon-id} heap start_profiler

For example:

.. prompt:: bash

ceph tell osd.1 heap start_profiler

Alternatively, if the CEPH_HEAP_PROFILER_INIT=true variable is found in the environment, the profile will be started when the daemon starts running.

Printing Stats

To print out statistics, run a command of the following form:

.. prompt:: bash

ceph tell {daemon-type}.{daemon-id} heap stats

For example:

.. prompt:: bash

ceph tell osd.0 heap stats

.. note:: The reporting of stats with this command does not require the profiler to be running and does not dump the heap allocation information to a file.

Dumping Heap Information

To dump heap information, run a command of the following form:

.. prompt:: bash

ceph tell {daemon-type}.{daemon-id} heap dump

For example:

.. prompt:: bash

ceph tell mds.a heap dump

.. note:: Dumping heap information works only when the profiler is running.

Releasing Memory

To release memory that tcmalloc has allocated but which is not being used by the Ceph daemon itself, run a command of the following form:

.. prompt:: bash

ceph tell {daemon-type}{daemon-id} heap release

For example:

.. prompt:: bash

ceph tell osd.2 heap release

Stopping the Profiler

To stop the heap profiler, run a command of the following form:

.. prompt:: bash

ceph tell {daemon-type}.{daemon-id} heap stop_profiler

For example:

.. prompt:: bash

ceph tell osd.0 heap stop_profiler

.. _Logging and Debugging: ../log-and-debug .. _Google Heap Profiler: http://goog-perftools.sourceforge.net/doc/heap_profiler.html

Alternative Methods of Memory Profiling

Running Massif heap profiler with Valgrind


The Massif heap profiler tool can be used with Valgrind to measure how much
heap memory is used. This method is well-suited to troubleshooting RadosGW.

See the `Massif documentation
<https://valgrind.org/docs/manual/ms-manual.html>`_ for more information.

Install Valgrind from the package manager for your distribution then start the
Ceph daemon you want to troubleshoot:

.. prompt:: bash

   sudo -u ceph valgrind --max-threads=1024 --tool=massif /usr/bin/radosgw -f --cluster ceph --name NAME --setuser ceph --setgroup ceph

When this command has completed its run, a file with a name of the form
``massif.out.<pid>`` will be saved in your current working directory. To run
the command above, the user who runs it must have write permissions in the
current directory.

Run the ``ms_print`` command to get a graph and statistics from the collected
data in the ``massif.out.<pid>`` file:

.. prompt:: bash

   ms_print massif.out.12345

The output of this command is helpful when submitting a bug report.