.. _ray-tracing:

Tracing

To help debug and monitor Ray applications, Ray integrates with OpenTelemetry to facilitate exporting traces to external tracing stacks such as Jaeger.

.. note::

Tracing is an Alpha feature and no longer under active development/being maintained. APIs are subject to change.

Installation

First, install OpenTelemetry:

.. code-block:: shell

pip install opentelemetry-api==1.34.1
pip install opentelemetry-sdk==1.34.1
pip install opentelemetry-exporter-otlp==1.34.1

Tracing startup hook

To enable tracing, you must provide a tracing startup hook with a function that sets up the :ref:Tracer Provider <tracer-provider>, :ref:Remote Span Processors <remote-span-processors>, and :ref:Additional Instruments <additional-instruments>. The tracing startup hook is expected to be a function that is called with no args or kwargs. This hook needs to be available in the Python environment of all the worker processes.

Below is an example tracing startup hook that sets up the default tracing provider, exports spans to files in /tmp/spans, and does not have any additional instruments.

.. testcode::

import ray import os from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ( ConsoleSpanExporter, SimpleSpanProcessor, )

def setup_tracing() -> None: # Creates /tmp/spans folder os.makedirs("/tmp/spans", exist_ok=True) # Sets the tracer_provider. This is only allowed once per execution # context and will log a warning if attempted multiple times. trace.set_tracer_provider(TracerProvider()) trace.get_tracer_provider().add_span_processor( SimpleSpanProcessor( ConsoleSpanExporter( out=open(f"/tmp/spans/{os.getpid()}.json", "a") ) ) )

For open-source users who want to experiment with tracing, Ray has a default tracing startup hook that exports spans to the folder /tmp/spans. To run using this default hook, run the following code sample to set up tracing and trace a simple Ray Task.

.. tab-set::

.. tab-item:: ray start

    .. code-block:: shell

        $ ray start --head --tracing-startup-hook=ray.util.tracing.setup_local_tmp_tracing:setup_tracing
        $ python
            ray.init()
            @ray.remote
            def my_function():
                return 1

            obj_ref = my_function.remote()

.. tab-item:: ray.init()

    .. testcode::

        ray.init(_tracing_startup_hook="ray.util.tracing.setup_local_tmp_tracing:setup_tracing")

        @ray.remote
        def my_function():
            return 1

        obj_ref = my_function.remote()

If you want to provide your own custom tracing startup hook, provide it in the format of module:attribute where the attribute is the setup_tracing function to be run.

.. _tracer-provider:

Tracer provider

This configures how to collect traces. View the TracerProvider API `here <https://opentelemetry-python.readthedocs.io/en/latest/sdk/trace.html#opentelemetry.sdk.trace.TracerProvider>`__.

.. _remote-span-processors:

Remote span processors

This configures where to export traces to. View the SpanProcessor API here <https://opentelemetry-python.readthedocs.io/en/latest/sdk/trace.html#opentelemetry.sdk.trace.SpanProcessor>__.

Users who want to experiment with tracing can configure their remote span processors to export spans to a local JSON file. Serious users developing locally can push their traces to Jaeger containers via the Jaeger exporter <https://opentelemetry.io/docs/languages/js/exporters/#jaeger>_.

.. _additional-instruments:

Additional instruments

If you are using a library that has built-in tracing support, the ``setup_tracing`` function you provide should also patch those libraries. You can find more documentation for the instrumentation of these libraries `here <https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation>`_.

Custom traces
*************
Add custom tracing in your programs. Within your program, get the tracer object with ``trace.get_tracer(__name__)`` and start a new span with ``tracer.start_as_current_span(...)``.

See below for a simple example of adding custom tracing.

.. testcode::

  from opentelemetry import trace

  @ray.remote
  def my_func():
      tracer = trace.get_tracer(__name__)

      with tracer.start_as_current_span("foo"):
          print("Hello world from OpenTelemetry Python!")