docs/advanced/telemetry.md
Starting with version 8, Polly provides telemetry for all built-in standard and chaos resilience strategies.
To enable telemetry in Polly, add the Polly.Extensions package to your project:
dotnet add package Polly.Extensions
Afterwards, you can use the ConfigureTelemetry(...) extension method on the ResiliencePipelineBuilder:
var telemetryOptions = new TelemetryOptions
{
// Configure logging
LoggerFactory = LoggerFactory.Create(builder => builder.AddConsole())
};
// Configure enrichers
telemetryOptions.MeteringEnrichers.Add(new MyMeteringEnricher());
// Configure telemetry listeners
telemetryOptions.TelemetryListeners.Add(new MyTelemetryListener());
var pipeline = new ResiliencePipelineBuilder()
.AddTimeout(TimeSpan.FromSeconds(1))
.ConfigureTelemetry(telemetryOptions) // This method enables telemetry in the builder
.Build();
The MyTelemetryListener and MyMeteringEnricher is implemented as:
internal sealed class MyTelemetryListener : TelemetryListener
{
public override void Write<TResult, TArgs>(in TelemetryEventArguments<TResult, TArgs> args)
{
Console.WriteLine($"Telemetry event occurred: {args.Event.EventName}");
}
}
internal sealed class MyMeteringEnricher : MeteringEnricher
{
public override void Enrich<TResult, TArgs>(in EnrichmentContext<TResult, TArgs> context)
{
context.Tags.Add(new("my-custom-tag", "custom-value"));
}
}
Alternatively, you can use the AddResiliencePipeline(...) extension method which automatically enables telemetry for defined pipeline:
var serviceCollection = new ServiceCollection()
.AddLogging(builder => builder.AddConsole())
.AddResiliencePipeline("my-strategy", builder => builder.AddTimeout(TimeSpan.FromSeconds(1)))
.Configure<TelemetryOptions>(options =>
{
// Configure enrichers
options.MeteringEnrichers.Add(new MyMeteringEnricher());
// Configure telemetry listeners
options.TelemetryListeners.Add(new MyTelemetryListener());
});
The metrics are emitted under the Polly meter name. The subsequent sections provide insights into the metrics produced by Polly. Please note that any custom enriched tags are not depicted in the following tables.
Every telemetry event has the following optional tags:
pipeline.name: comes from ResiliencePipelineBuilder.Name.pipeline.instance: comes from ResiliencePipelineBuilder.InstanceName.strategy.name: comes from RetryStrategyOptions.Name.operation.key: comes from ResilienceContext.OperationKey.The sample below demonstrates how to assign these tags:
<!-- snippet: telemetry-tags -->var builder = new ResiliencePipelineBuilder();
builder.Name = "my-name";
builder.InstanceName = "my-instance-name";
builder.AddRetry(new RetryStrategyOptions
{
// The default value is "Retry"
Name = "my-retry-name"
});
ResiliencePipeline pipeline = builder.Build();
// Create resilience context with operation key
ResilienceContext resilienceContext = ResilienceContextPool.Shared.Get("my-operation-key");
// Execute the pipeline with the context
pipeline.Execute(
context =>
{
// Your code comes here
},
resilienceContext);
[!NOTE] Beware of using very large or unbounded combinations of tag values for the tags above. See best practices for more details.
These values are subsequently reflected in the following metering instruments exposed by Polly:
resilience.polly.strategy.eventsTags:
| Name | Description |
|---|---|
event.name | The name of the emitted event. |
event.severity | The severity of the event (Debug, Information, Warning, Error, Critical). |
pipeline.name | The name of the pipeline corresponding to the resilience pipeline. |
pipeline.instance | The instance name of the pipeline corresponding to the resilience pipeline. |
strategy.name | The name of the strategy generating this event. |
operation.key | The operation key associated with the call site. |
exception.type | The full name of the exception assigned to the execution result (System.InvalidOperationException). |
The event.name tag is reported by individual resilience strategies. The built-in strategies report the following events:
OnRetryOnFallbackOnHedgingOnTimeoutOnCircuitClosedOnCircuitOpenedOnCircuitHalfOpenedOnRateLimiterRejectedChaos.OnFaultChaos.OnOutcomeChaos.OnLatencyChaos.OnBehaviorresilience.polly.strategy.attempt.durationRetry and Hedging resilience strategies.Tags:
| Name | Description |
|---|---|
event.name | The name of the emitted event. Currently, the event name is always ExecutionAttempt. |
event.severity | The severity of the event (Debug, Information, Warning, Error, Critical). |
pipeline.name | The name of the pipeline corresponding to the resilience pipeline. |
pipeline.instance | The instance name of the pipeline corresponding to the resilience pipeline. |
strategy.name | The name of the strategy generating this event. |
operation.key | The operation key associated with the call site. |
exception.type | The full name of the exception assigned to the execution result (System.InvalidOperationException). |
attempt.number | The execution attempt number, starting at 0 (0, 1, 2, etc.). |
attempt.handled | Indicates if the execution outcome was handled. A handled outcome indicates execution failure and the need for retry (true, false). |
resilience.polly.pipeline.durationTags:
| Name | Description |
|---|---|
pipeline.name | The name of the pipeline corresponding to the resilience pipeline. |
pipeline.instance | The instance name of the pipeline corresponding to the resilience pipeline. |
operation.key | The operation key associated with the call site. |
exception.type | The full name of the exception assigned to the execution result (System.InvalidOperationException). |
Polly API lets you add extra tags to any resilience event created by resilience strategies. To do this, derive from the xref:Polly.Telemetry.MeteringEnricher class and add your custom enricher to the xref:Polly.Telemetry.TelemetryOptions.MeteringEnrichers list.
The custom enricher:
<!-- snippet: metering-enricher -->internal sealed class CustomMeteringEnricher : MeteringEnricher
{
public override void Enrich<TResult, TArgs>(in EnrichmentContext<TResult, TArgs> context)
{
// You can read additional details from any resilience event and use it to enrich the telemetry
if (context.TelemetryEvent.Arguments is OnRetryArguments<TResult> retryArgs)
{
// See https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/metrics.md for more details
// on how to name the tags.
context.Tags.Add(new("retry.attempt", retryArgs.AttemptNumber));
}
}
}
Registering the custom enricher:
<!-- snippet: metering-enricher-registration -->var telemetryOptions = new TelemetryOptions();
// Register custom enricher
telemetryOptions.MeteringEnrichers.Add(new CustomMeteringEnricher());
var pipeline = new ResiliencePipelineBuilder()
.AddRetry(new RetryStrategyOptions())
.ConfigureTelemetry(telemetryOptions) // This method enables telemetry in the builder
.Build();
Logs are registered under the Polly logger name. Here are some examples of the logs:
// This log is recorded whenever a resilience event occurs. EventId = 0
Resilience event occurred. EventName: '{EventName}', Source: '{PipelineName}/{PipelineInstance}/{StrategyName}', Operation Key: '{OperationKey}', Result: '{Result}'
// This log is recorded when a resilience pipeline begins executing. EventId = 1
Resilience pipeline executing. Source: '{PipelineName}/{PipelineInstance}', Operation Key: '{OperationKey}'
// This log is recorded when a resilience pipeline finishes execution. EventId = 2
Resilience pipeline executed. Source: '{PipelineName}/{PipelineInstance}', Operation Key: '{OperationKey}', Result: '{Result}', Execution Health: '{ExecutionHealth}', Execution Time: {ExecutionTime}ms
// This log is recorded upon the completion of every execution attempt. EventId = 3
Execution attempt. Source: '{PipelineName}/{PipelineInstance}/{StrategyName}', Operation Key: '{OperationKey}', Result: '{Result}', Handled: '{Handled}', Attempt: '{Attempt}', Execution Time: '{ExecutionTime}ms'
Each resilience strategy can generate telemetry data through the ResilienceStrategyTelemetry API. Polly encapsulates event details as TelemetryEventArguments and emits them via TelemetryListener.
To leverage this telemetry data, users should assign a TelemetryListener instance to ResiliencePipelineBuilder.TelemetryListener and then consume the TelemetryEventArguments.
For common scenarios, it is expected that users would make use of Polly.Extensions. This extension enables telemetry configuration through the ResiliencePipelineBuilder.ConfigureTelemetry(...) method, which processes TelemetryEventArguments to generate logs and metrics.
To customize the severity of telemetry events, set the SeverityProvider delegate that allows changing the default severity of resilience events:
services.AddResiliencePipeline("my-strategy", (builder, context) =>
{
// Create a new instance of telemetry options by using copy-constructor of the global ones.
// This ensures that common configuration is preserved.
var telemetryOptions = new TelemetryOptions(context.GetOptions<TelemetryOptions>());
telemetryOptions.SeverityProvider = args => args.Event.EventName switch
{
// Decrease severity of specific events
"OnRetry" => ResilienceEventSeverity.Debug,
"ExecutionAttempt" => ResilienceEventSeverity.Debug,
_ => args.Event.Severity
};
builder.AddRetry(new RetryStrategyOptions());
// Override the telemetry configuration for this pipeline.
builder.ConfigureTelemetry(telemetryOptions);
});
There are telemetry events which are specific to strategies. Like the above overrides are specific to Retry (and partially to Hedging).
[!TIP] If you want to define X severity for the
ExecutionAttemptevent of Retry and Y severity for theExecutionAttemptevent of Hedging then use theargs.Source.StrategyNameinformation as well inside the switch expression.
There are also pipeline specific telemetry events
PipelineExecuting: by default reported at Debug levelPipelineExecuted: by default reported at Information levelIn general the suggestion is to use SeverityProvider to override strategy specific telemetry events' severity.
[!TIP] If you want to suppress the specific event completely then use
ResilienceEventSeverity.None.