Diagnostics in Kestrel

[!INCLUDE]

By Sourabh Shirhatti

This article provides guidance for gathering diagnostics from Kestrel to help troubleshoot issues. Topics covered include:

• Logging: Structured logs written to .NET logging. xref:Microsoft.Extensions.Logging.ILogger is used by app frameworks to write logs, and by users for their own logging in an app.
• Metrics: Representation of data measures over intervals of time, for example, requests per second. Metrics are emitted using EventCounter and can be observed using the dotnet-counters command line tool or with Application Insights.
• DiagnosticSource: DiagnosticSource is a mechanism for production-time logging with rich data payloads for consumption within the process. Unlike logging, which assumes data will leave the process and expects serializable data, DiagnosticSource works well with complex data.

Logging

Like most components in ASP.NET Core, Kestrel uses Microsoft.Extensions.Logging to emit log information. Kestrel employs the use of multiple categories which allows you to be selective on which logs you listen to.

Connection logging

Kestrel also supports the ability to emit Debug level logs for byte-level communication and can be enabled on a per-endpoint basis. To enable connection logging, see configure endpoints for Kestrel

Metrics

Metrics is a representation of data measures over intervals of time, for example, requests per second. Metrics data allows observation of the state of an app at a high-level. Kestrel metrics are emitted using EventCounter.

[!NOTE] The connections-per-second and tls-handshakes-per-second counters are named incorrectly. The counters:

• Do not always contain the number of new connections or TLS handshakes per second
• Display the number of new connection or TLS handshakes in the last update interval as requested as the consumer of Events via the EventCounterIntervalSec argument in the filterPayload to KestrelEventSource.

We recommend consumers of these counters scale the metric value based on the DisplayRateTimeScale of one second.

DiagnosticSource

Kestrel emits a DiagnosticSource event for HTTP requests rejected at server layer such as malformed requests and protocols violations. As such, these requests never make it into the hosting layer of ASP.NET Core.

Kestrel emits these events with the Microsoft.AspNetCore.Server.Kestrel.BadRequest event name and an IFeatureCollection as the object payload. The underlying exception can be retrieved by accessing the IBadRequestExceptionFeature on the feature collection.

Resolving these events is a two-step process. An observer for DiagnosticListener must be created:

class BadRequestEventListener : IObserver<KeyValuePair<string, object>>, IDisposable
{
    private readonly IDisposable _subscription;
    private readonly Action<IBadRequestExceptionFeature> _callback;

    public BadRequestEventListener(DiagnosticListener diagnosticListener, Action<IBadRequestExceptionFeature> callback)
    {
        _subscription = diagnosticListener.Subscribe(this!, IsEnabled);
        _callback = callback;
    }
    private static readonly Predicate<string> IsEnabled = (provider) => provider switch
    {
        "Microsoft.AspNetCore.Server.Kestrel.BadRequest" => true,
        _ => false
    };
    public void OnNext(KeyValuePair<string, object> pair)
    {
        if (pair.Value is IFeatureCollection featureCollection)
        {
            var badRequestFeature = featureCollection.Get<IBadRequestExceptionFeature>();

            if (badRequestFeature is not null)
            {
                _callback(badRequestFeature);
            }
        }
    }
    public void OnError(Exception error) { }
    public void OnCompleted() { }
    public virtual void Dispose() => _subscription.Dispose();
}

Subscribe to the ASP.NET Core DiagnosticListener with the observer. In this example, we create a callback that logs the underlying exception.

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
var diagnosticSource = app.Services.GetRequiredService<DiagnosticListener>();
using var badRequestListener = new BadRequestEventListener(diagnosticSource, (badRequestExceptionFeature) =>
{
    app.Logger.LogError(badRequestExceptionFeature.Error, "Bad request received");
});
app.MapGet("/", () => "Hello world");
app.Run();

Behavior with debugger attached

Certain timeouts and rate limits aren't enforced when a debugger is attached to a Kestrel process. For more information, see Behavior with debugger attached.