design/tracing-design.md
Status: Accepted
Envoy has rich support for distributed tracing. Exporting trace data to OpenTelemetry has been supported since envoy v1.23. However, Contour does not currently offer a way to configure Envoy for tracing. This design document provides the implementation of exporting tracing data to OpenTelemetry in Contour.
Envoy's documentation on tracing says:
Distributed tracing allows developers to obtain visualizations of call flows in large service oriented architectures. It can be invaluable in understanding serialization, parallelism, and sources of latency.
Envoy supports:
x-client-trace-id headerThe tracing ecosystem is complex right now. However, a few observations can be made:
The Contour configuration file and ContourConfiguration CRD will be extended with a new optional tracing section. This configuration block, if present, will enable tracing and will define the trace format and other properties needed to generate and export trace data.
The OpenTelemetry trace format will be supported.
The parameters defined in the configuration file will be used to configure each HTTP connection manager's tracing section.
Since the tracing backend is pluggable, Contour will not package any particular backend. The user is responsible for deploying and operating the tracing backend of their choice, and configuring Contour to make use of it.
When using tracing, the user must first define an ExtensionService with the cluster-level details for the otel-collector itself.
For example:
apiVersion: projectcontour.io/v1alpha1
kind: ExtensionService
metadata:
namespace: projectcontour
name: otel-collector
spec:
protocol: h2c
services:
- name: otel-collector
port: 4317
timeoutPolicy:
response: 50ms
The tracing section of the Contour config file will look like:
tracing:
# includePodDetail defines a flag. If it is true, contour will add the pod name and namespace to the span of the trace. the default is true
includePodDetail: true
# serviceName defines a configurable service name.the default is contour
serviceName: contour
# overallSampling defines the sampling rate of trace data.
overallSampling: 100
# maxPathTagLength defines maximum length of the request path to extract and include in the HttpUrl tag.
maxPathTagLength: 256
# customTags defines a list of custom tags with unique tag name.
# A customTag can only set one of literal and requestHeaderName
customTags:
# tagName used to populate the tag name
- tagName: literal
# literal is a static custom tag value.
literal: 'this is literal'
- tagName: requestHeaderName
# requestHeaderName indicates which request header the label value is obtained from.
requestHeaderName: ':path'
# extensionService Identifies the extension service defining the openTelemetry collector service.
# formatted as <namespace>/<name>.
extensionService: projectcontour/otel-collector
If there are alternative high level or detailed designs that were not pursued they should be called out here with a brief explanation of why they were not pursued.
Similar to other extensionservices, Envoy's connection with the tracing provider server can be secured over TLS using existing mechanisms.
A discussion of any compatibility issues that need to be considered
In the presence of the tracing block above, all HTTP connection managers (HCM) will be configured for tracing.
A Tracing HCM will include the following as an example:
...
Tracing: &http.HttpConnectionManager_Tracing{
OverallSampling: &envoy_type.Percent{
Value: 100,
},
MaxPathTagLength: wrapperspb.UInt32(256),
CustomTags: []*envoy_trace_v3.CustomTag{
{
Tag: "literal",
Type: &envoy_trace_v3.CustomTag_Literal_{
Literal: &envoy_trace_v3.CustomTag_Literal{
Value: "this is literal",
},
},
},
{
Tag: "podName",
Type: &envoy_trace_v3.CustomTag_Environment_{
Environment: &envoy_trace_v3.CustomTag_Environment{
Name: "HOSTNAME",
},
},
},
{
Tag: "podNamespaceName",
Type: &envoy_trace_v3.CustomTag_Environment_{
Environment: &envoy_trace_v3.CustomTag_Environment{
Name: "CONTOUR_NAMESPACE",
},
},
},
{
Tag: "requestHeaderName",
Type: &envoy_trace_v3.CustomTag_RequestHeader{
RequestHeader: &envoy_trace_v3.CustomTag_Header{
Name: ":path",
},
},
},
},
Provider: &envoy_config_trace_v3.Tracing_Http{
Name: "envoy.tracers.opentelemetry",
ConfigType: &envoy_config_trace_v3.Tracing_Http_TypedConfig{
TypedConfig: protobuf.MustMarshalAny(&envoy_config_trace_v3.OpenTelemetryConfig{
GrpcService: &envoy_core_v3.GrpcService{
TargetSpecifier: &envoy_core_v3.GrpcService_EnvoyGrpc_{
EnvoyGrpc: &envoy_core_v3.GrpcService_EnvoyGrpc{
ClusterName: "extension/projectcontour/otel-collector",
Authority: "extension/projectcontour/otel-collector",
},
},
Timeout: durationpb.New(time.Millisecond * 50),
},
ServiceName: "contour",
}),
},
},
},
...