docs/content/migrate/v3.md
This guide provides detailed migration steps for upgrading between different Traefik v3 versions. Each section covers breaking changes, deprecations, and configuration updates required for a smooth transition.
From version v3.7.12 onwards, a new aliasHeadersStrategy entry point option deprecates and replaces the
underscoreHeadersStrategy option introduced in v3.6.20.
Go canonicalizes header names on dashes only, so it handles X-Auth-User, X_Auth_User and X.Auth.User as three
distinct headers, while the backends deriving their variable names from the header names (CGI, WSGI, PHP, NGINX, ...)
uppercase the name and replace every character that is neither a letter nor a digit with an underscore: for them,
the three names above are the same HTTP_X_AUTH_USER variable.
The underscoreHeadersStrategy option only handles the header names containing an underscore character,
which leaves the other aliasing forms untouched. Its behavior is unchanged.
Underscores are only one of the characters building such an alias: every character HTTP allows in a header name except
the letters, the digits and the dash does, that is !, #, $, %, &, ', *, +, ., ^, _, `, |
and ~. The aliasHeadersStrategy option handles them all:
keep (default): request headers with an aliasing name are forwarded as is.delete: any request header whose name contains a character which is neither a letter, a digit, nor a dash is silently removed from the request.reject: any request carrying a header whose name contains a character which is neither a letter, a digit, nor a dash is rejected with a 400 Bad Request response.The default value is keep, so the existing behavior is preserved.
entryPoints:
websecure:
address: ':443'
http:
aliasHeadersStrategy: delete
[entryPoints.websecure]
address = ":443"
[entryPoints.websecure.http]
aliasHeadersStrategy = "delete"
--entryPoints.websecure.address=:443
--entryPoints.websecure.http.aliasHeadersStrategy=delete
Setting underscoreHeadersStrategy logs a deprecation warning.
Configuring both options with different values makes the install configuration invalid.
Traefik now logs a warning at startup for every entry point left without this option configured, as the middlewares managing request headers rely on it to not be spoofed with an aliasing name.
Please check out the entry point aliasHeadersStrategy option and the Headers with Aliasing Names documentation for more details.
Negative weights in a TCP or UDP weighted service are now rejected when the service is built. Such a configuration previously made the load-balancer selection loop endlessly while holding its lock, which consumed a CPU core and blocked every subsequent connection to that service until Traefik was restarted.
A weighted service declaring a negative weight is now reported as disabled in the API, with the error attached,
and the routers referencing it are not created.
Replace any negative weight with a positive one, or with 0 to take a child service out of the rotation.
When a child of a TCP or UDP weighted service cannot be built, for instance because it does not exist,
the parent weighted service is now reported as disabled in the API, with the error attached,
as the HTTP weighted service already was.
Such a parent service previously reported neither an error nor a status.
This only changes what is reported: the routers referencing the parent service were already not created.
Starting with v3.7.11, the new safeNaming provider option enables collision-safe naming for the routers,
middlewares and services generated by the Kubernetes CRD provider, instead of the current naming scheme, under
which names can collide across namespaces or resources.
Generated names are derived from the identity of the object they come from instead of being flattened and
normalized, and the ones generated for a route are derived from the route index instead of its rule. For example,
for a Kubernetes Service named whoami and an IngressRoute named test.route, both in the default namespace:
default-whoami-80 -> default_whoami_80
default-test-route-6b204d94623b3df4370c -> default_test.route_0
The option is disabled by default, which preserves the existing behavior.
!!! warning "Observability"
These names are user-visible: they appear in the dashboard and API, in the access logs `RouterName` and `ServiceName` fields,
and in the `router` and `service` labels of the metrics.
Dashboards, alerting rules, and log queries that match on Kubernetes CRD router, middleware or service names must be updated accordingly
when `safeNaming` is enabled.
Please check out the safeNaming provider documentation for more details.
The TLS options are configured on a router, but they are applied during the TLS handshake, before the routing occurs,
and are therefore mapped to the host names found in the router rule rather than to the router itself.
When several routers on the same entry point serve the same host name with different TLS options,
Traefik cannot decide which options to apply, and falls back to the default TLS options for that host name.
The default TLS options being the fallback of the conflict resolution, they should not be less secure than the options
they can replace: a router relying on a mutual TLS authentication (clientAuth), for example, no longer enforces it
if a conflict on its host name falls back to default TLS options that do not require it.
See GHSA-g55h-rg46-x9c5 for more details.
Starting with v3.7.11, the new core.strictTLSOptions install configuration option disables this fallback:
the routers involved in the conflict are marked in error and are not built at all.
The option is disabled by default, to preserve the existing behavior, but enabling it is recommended:
## Install configuration
core:
strictTLSOptions: true
## Install configuration
[core]
strictTLSOptions = true
## Install configuration
--core.strictTLSOptions=true
!!! warning "Disabled routers"
This option fails closed: a conflict disables all the routers serving the conflicting host name
on the concerned entry point, until the conflict is resolved.
Please check out the Conflicting TLS Options documentation for more details.
In the Kubernetes CRD provider, the TLSOption and the TLSStore named default are cluster-wide, whatever
the namespace they are defined in. Anyone allowed to create one in a single namespace can therefore replace the
TLS policy, mutual TLS authentication included, of the routers that do not reference TLS options explicitly.
Starting with v3.7.11, the new defaultTLSResourcesNamespace provider option reserves these resources
to a namespace the cluster operator controls.
The option is empty by default, to preserve the existing behavior, but setting it is recommended:
## Install configuration
providers:
kubernetesCRD:
defaultTLSResourcesNamespace: traefik
## Install configuration
[providers.kubernetesCRD]
defaultTLSResourcesNamespace = "traefik"
## Install configuration
--providers.kubernetescrd.defaultTLSResourcesNamespace=traefik
!!! warning "Ignored resources"
A `default` resource defined outside of the configured namespace is ignored, and cannot be referenced
under its namespaced name either. For a `TLSStore`, this also applies to the certificates it defines.
Please check out the Kubernetes CRD provider documentation for more details.
Starting with v3.7.10, the Kubernetes Gateway API provider derives the hash suffix of the router names it generates from all the route, Gateway, and listener identifying fields, instead of from the route rule alone.
This is required to prevent distinct routes whose namespace, name, Gateway, and listener happen to concatenate to the same string from colliding on a single generated router name.
Generated router names keep the same overall shape:
<route kind>-<route namespace>-<route name>-gw-<gateway namespace>-<gateway name>-ep-<entry point>-<rule index>-<hash>
but the <hash> suffix changes, since it is no longer computed from the route rule alone.
For example, an HTTPRoute named http-app-1 in the default namespace previously generated the router httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-af329269dd38031b03e3 and is now generated as
httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-799bcbf0c2317c5d1c93.
Middleware names are derived from the router name, so they change accordingly, for example <router name>-requestheadermodifier-0.
Generated service names were previously built from the backend reference only:
<backend namespace>-<backend name>-<port>
They are now prefixed with the route rule they are generated for:
<router name>-svc-<backend namespace>-<backend name>-<backend index>
For example, a whoami backend in the default namespace previously exposed as default-whoami-http-80@kubernetesgateway is now exposed as
httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-799bcbf0c2317c5d1c93-svc-default-whoami-0@kubernetesgateway.
!!! warning "Observability"
These names are user-visible: they appear in the dashboard and API, in the access logs `RouterName` and `ServiceName` fields, and in the `router`/`service` labels of the metrics.
Dashboards, alerting rules, and log queries that match on Gateway API router, middleware, or service names must be updated accordingly.
Starting with v3.7.10, the Kubernetes Gateway API provider supports version v1.6.1 of the specification.
TCPRoute graduated to the Standard channel in Gateway API v1.6.0, with a new v1 version.
Traefik v3.7 still watches TCPRoute through its v1alpha2 version,
which the Standard channel CRDs no longer serve.
If you use TCPRoute and upgrade to Gateway API v1.6.1 CRDs,
you must install the experimental channel CRDs.
The experimentalChannel option remains required to enable TCPRoute support in Traefik.
!!! warning "Standard channel CRDs with experimentalChannel enabled"
Traefik cannot watch the `v1alpha2` version of `TCPRoute`, and the Kubernetes Gateway provider
never completes its startup: no Gateway API resource is served, not only `TCPRoute`.
No error is logged, Traefik keeps running, and other providers are unaffected.
Traefik v3.7 remains backward compatible with the v1.5.x CRDs: upgrading the CRDs in the cluster is only required to rely on the v1.6.1 resources.
!!! note "Upcoming in v3.8"
Traefik `v3.8` will require updating the Gateway API CRDs to `v1.6.x`, and in return the `experimentalChannel` option will no longer be needed for `TCPRoute`.
(Optional) Apply v1.6.1 CRDs:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/standard-install.yaml
For the experimental channel (needed for TCPRoute):
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/experimental-install.yaml
Starting with v3.7.9, HTTP/1 CONNECT requests are rejected with a 501 Not Implemented response.
HTTP/1 CONNECT requests were not functional before this change, so this rejection makes it explicit.
errorRequestHeadersThe errorRequestHeaders option, introduced in v3.6.15 for the Errors middleware,
was not exposed on the Kubernetes CRD provider.
Starting with v3.7.8, it can now be configured on the Middleware CRD.
To use this new option, the Kubernetes CRDs must be updated in the cluster before upgrading Traefik. To do so, please apply the CRDs manifest for v3.7:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.7/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Please check out the Error Pages middleware documentation for more details.
From version v3.7.7 onwards, the Host matcher treats a bare * as a catch-all, consistent with the TCP HostSNI(*) matcher.
Host(`*`) now matches every request regardless of its host, including requests with no host at all.
Previously, the * was treated as a single wildcard label, so Host(`*`) only matched hosts made of a single segment (e.g. localhost) and not multi-segment hosts (e.g. example.com).
Please check out the HTTP routing rules documentation for more details.
Starting with v3.7.6, the Kubernetes Gateway API provider derives the name of the services and middlewares it generates from the route rule they belong to, instead of from the backend reference alone.
This is required to prevent distinct route rules referencing the same backend from colliding on a single generated configuration.
Generated service names were previously built from the backend reference only:
<backend namespace>-<backend name>-<port>
They are now prefixed with the route rule they are generated for:
<route kind>-<route namespace>-<route name>-gw-<gateway namespace>-<gateway name>-ep-<entry point>-<rule index>-<rule hash>-svc-<backend namespace>-<backend name>-<backend index>
For example, a whoami backend in the default namespace previously exposed as default-whoami-http-80@kubernetesgateway is now exposed as
httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-af329269dd38031b03e3-svc-default-whoami-0@kubernetesgateway.
!!! warning "Observability"
These names are user-visible: they appear in the dashboard and API, in the access logs `ServiceName` field, and in the `service` label of the metrics.
Dashboards, alerting rules, and log queries that match on Gateway API service names must be updated accordingly.
!!! warning "Deprecated since v3.7.12"
Please use the [`aliasHeadersStrategy`](../reference/install-configuration/entrypoints.md#opt-http-aliasHeadersStrategy) option instead,
which handles every aliasing character instead of the underscore only.
From version v3.6.20 onwards, a new underscoreHeadersStrategy entry point option defines how request headers with
underscores in their names are handled before routing:
keep (default): request headers with underscores are forwarded as is.delete: any request header whose name contains an underscore character is silently removed from the request.reject: any request carrying a header whose name contains an underscore character is rejected with a 400 Bad Request response.The default value is keep, so the existing behavior is preserved.
This option exists because underscores are valid characters in HTTP header names, but Go canonicalizes header names only on dashes.
As a result, a middleware managing a header in its dash form (e.g. X-Auth-User set by the ForwardAuth authResponseHeaders option)
does not see, and therefore cannot overwrite or remove, an underscore variant of that header (e.g. X_Auth_User).
Many backends map both forms to the same variable (CGI, WSGI, PHP, NGINX, ...): for them, X-Auth-User and X_Auth_User
are the same header. Against such a backend, a client can smuggle the underscore variant past a middleware that only manages
the dash form, and have the backend read the spoofed value, bypassing the protection the middleware was meant to provide.
!!! warning "Security"
When an entry point fronts a backend that interprets underscores and dashes in header names identically,
keeping the default `keep` strategy is not recommended, as it leaves the backend open to the header spoofing described above.
Set `underscoreHeadersStrategy` to `delete` or `reject` on such entry points.
Starting with v3.7.3, the QPS and Burst values of the Kubernetes client used by the Kubernetes Gateway API provider have been increased to 50 and 100 respectively (10x the default values of the Kubernetes client).
The Kubernetes Gateway API provider writes status updates intensively to comply with the Kubernetes Gateway API specification. This change helps avoid performance issues related to Kubernetes API rate limiting, which can increase the setup time when a new routing configuration is built.
These values are configurable through the kubernetesGateway.qps
and kubernetesGateway.burst provider options.
From version v3.7.3 onwards, the BasicAuth middleware requires a non-empty users configuration in order to be built successfully.
Previously, the middleware would be built successfully but always return a 401 status code for any request.
Now, an error occurs and any routers using it will be unmounted. For the same request, a 404 status code is served instead of a 401 status code.
From version v3.7.3 onwards, the StripPrefix middleware and the StripPrefixRegex middleware reject requests (400 Bad Request)
when stripping the configured prefix produces a path that differs from its normalised form
(i.e. a path containing . or .. segments that would be collapsed by normalisation).
This prevents the stripped path from being interpreted as a different resource by the upstream service.
Examples with a configured prefix of /api:
| Request path | Path after strip | Normalised path | Result |
|---|---|---|---|
/api/foo | /foo | /foo | 200 (sent) |
/api/ | / | / | 200 (sent) |
/api./foo | /./foo | /foo | 400 |
/api../foo | /../foo | /foo | 400 |
crossProviderNamespacesIn v3.7.1, a new crossProviderNamespaces option is available on the Kubernetes CRD, Ingress, and Gateway providers.
Traefik offers the possibility to reference resources from one provider to another (cross-provider references).
However, in the context of Kubernetes providers,
those references (e.g. myservice@kubernetescrd) allow a user to cross namespace boundaries,
as well as exposing @internal services, that only the operator should be able to expose.
This new crossProviderNamespaces option restricts in which namespaces Kubernetes resources are allowed to use cross-provider references.
The behavior is as follows:
| Value | Behavior |
|---|---|
| not set | All Kubernetes resources can declare cross-provider references. |
[] | Every Kubernetes resource declaring a cross-provider reference is rejected. |
["ns-a"] | Only Kubernetes resources in the listed namespaces can declare cross-provider references. |
Please check out the Kubernetes CRD, Kubernetes Ingress, and Kubernetes Gateway provider documentation for more details.
Starting with v3.7.0, the Ingress NGINX provider now supports the nginx.ingress.kubernetes.io/custom-headers annotation to add custom headers to the response forwarded to the client.
Therefore, in the corresponding RBACs (see KubernetesIngressNGINX provider RBACs) the configmaps right has been added.
Required RBAC Updates:
...
- apiGroups:
- ""
resources:
- configmaps
verbs:
- list
- watch
...
Starting with v3.7.0, the Kubernetes Gateway API provider supports version v1.5.1 of the specification,
which requires the Gateway API CRDs to be updated.
TLSRoute has graduated to the Standard channel and no longer requires the experimentalChannel option.
The experimentalChannel option is now only needed for TCPRoute.
Apply Updated CRDs:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml
For the experimental channel:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yaml
To use the new options of the retry middleware or the new ingressClassName field with the Kubernetes CRD provider, you need to update your CRDs.
Apply Updated CRDs:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.7/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Since v3.7.0, the Host and HostSNI matchers support wildcard subdomain matching (e.g., *.example.com).
This allows matching any direct subdomain of a domain with a single-level wildcard prefix.
For example, *.example.com matches foo.example.com but not foo.bar.example.com or example.com itself.
This feature is only available with the v3 rule syntax (the default).
Since v3.7.0, TLSOptions can now be associated with routers using wildcard Host and HostSNI matchers (e.g., Host(*.example.com)).
This enables configuring different TLS options for wildcard domains.
Previously, TLSOptions selection was limited to exact Host matches, and using HostRegexp or wildcards would fall back to the default TLS options with a warning message like: No domain found in rule HostRegexp(...) the TLS option foo cannot be applied.
Note: TLSOptions for HostRegexp matchers remains unsupported. Use wildcard Host matchers as an alternative.
Starting with v3.6.25, the Kubernetes Gateway API provider derives the hash suffix of the router names it generates from all the route, Gateway, and listener identifying fields, instead of from the route rule alone.
This is required to prevent distinct routes whose namespace, name, Gateway, and listener happen to concatenate to the same string from colliding on a single generated router name.
Generated router names keep the same overall shape:
<route kind>-<route namespace>-<route name>-gw-<gateway namespace>-<gateway name>-ep-<entry point>-<rule index>-<hash>
but the <hash> suffix changes, since it is no longer computed from the route rule alone.
For example, an HTTPRoute named http-app-1 in the default namespace previously generated the router httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-af329269dd38031b03e3 and is now generated as
httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-799bcbf0c2317c5d1c93.
Middleware names are derived from the router name, so they change accordingly, for example <router name>-requestheadermodifier-0.
Generated service names were previously built from the backend reference only:
<backend namespace>-<backend name>-<port>
They are now prefixed with the route rule they are generated for:
<router name>-svc-<backend namespace>-<backend name>-<backend index>
For example, a whoami backend in the default namespace previously exposed as default-whoami-http-80@kubernetesgateway is now exposed as
httproute-default-http-app-1-gw-default-my-gateway-ep-web-0-799bcbf0c2317c5d1c93-svc-default-whoami-0@kubernetesgateway.
!!! warning "Observability"
These names are user-visible: they appear in the dashboard and API, in the access logs `RouterName` and `ServiceName` fields, and in the `router`/`service` labels of the metrics.
Dashboards, alerting rules, and log queries that match on Gateway API router, middleware, or service names must be updated accordingly.
errorRequestHeadersThe errorRequestHeaders option, introduced in v3.6.15 for the Errors middleware,
was not exposed on the Kubernetes CRD provider.
Starting with v3.6.24, it can now be configured on the Middleware CRD.
To use this new option, the Kubernetes CRDs must be updated in the cluster before upgrading Traefik. To do so, please apply the CRDs manifest for v3.6:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Please check out the Error Pages middleware documentation for more details.
Starting with v3.6.24, HTTP/1 CONNECT requests are rejected with a 501 Not Implemented response.
HTTP/1 CONNECT requests were not functional before this change, so this rejection makes it explicit.
!!! warning "Deprecated since v3.7.12"
Please use the [`aliasHeadersStrategy`](../reference/install-configuration/entrypoints.md#opt-http-aliasHeadersStrategy) option instead,
which handles every aliasing character instead of the underscore only.
From version v3.6.22 onwards, a new underscoreHeadersStrategy entry point option defines how request headers with
underscores in their names are handled before routing:
keep (default): request headers with underscores are forwarded as is.delete: any request header whose name contains an underscore character is silently removed from the request.reject: any request carrying a header whose name contains an underscore character is rejected with a 400 Bad Request response.The default value is keep, so the existing behavior is preserved.
This option exists because underscores are valid characters in HTTP header names, but Go canonicalizes header names only on dashes.
As a result, a middleware managing a header in its dash form (e.g. X-Auth-User set by the ForwardAuth authResponseHeaders option)
does not see, and therefore cannot overwrite or remove, an underscore variant of that header (e.g. X_Auth_User).
Many backends map both forms to the same variable (CGI, WSGI, PHP, NGINX, ...): for them, X-Auth-User and X_Auth_User
are the same header. Against such a backend, a client can smuggle the underscore variant past a middleware that only manages
the dash form, and have the backend read the spoofed value, bypassing the protection the middleware was meant to provide.
!!! warning "Security"
When an entry point fronts a backend that interprets underscores and dashes in header names identically,
keeping the default `keep` strategy is not recommended, as it leaves the backend open to the header spoofing described above.
Set `underscoreHeadersStrategy` to `delete` or `reject` on such entry points.
Starting with v3.6.19, the QPS and Burst values of the Kubernetes client used by the Kubernetes Gateway API provider have been increased to 50 and 100 respectively (10x the default values of the Kubernetes client).
The Kubernetes Gateway API provider writes status updates intensively to comply with the Kubernetes Gateway API specification. This change helps avoid performance issues related to Kubernetes API rate limiting, which can increase the setup time when a new routing configuration is built.
These values are configurable through the kubernetesGateway.qps
and kubernetesGateway.burst provider options.
From version v3.6.19 onwards, the BasicAuth middleware requires a non-empty users configuration in order to be built successfully.
Previously, the middleware would be built successfully but always return a 401 status code for any request.
Now, an error occurs and any routers using it will be unmounted. For the same request, a 404 status code is served instead of a 401 status code.
From version v3.6.19 onwards, the StripPrefix middleware and the StripPrefixRegex middleware reject requests (400 Bad Request)
when stripping the configured prefix produces a path that differs from its normalised form
(i.e. a path containing . or .. segments that would be collapsed by normalisation).
This prevents the stripped path from being interpreted as a different resource by the upstream service.
Examples with a configured prefix of /api:
| Request path | Path after strip | Normalised path | Result |
|---|---|---|---|
/api/foo | /foo | /foo | 200 (sent) |
/api/ | / | / | 200 (sent) |
/api./foo | /./foo | /foo | 400 |
/api../foo | /../foo | /foo | 400 |
crossProviderNamespacesIn v3.6.17, a new crossProviderNamespaces option is available on the Kubernetes CRD, Ingress, and Gateway providers.
Traefik offers the possibility to reference resources from one provider to another (cross-provider references).
However, in the context of Kubernetes providers,
those references (e.g. myservice@kubernetescrd) allow a user to cross namespace boundaries,
as well as exposing @internal services, that only the operator should be able to expose.
This new crossProviderNamespaces option restricts in which namespaces Kubernetes resources are allowed to use cross-provider references.
The behavior is as follows:
| Value | Behavior |
|---|---|
| not set | All Kubernetes resources can declare cross-provider references. |
[] | Every Kubernetes resource declaring a cross-provider reference is rejected. |
["ns-a"] | Only Kubernetes resources in the listed namespaces can declare cross-provider references. |
Please check out the Kubernetes CRD, Kubernetes Ingress, and Kubernetes Gateway provider documentation for more details.
Starting with v3.6.16, the Docker provider requires Docker API version v1.40 or
above (Docker Engine v19.03).
Users running older (end of life) versions of Docker Engine should update their
Docker Engine or use the DOCKER_API_VERSION
environment variable to override the API version used by Traefik.
In v3.6.15, a new errorRequestHeaders option has been added to the Errors middleware.
By default, the behavior is unchanged: all original request headers are forwarded to the error page service.
If the error page service is in a separate trust domain, consider using errorRequestHeaders to restrict which headers are forwarded.
Please check out the Error Pages middleware documentation for more details.
allowCrossNamespaceIn v3.6.14, the Chain middleware now honors the Kubernetes CRD provider's allowCrossNamespace option.
Previously, a Chain could reference middlewares in other namespaces regardless of the allowCrossNamespace configuration.
If allowCrossNamespace is set to false (the default) and a Chain middleware references a middleware in a different namespace from its own,
the whole Chain is now rejected and an error is logged.
trustForwardHeaderStarting with v3.6.14, the trustForwardHeader option has been deprecated and will be removed in the next major version.
Configure the trusted IPs at the EntryPoint level by using the forwardedHeaders.trustedIPs option,
and set trustForwardHeader to true on this middleware.
When trustForwardHeader is not explicitly set, Traefik logs a warning as its behavior is inconsistent:
some X-Forwarded-* headers (e.g. X-Forwarded-For, X-Forwarded-Proto) are removed while others (e.g. X-Forwarded-Prefix) are forwarded untouched.
To silence the warning and avoid security concerns, explicitly set trustForwardHeader to true or false in your ForwardAuth middleware configuration.
Please check out the ForwardAuth middleware documentation for more details.
maxResponseBodySize configuration on ForwardAuth middlewareIn v3.6.9, a new maxResponseBodySize option has been added to the ForwardAuth middleware configuration.
The default value for this option is -1, which means there is no limit to the response body size.
However, it is strongly recommended to set this option to a suitable value to avoid performance and security issues,
such as DoS attacks and memory exhaustion.
Please check out the ForwardAuth middleware documentation for more details.
To use the new maxResponseBodySize option in the ForwardAuth middleware with the Kubernetes CRD provider, you need to update your CRDs.
Apply Updated CRDs:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Since v3.6.8, the configured path for the health check request is now verified to be a relative URL, and the health check will fail if it is not.
Since v3.6.7, the options for encoded characters now have a true default value.
This means that Traefik will not reject requests with a path containing a specific set of encoded characters by default.
It is now up to the users to configure the security hardening of encoded characters.
Here is the list of the encoded characters that can be configured to false to disallow them:
| Encoded Character | Character | Config options | Default value |
|---|---|---|---|
%2f or %2F | / (slash) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedSlash | true | ||
%5c or %5C | \ (backslash) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedBackSlash | true | ||
%00 | NULL (null character) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedNullCharacter | true | ||
%3b or %3B | ; (semicolon) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedSemicolon | true | ||
%25 | % (percent) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedPercent | true | ||
%3f or %3F | ? (question mark) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedQuestionMark | true | ||
%23 | # (hash) | entryPoints.<name> | |
.http.encodedCharacters | |||
.allowEncodedHash | true |
Note: This check is not done against query parameters, but only against the request path as defined in RFC3986 section-3.
Please check out the entrypoint encodedCharacters option documentation for more details.
Starting with v3.6.4, for security reasons, Traefik now rejects requests with a path containing a specific set of encoded characters by default.
When such a request is received, Traefik responds with a 400 Bad Request status code.
Here is the list of the encoded characters that are rejected by default, along with the corresponding configuration option to allow them:
| Encoded Character | Character | Config option to allow the encoded character |
|---|---|---|
%2f or %2F | / (slash) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedSlash | ||
%5c or %5C | \ (backslash) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedBackSlash | ||
%00 | NULL (null character) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedNullCharacter | ||
%3b or %3B | ; (semicolon) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedSemicolon | ||
%25 | % (percent) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedPercent | ||
%3f or %3F | ? (question mark) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedQuestionMark | ||
%23 | # (hash) | entryPoints.<name> |
.http.encodedCharacters | ||
.allowEncodedHash |
Please check out the entrypoint encodedCharacters option documentation for more details.
The KubernetesIngressNGINX Provider is no longer experimental in v3.6.2 and can be enabled without the experimental.kubernetesIngressNGINX option.
Deprecated Configuration:
??? example "Experimental kubernetesIngressNGINX option (deprecated)"
```yaml tab="File (YAML)"
experimental:
kubernetesIngressNGINX: true
```
```toml tab="File (TOML)"
[experimental]
kubernetesIngressNGINX=true
```
```bash tab="CLI"
--experimental.kubernetesIngressNGINX=true
```
Migration Steps:
kubernetesIngressNGINX option from the experimental sectionStarting with v3.6.0, the Kubernetes Gateway API provider only supports version v1.4.0 of the specification,
which requires the Gateway API CRDs to be updated.
Apply Updated CRDs:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml
For the experimental channel:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/experimental-install.yaml
To use the new leasttime load-balancer algorithm with the Kubernetes CRD provider, you need to update your CRDs.
Apply Updated CRDs:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Starting with v3.5.4, and when using OpenTelemetry, the traefik_tls_certs_not_after_milliseconds metric is renamed to traefik_tls_certs_not_after_seconds.
This change aligns the metric name with its real unit precision, which is in seconds.
Starting with v3.5.2, the proxyProtocol option for TCP LoadBalancer is deprecated.
This option can now be configured at the TCPServersTransport level, please check out the documentation for more details.
To use the new proxyprotocol option in the Kubernetes CRD provider, you need to update your CRDs.
Apply Updated CRDs:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.5/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Starting with v3.5.0, a new traceVerbosity option is available for both entrypoints and routers.
This option allows you to control the level of detail for tracing spans.
Routers can override the value inherited from their entrypoint.
Impact:
minimal unless overridden, which will result in fewer spans being generated than before.Possible values are:
minimal: produces a single server span and one client span for each request processed by a router.detailed: enables the creation of additional spans for each middleware executed for each request processed by a router.See the updated documentation for entrypoints and dynamic routers.
Since v3.5.0, the semconv attributes k8s.pod.name and k8s.pod.uid are injected automatically in OTel resource attributes when OTel tracing/logs/metrics are enabled.
For that purpose, the following right has to be added to the Traefik Kubernetes RBACs:
...
- apiGroups:
- ""
resources:
- pods
verbs:
- get
...
Since v3.4.5, the MultiPath TCP support introduced with v3.4.2 has been removed.
It appears that enabling MPTCP on some platforms can cause Traefik to stop with the following error logs message:
set tcp X.X.X.X:X->X.X.X.X:X: setsockopt: operation not supportedHowever, it can be re-enabled by setting the multipathtcp variable in the GODEBUG environment variable, see the related go documentation.
Starting with v3.4.1, request paths are now normalized according to RFC 3986 standards for better consistency and security.
Normalization Process:
%2E (.) are decoded to their literal form%2e becomes %2E)This follows RFC 3986 percent-encoding normalization and case normalization standards.
Processing Order:
Starting with v3.4.1, reserved characters (per RFC 3986) remain encoded during router rule matching to prevent routing ambiguity.
Why This Matters: Reserved characters change the meaning of request paths when decoded. Keeping them encoded during routing prevents security vulnerabilities and ensures predictable routing behavior.
The following table illustrates how path matching behavior has changed:
| Request Path | Router Rule | Traefik v3.4.0 | Traefik v3.4.1 | Explanation |
|---|---|---|---|---|
/foo%2Fbar | PathPrefix(`/foo/bar`) | Match | No match | %2F (/) stays encoded, preventing false matches |
/foo/../bar | PathPrefix(`/foo`) | No match | No match | Path traversal is sanitized away |
/foo/../bar | PathPrefix(`/bar`) | Match | Match | Resolves to /bar after sanitization |
/foo/%2E%2E/bar | PathPrefix(`/foo`) | Match | No match | Encoded dots normalized then sanitized |
/foo/%2E%2E/bar | PathPrefix(`/bar`) | No match | Match | Resolves to /bar after normalization + sanitization |
Starting with v3.4, HTTP service definitions now support additional load-balancing strategies for better traffic distribution.
Apply Updated CRDs:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.4/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
New Strategy Values:
wrr (Weighted Round Robin)p2c (Power of Two Choices)!!! warning "Deprecation"
The RoundRobin strategy is deprecated but still supported (equivalent to wrr). It will be removed in the next major release.
Refer to the HTTP Services Load Balancing documentation for detailed information.
A new rootCAs option has been added to the ServersTransport and ServersTransportTCP CRDs. It supports both ConfigMaps and Secrets for CA certificates and replaces the rootCAsSecrets option.
Apply Updates:
# Update CRDs
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.4/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
# Update RBACs
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.4/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml
New Configuration Format:
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: foo
namespace: bar
spec:
rootCAs:
- configMap: ca-config-map
- secret: ca-secret
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransportTCP
metadata:
name: foo
namespace: bar
spec:
rootCAs:
- configMap: ca-config-map
- secret: ca-secret
!!! warning "Deprecation"
The rootCAsSecrets option (Secrets only) is still supported but deprecated. It will be removed in the next major release.
In v3.4, rule syntax configuration options will be removed in the next major version.
Deprecated Options:
core.defaultRuleSyntax (static configuration)ruleSyntax (router option)These options were transitional helpers for migrating from v2 to v3 syntax. Please ensure all router rules use v3 syntax before the next major release.
Starting with v3.3.6, incoming request paths are now automatically cleaned before processing for security and consistency.
What's Changed:
The following path segments are now interpreted and collapsed:
/../ (parent directory references)/./ (current directory references)//)Disabling Sanitization:
# EntryPoint HTTP configuration
entryPoints:
web:
address: ":80"
http:
sanitizePath: false # Not recommended
!!! danger "Security Warning"
Setting sanitizePath: false is not safe. This option should only be used with legacy clients that don't properly URL-encode data. Always ensure requests are properly URL-encoded instead of disabling this security feature.
Example Risk: Base64 data containing "/" characters can lead to unsafe routing when path sanitization is disabled and the data isn't URL-encoded.
In v3.3.5, the default compression algorithms have been reordered to favor gzip compression.
New Default: gzip, br, zstd
This change affects requests that either:
Accept-Encoding headerAccept-Encoding headerThe reordering helps ensure better compatibility with older clients that may not support newer compression algorithms.
In v3.3.4, the OpenTelemetry Request Duration metric unit has been standardized to match other providers and naming conventions.
Change Details:
traefik_(entrypoint|router|service)_request_duration_secondsThis change ensures consistency across all metrics providers and follows standard naming conventions.
In v3.3, DNS challenge configuration options have been reorganized for better clarity.
Migration Required:
| Deprecated Option | New Option |
|---|---|
acme.dnsChallenge.delaybeforecheck | acme.dnsChallenge.propagation.delayBeforeChecks |
acme.dnsChallenge.disablepropagationcheck | acme.dnsChallenge.propagation.disableChecks |
In v3.3, the tracing configuration has been clarified to better reflect its purpose.
Migration Required:
tracing.globalAttributestracing.resourceAttributesThe old option name was misleading as it specifically adds resource attributes for the collector, not global span attributes.
In v3.2.2, Swarm-specific labels have been deprecated and will be removed in a future version.
Migration Required:
| Deprecated Label | New Label |
|---|---|
traefik.docker.network | traefik.swarm.network |
traefik.docker.lbswarm | traefik.swarm.lbswarm |
X-Forwarded-Prefix Header ChangesIn v3.2.1, the X-Forwarded-Prefix header is now handled like other X-Forwarded-* headers - Traefik removes it when sent from untrusted sources.
This change improves security by preventing header spoofing from untrusted clients. Refer to the Forwarded headers documentation for configuration details.
New optional fields have been added to several CRDs. These updates are backward compatible and only add new functionality.
Apply the latest CRDs:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v3.3/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
Updated Resources:
Starting with v3.2, the Kubernetes Gateway Provider now supports GRPCRoute resources.
Therefore, in the corresponding RBACs (see KubernetesGateway provider RBACs),
the grcroutes and grpcroutes/status rights have to be added.
Required RBAC Updates:
...
- apiGroups:
- gateway.networking.k8s.io
resources:
- grpcroutes
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- grpcroutes/status
verbs:
- update
...
Due to breaking changes in Kubernetes Gateway v1.2.0-rc1, Traefik v3.3 only supports Kubernetes Gateway v1.2.x when experimental features are enabled.
New Feature: BackendTLSPolicy Support
The provider now supports BackendTLSPolicy resources.
Therefore, in the corresponding RBACs (see KubernetesGateway provider RBACs),
the backendtlspolicies and backendtlspolicies/status rights have to be added.
Required RBAC Updates:
...
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- backendtlspolicies
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- backendtlspolicies/status
verbs:
- update
...
The disableIngressClassLookup option has been deprecated and will be removed in the next major version.
Migration Required:
disableIngressClassLookupdisableClusterScopeResourcesThe new option provides broader control over cluster scope resources discovery, including both IngressClass and Nodes resources.
Starting with v3.1, Traefik's Kubernetes Providers use the EndpointSlices API (requires Kubernetes >=v1.21) for service endpoint discovery. This change also introduces NodePort load-balancing capabilities.
The following RBAC updates are required for all Kubernetes providers:
# Remove this section from your RBAC
# - apiGroups: [""]
# resources: ["endpoints"]
# verbs: ["get", "list", "watch"]
# Add this section instead
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
!!! note "Affected Providers" These changes apply to:
- [KubernetesIngress](../reference/install-configuration/providers/kubernetes/kubernetes-ingress.md) provider
- [KubernetesCRD](../reference/install-configuration/providers/kubernetes/kubernetes-crd.md#requirements) provider
- [KubernetesGateway](../reference/install-configuration/providers/kubernetes/kubernetes-gateway.md#requirements) provider
The KubernetesGateway Provider is no longer experimental in v3.1 and can be enabled without the experimental.kubernetesgateway option.
Deprecated Configuration:
??? example "Experimental kubernetesgateway option (deprecated)"
```yaml tab="File (YAML)"
experimental:
kubernetesgateway: true
```
```toml tab="File (TOML)"
[experimental]
kubernetesgateway=true
```
```bash tab="CLI"
--experimental.kubernetesgateway=true
```
Migration Steps:
kubernetesgateway option from the experimental section