docs-v1/design_proposals/configurable-kubecontext.md
Note :exclamation: As the global config option adds considerable complexity, that part of this design proposal has been put on hold to await more evidence. Ideally, the CLI and
skaffold.yamloptions are already enough for the great majority of Skaffold users.
So far, Skaffold always uses the currently active kubecontext when interacting with a Kubernetes cluster. This is problematic when users want to deploy multiple projects with different kubecontexts, because the user needs to manually switch the context before starting Skaffold. In particular when working on multiple such projects in parallel, the current behavior is limiting.
Open issues concerning this problem are
There also was an attempt to add a configuration option to skaffold.yaml (Support for overriding kubectl context during deployment #1540).
The goal of this document is to create an agreement on what options should be supported and identify edge cases.
Use CLI flag or environment variable.
Use the kubecontext configuration in skaffold.yaml.
Think twice before using this approach in open source projects, as the setting will not be portable.
Use kubecontext setting in the global Skaffold config (via skaffold config set ...).
Use the kubecontext configuration in skaffold.yaml.
There are four places where kubecontext activation can be added:
<table> <thead> <th>Precedence</th> <th>Kind</th> <th>Details</th> </thead> <tbody> <tr> <td>1. (highest)</td> <td>CLI option</td> <td> The Kubernetes standard to set the kubecontext is <code>--context</code>. However, in Skaffold this term is so overloaded that it should more precisely be named <code>--kube-context</code>. This flag is necessary for IDE integration. </td> </tr> <tr> <td>2.</td> <td>Env variable</td> <td> <code>SKAFFOLD_KUBE_CONTEXT</code>, similar to other Skaffold flags. </td> </tr> <tr> <td>3.</td> <td><code>skaffold.yaml</code></td> <td> Json-path <code>deploy.kubeContext</code>. This option is shareable, and requires some error handling for profile activation by kubecontext (see below). </td> </tr> <tr> <td>4. (lowest)</td> <td>Global Skaffold config</td> <td> This should give users the possibility to define a default context globally or per project. This variant is not shareable. </td> </tr> </tbody> </table>Beside the kubecontext, also the namespace needs to be specified. Ideally, the namespace should also offer the same override variants as kubecontext. This is out of scope for this design proposal. As long as this is not implemented, there is always the workaround, to duplicate a kubecontext and set the default namespace for this kubecontext to the desired value. Then this kubecontext/namespace pair can be activated with the kubecontext activation machinery detailed in this design proposal.
skaffold.yamlA configuration option in skaffold.yaml has the advantage of being most discoverable:
it is in the place where users configure all aspects of their pipeline.
In addition, it allows to define the kubecontext per Skaffold profile.
A natural place for the config in skaffold.yaml is in latest.DeployConfig, resulting in a json path deploy.kubeContext.
Profiles have a double role, because they may override the kubecontext to a different value as before, but they may also be activated by a kubecontext. To catch unexpected behavior, the profile activation will perform the following logic:
It is therefore possible that subsequent profiles repeatedly switch the kubecontext, in which case the last one wins.
For example, the following sequence will result in an error:
minikube and activates some profile.gke_abc_123.minikube-specific profile would be deployed to gke_abc_123, and this will be forbidden.Note: skaffold.yaml is meant to be shared, but kubecontext names vary across users.
Sharing therefore makes only sense in a corporate setting where context names are the same across different machines.
Specifying a default kubecontext globally is straightforward. For example, via new config option
global:
default-context: docker-for-desktop
On the other hand, building a relation between projects and kubecontext needs to solve two questions:
There are at least three possibilities:
metadata.name entry in skaffold.yaml (see also #2200).
This has the drawback of being potentially not unique, so that users accidentally pin the kubecontext for more projects than intended.
On the other hand, this is the standard approach taken by Kubernetes resources.<What option has the best tradeoffs?>
Resolution: We will go ahead with the metadata.name approach. As the name may not be unique, this requires special documentation.
Currently, the Skaffold config uses the kubecontext as identifier.
There are two possibilities to add the relation:
kube-context entries:
kubecontexts:
- kube-context: my-context
skaffoldConfigs:
- config-name
kubecontexts:
- kube-context: context1
skaffoldConfigs:
- my-project
- kube-context: context2
skaffoldConfigs:
- my-project
global: {}
kubecontexts: []
skaffoldConfigs:
my-project: my-context
my-other-project: my-other-context
'*': default-context
skaffold config.<What Skaffold config structure has the best tradeoffs?>
Resolution: The top-level entry (option 2) overall has the better trade-offs.
skaffold.yaml variant. #2510skaffoldConfigs).skaffoldConfigs['*']).A single test covers the overall kubecontext override functionality sufficiently (part of #2447). Other test-cases such as precedence of the different variants and error cases should be covered by unit tests.