docs/proposals/parameterized-config-management-plugins.md
Config Management Plugin (CMP) parameterization defines a way for plugins to "announce" and then consume acceptable parameters for an Application. Announcing parameters allows CMPs to provide a UI experience similar to native config management tools (Helm, Kustomize, etc.).
Should we write examples in documentation in Python instead of shell scripts?
It's very easy to write an insecure shell script. People copy/paste code from documentation to start their own work. Maybe by using a different language in examples, we can encourage more secure CMP development.
Config Management Plugins allow Argo CD administrators to define custom manifest generation tooling.
The only existing way for users to parameterize manifest generation is with environment variables.
This proposed feature will allow a plugin to "announce" acceptable parameters for an Application. It will also allow the plugin to consume parameters once the user has set them.
Parameters definitions may be simple (advertising a simple key/value string pair) or more complex (accepting an array of strings or a map of string keys to string values). Parameter definitions can also specify a data type (string, number, or boolean) to help the UI present the most relevant input field.
CMPs, especially the sidecar type, are under-utilized. Making them more robust will increase adoption. Increased adoption will help us find bugs and then make CMPs more robust. In other words, we need to reach a critical mass of CMP users.
More robust CMPs will make it easier to start supporting tools like Tanka.
For example, there's a Helm bug affecting Argo CD users. The fix would involve importing the Helm SDK (a very large dependency) into Argo CD. Implementing Helm support as a CMP would allow us to use that SDK without embedding it in the core code.
Offloading Ksonnet to a plugin would allow us to support existing users without maintaining Ksonnet code in the more actively-developed base. But we need CMP parameters to provide Ksonnet support on-par with native support.
Parameterized CMPs must be:
An Argo CD admin should be able to write a simple parameterized CMP in just a few lines of code.
An Argo CD admin should be able to write an advanced parameterized CMP server relying on thorough docs.
Writing a custom CMP server might be preferable if the parameters announcement code gets too complex to be an inline shell script.
We should not:
As an Argo CD developer, I would like to be able to build Argo CD without including the Helm SDK as a dependency.
The Helm SDK includes the Kubernetes code base. That's a lot of code, and it will make builds unacceptably slow.
As an Argo CD user, I would like to be able to parameterize manifests built by a CMP.
For example, if the Argo CD administrator has installed a CMP which applies a last-mile kustomize overlay to a Helm repo, I would like to be able to pass values to the Helm chart without having to manually discover those parameter names (in other words, they should show up in the Application UI just like with a native Helm Application). I also shouldn't have to ask my Argo CD admin to modify the CMP to accommodate the values as environment variables.
Since this proposal is designed to increase CMP adoption, we need to make sure there aren't any bugs that make CMPs less robust than native tools.
Bugs to fix:
argocd app sync/diff --local doesn't account for sidecar CMPsParameter announcement: an instance of a data structure which describes an individual parameter that may be applied to a specific Application. (See the schema below.)
Parameters announcement: a list of parameter announcements. (See the schema below.)
"Parameters" is plural because each "announcement" will be a list of multiple parameter announcements.
Parameterized CMP: a CMP which supports rich parameters (i.e. more than environment variables). A CMP is parameterized if either of these is true:
This proposal adds a new parameters key to the ConfigManagementPlugin config spec.
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: cmp-plugin
spec:
version: v1.0
generate:
command: ["example.sh"]
discover:
fileName: "./subdir/s*.yaml"
# NEW KEY
parameters:
static:
# The static announcement follows the parameters announcement schema. This is where a parameter description
# should go if it applies to all apps for this CMP.
- name: values-file
title: Values File
tooltip: Path of a Helm values file to apply to the chart.
dynamic:
# The (optional) generated announcement is combined with the declarative announcement (if present). This is where
# a parameter description should be generated if it applies only to a specific app which the CMP handles.
command: ["example-params.sh"]
The currently-configured parameters (if there are any) will be communicated to both generate.command and
parameters.dynamic.command via an ARGOCD_APP_PARAMETERS environment variable. The parameters will be encoded
according to the parameters serialization format defined below.
Passing the parameters to the parameters.dynamic.command will allow configuration of parameter discovery. For example,
if my CMP is designed to handle Kustomize projects which contain Helm charts, I might have the CMP accept an
ignore-helm-charts parameter to avoid announcing parameters for those charts.
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
source:
plugin:
parameters:
- name: ignore-helm-charts
array: [chart-a, chart-b]
Users persist parameter values in an Application's spec.source.plugin.parameters list.
Each parameter has a name and a value stored in the string, array, or map field, according to the parameter's
collectionType. The name should match the name of some parameter announced by the CMP. (But
the user can set any parameter name, so it's the CMP's job to ignore invalid parameters.)
This example is for a hypothetical Helm CMP. This CMP accepts a values and a values-files parameter.
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
source:
repoURL: https://github.com/argoproj/argocd-example-apps.git
plugin:
parameters:
- name: values
string: >-
resources:
cpu: 100m
memory: 128Mi
- name: values-files
array: [values.yaml]
- name: helm-parameters
map:
image.repository: my.example.com/gcr-proxy/heptio-images/ks-guestbook-demo
image.tag: "0.1"
When Argo CD generates manifests (for example, when the user clicks "Hard Refresh" in the UI), Argo CD will send these
parameters to the CMP as JSON (using the equivalent structure to what's shown above) on an environment variable called
ARGOCD_APP_PARAMETERS.
echo "$ARGOCD_APP_PARAMETERS" | jq
That command, when run by a CMP with the above Application manifest, will print the following:
[
{
"name": "values",
"string": "resources:\n cpu: 100m\n memory: 128Mi"
},
{
"name": "values-files",
"array": ["values.yaml"]
},
{
"name": "helm-parameters",
"map": {
"image.repository": "my.example.com/gcr-proxy/heptio-images/ks-guestbook-demo",
"image.tag": "0.1"
}
}
]
Another way the CMP can access parameters is via environment variables. For example:
echo "$VALUES" > /tmp/values.yaml
helm template --values /tmp/values.yaml .
Environment variable names are set according to these rules:
string, the format is PARAM_{escaped(name)} (escaped is defined below).array, the format is PARAM_{escaped(name_{index})} (where the first index is 0).map, the format is PARAM_{escaped(name_key)}.The escaped function will perform the following tasks:
[^A-Z0-9_].The above example will produce the following env vars:
echo "$PARAM_VALUES"
echo "$PARAM_VALUES_FILES_0"
echo "$PARAM_HELM_PARAMETERS_IMAGE_REPOSITORY"
echo "$PARAM_HELM_PARAMETERS_IMAGE_TAG"
The parameters in the Application manifest are represented behind the scenes with the following Go types:
package cmp
// Parameter represents a single parameter name and its value. One of Value, Map, or Array must be set.
type Parameter struct {
// Name is the name identifying a parameter. (required)
Name string `json:"name,omitempty"`
String string `json:"string,omitempty"`
Map map[string]string `json:"map,omitempty"`
Array []string `json:"array,omitempty"`
}
// Parameters is a list of parameters to be sent to a CMP for manifest generation.
type Parameters []Parameter
The CMP developer will have two ways to announce acceptable parameters: statically (declaratively) and dynamically.
Static parameter announcements are written directly into the CMP config file:
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: helm
spec:
parameters:
static:
- name: values-files
title: Values Files
collectionType: array
Since this hypothetical Helm CMP will accept an array of values.yaml files for every app it handles, the CMP developer can add that parameter as a static parameter announcement in the CMP config.
Dynamic parameters are generated by a CMP developer-defined command.
A parameter definition is an object with following schema:
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: helm
spec:
parameters:
dynamic:
command:
- sh
- -c
- |
# Use yq to generate a list of parameters. Then use jq to convert that list of parameters to a parameters
# announcement list.
yq e -o=p values.yaml | jq -nR '
[{
name: "helm-parameters",
title: "Helm Parameters",
tooltip: "Parameters to override when generating manifests with Helm",
collectionType: "map",
map: (inputs | capture("(?<key>.*) = (?<value>.*)") | from_entries)
}]'
For a Helm chart with only an image.repository and image.tag in values.yaml, the parameter announcement would look
like this:
[
{
"name": "helm-parameters",
"collectionType": "map",
"title": "Helm Parameters",
"tooltip": "Parameters to override when generating manifests with Helm",
"map": {
"image.repository": "my.example.com/gcr-proxy/heptio-images/ks-guestbook-demo",
"image.tag": "0.1"
}
}
]
Before sending a parameters announcement to the UI, the CMP server will combine the static and dynamic parameters. (Behind the scenes, the list is actually communicated to the UI via gRPC, but they're presented here as JSON for readability.)
[
{
"name": "values-files",
"title": "Values Files",
"collectionType": "array"
},
{
"name": "helm-parameters",
"collectionType": "map",
"title": "Helm Parameters",
"tooltip": "Parameters to override when generating manifests with Helm",
"map": {
"image.repository": "my.example.com/gcr-proxy/heptio-images/ks-guestbook-demo",
"image.tag": "0.1"
}
}
]
This is the full parameters announcement schema as Go types.
package cmp
// ParameterItemType is the primitive data type of each of the parameter's value (or each of its values, if it's an array or
// a map).
type ParameterItemType string
// Anything besides "number" and "boolean" is treated as string.
const (
ParameterItemTypeNumber ParameterItemType = "number"
ParameterItemTypeBoolean ParameterItemType = "boolean"
)
// ParameterCollectionType is a parameter's value's type - a single value (like a string) or a collection (like an array or a
// map).
type ParameterCollectionType string
// Anything besides "number" and "boolean" is treated as string.
const (
ParameterCollectionTypeMap ParameterCollectionType = "map"
ParameterCollectionTypeArray ParameterCollectionType = "array"
)
// ParameterAnnouncement represents a CMP's announcement of one acceptable parameter (though that parameter may contain
// multiple elements, if the value holds an array or a map).
type ParameterAnnouncement struct {
// Name is the name identifying a parameter. (required)
Name string `json:"name,omitempty"`
// Title is a human-readable text of the parameter name. (optional)
Title string `json:"title,omitempty"`
// Tooltip is a human-readable description of the parameter. (optional)
Tooltip string `json:"tooltip,omitempty"`
// Required defines if this given parameter is mandatory. (optional: default false)
Required bool `json:"required,omitempty"`
// ItemType determines the primitive data type represented by the parameter. Parameters are always encoded as
// strings, but ParameterTypes lets them be interpreted as other primitive types.
ItemType ParameterItemType `json:"itemType,omitempty"`
// CollectionType is the type of value this parameter holds - either a single value (a string) or a collection (array or map).
// If Type is set, only the field with that type will be used. If Type is not set, `string` is the default. If Type
// is set to an invalid value, a validation error is thrown.
CollectionType ParameterCollectionType `json:"collectionType,omitempty"`
String string `json:"string,omitempty"`
Map map[string]string `json:"map,omitempty"`
Array []string `json:"array,omitempty"`
}
// ParametersAnnouncement is a list of announcements. This list represents all the parameters which a CMP is able to
// accept.
type ParametersAnnouncement []ParameterAnnouncement
Question: What do we do if the CMP announcement sets more than one value.{collection}?
Answer: We ignore all but the configured collectionType.
- name: images
collectionType: map
array: # this gets ignored because collectionType is 'map'
- ubuntu:latest=docker.example.com/proxy/ubuntu:latest
- guestbook:v0.1=docker.example.com/proxy/guestbook:v0.1
map:
ubuntu:latest: docker.example.com/proxy/ubuntu:latest
guestbook:v0.1: docker.example.com/proxy/guestbook:v0.1
Question: What do we do if the CMP user sets more than one of value/array/map in the Application spec?
Answer: We send all given information to the CMP and allow it to select the relevant field.
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
source:
plugin:
parameters:
- name: images
array: # this gets sent to the CMP, but the CMP should ignore it
- ubuntu:latest=docker.example.com/proxy/ubuntu:latest
- guestbook:v0.1=docker.example.com/proxy/guestbook:v0.1
map:
ubuntu:latest: docker.example.com/proxy/ubuntu:latest
guestbook:v0.1: docker.example.com/proxy/guestbook:v0.1
Question: How will the UI know that adding more items to an array or a map is allowed?
Answer: Always assume it's allowed to add to a map or array.
- name: images
collectionType: map # users will be allowed to add new items, because this is a map
map:
ubuntu:latest: docker.example.com/proxy/ubuntu:latest
guestbook:v0.1: docker.example.com/proxy/guestbook:v0.1
If the CMP author wants an immutable array or map, they should just break it into individual parameters.
- name: ubuntu:latest
string: docker.example.com/proxy/ubuntu:latest
- name: guestbook:v0.1
string: docker.example.com/proxy/guestbook:v0.1
Question: What do we do if a CMP announcement doesn't include a collectionType?
Answer: Default to string.
- name: name-prefix # expects a string
- name: helm-parameters-incorrect # expects a string, the map is ignored
map:
global.image.repository: quay.io/argoproj/argocd
- name: helm-parameters # expects a map
collectionType: map
map:
global.image.repository: quay.io/argoproj/argocd
Question: What do we do if a parameter has a missing or absent top-level name field?
Answer: Throw a validation error in the CMP server when handling an announcement. Throw a validation error in the controller and mark the Application as unhealthy if the invalid spec is in the Application. Throw an error in the CMP server and refuse to generate manifests in the CMP server if given invalid parameters.
# needs a `name` field
- title: Parameter Overrides
collectionType: map
map:
global.image.repository: quay.io/argoproj/argocd
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: trivial-cmp
spec:
version: v1.0
generate:
command:
- sh
- -c
- |
# Pull one parameter value from the "main" section of the given parameters.
CM_NAME_SUFFIX=$(echo "$ARGOCD_APP_PARAMETERS" | jq -r '.["main"][] | select(.name == "cm-name-suffix").value')
cat << EOM
{
"kind": "ConfigMap",
"apiVersion": "v1",
"metadata": {
"name": "$ARGOCD_APP_NAME-$CM_NAME_SUFFIX",
"namespace": "$ARGOCD_APP_NAMESPACE"
}
}
EOM
discover:
fileName: "./trivial-cmp"
parameters:
command:
- sh
- -c
- |
echo '[{"name": "cm-name-suffix"}]'
Plugin config
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: kustomize-helm-proxy-cmp
spec:
version: v1.0
generate:
command: [/home/argocd/generate.sh]
discover:
fileName: "./kustomization.yaml"
parameters:
static:
- name: version
title: VERSION
string: v4.3.0
- name: name-prefix
title: NAME PREFIX
- name: name-suffix
title: NAME SUFFIX
dynamic:
command: [/home/argocd/get-parameters.sh]
generate.sh
This script would be non-trivial. Kustomize only accepts YAML-formatted values for Helm charts. The script would have to convert the dot-notated parameters to a YAML file.
get-parameters.sh
kustomize build . --enable-helm > /dev/null
get_parameters() {
while read -r chart; do
yq e -o=p "charts/$chart/values.yaml" | jq --arg chart "$chart" --slurp --raw-input '
{
name: "\($chart)-helm-parameters",
title: "\($chart) Helm parameters",
tooltip: "Parameter overrides for the \($chart) Helm chart.",
collectionType: "map",
map: split("\\n") | map(capture("(?<key>.*) = (?<value>.*)")) | from_entries
}'
done << EOF
$(yq e '.helmCharts[].name' kustomization.yaml)
EOF
}
# Collect the parameters generated for each chart into one array.
get_parameters | jq --slurp
Dockerfile
FROM ubuntu:20.04
RUN apt install jq yq helm kustomize -y
ADD get-parameters.sh /home/argocd/get-parameters.sh
This example demonstrates how the Helm parameters interface could be achieved with a parameterized CMP.
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: simple-helm-cmp
spec:
version: v1.0
generate:
command: [/home/argocd/generate.sh]
discover:
fileName: "./values.yaml"
parameters:
static:
- name: values-files
title: VALUES FILES
collectionType: array
dynamic:
command: [/home/argocd/get-parameters.sh]
generate.sh
# Convert the values-files parameter value to a newline-delimited list of Helm CLI arguments.
ARGUMENTS=$(echo "$ARGOCD_APP_PARAMETERS" | jq -r '.[] | select(.name == "values-files").array | .[] | "--values=" + .')
# Convert JSON parameters to comma-delimited k=v pairs.
PARAMETERS=$(echo "$ARGOCD_APP_PARAMETERS" | jq -r '.[] | select(.name == "helm-parameters").map | to_entries | map("\(.key)=\(.value)") | .[] | "--set=" + .')
# Add parameters to the arguments variable.
ARGUMENTS="$ARGUMENTS\n$PARAMETERS"
echo "$ARGUMENTS" | xargs helm template .
The manifest generation command will be
helm template . --values=a.yaml --values=b.yaml --set=image.repo=alpine --set=image.tag=latest
for the following value of $ARGOCD_APP_PARAMETERS:
[
{
"name": "values-files",
"array": ["a.yaml", "b.yaml"]
},
{
"name": "helm-parameters",
"map": {
"image.repo": "alpine",
"image.tag": "latest"
}
}
]
get-parameters.sh
yq e -o=p values.yaml | jq --slurp --raw-input '
[{
name: "helm-parameters",
title: "Helm Parameters",
collectionType: "map",
map: split("\\n") | map(capture("(?<key>.*) = (?<value>.*)")) | from_entries
}]'
Consider a very simple values.yaml:
image:
repo: quay.io/argoproj/argocd
tag: latest
The script above will produce the following parameters announcement:
[
{
"name": "helm-parameters",
"title": "Helm Parameters",
"collectionType": "map",
"map": {
"image.repo": "quay.io/argoproj/argocd",
"image.tag": "latest"
}
}
]
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: kustomize
spec:
parameters:
static:
- name: version
title: VERSION
string: v4.3.0
- name: name-prefix
title: NAME PREFIX
- name: name-suffix
title: NAME SUFFIX
dynamic:
command: ["generate-params.sh"]
parameters.dynamic.command will produce something like this:
[
{
"name": "images",
"title": "Image Overrides",
"collectionType": "map",
"map": {
"quay.io/argoproj/argocd": "docker.example.com/proxy/argoproj/argocd",
"ubuntu:latest": "docker.example.com/proxy/argoproj/argocd"
}
}
]
Our examples will have shell scripts, and users will write shell scripts. Scripts are difficult to write securely - this is especially true when the scripts are embedded in YAML, and developers don't get helpful warnings from the IDE.
Our docs should emphasize the importance of handling input carefully in any scripts (or other programs) which will be executed as part of CMPs.
The docs should also warn against embedding large scripts in YAML and recommend plugin authors instead build custom images with the script invoked as its own file. The docs should also recommend taking advantage of IDE plugins as well as image and source code scanning tools in CI/CD.
Risk: encouraging CMP adoption while missing critical features from native tools.
Mitigation: rewrite the Helm config management tool as a CMP and test as many common use cases as possible. Write a document before starting on the Helm CMP documenting all major features which must be tested.
Upgrading will only require using a new version of Argo CD and adding the parameters settings to the plugin config.
Downgrading will only require using an older version of Argo CD. The parameters section of the plugin config will
simply be ignored.
Sidecar CMPs aren't really battle-tested. If there are major issues we've missed, then moving more users towards CMPs could involve a lot of growing pains.