proposals/22-03-value-in-the-structured-data.md
Authors:
Reviewers:
Status: implementable
<!-- In general, all proposals made should be merged for the record, whether or not they are accepted. Use the status field to record the results of the latest review: - implementable: The default for this repo. If the proposal is merged, you can start working on it. - deferred: The proposal may be accepted in the future, but it has been shelved for the time being. A new PR must be opened to update the proposal and gain reviewer consensus before work can begin. - withdrawn: The author changed their mind and no longer wants to pursue the proposal. A new PR must be opened to update the proposal and gain reviewer consensus before work can begin. - rejected: This proposal should not be implemented. - replaced: If you submit a new proposal that supersedes an older one, update the older one's status to "replaced by <link>". -->This proposal decides the interfaces to change values in the structured data (like json,yaml) inside a Kubernetes objects' field value and implements changing function target a few formats (mainly json).
kustomize can apply structured edits to Kubernetes objects defined in yaml files.
Sometimes structured multi-line or long single line string (ex. json,yaml, and other structured format data) is injected in Kubernetes objects' string literal field. From the kustomize perspective, these "structured" multiline strings are just arbitrary unstructured strings.
So, kustomize can't manipulate one value on structured, formatted data in the Kubernetes object's string literal field. This function is expected behavior, but kustomize will be very helpful if it can change the value of structured data like json and yaml substrings in a string literal.
This proposal allows the user to identify strings literals containing JSON/YAML data so that Kustomize can make structured edits to the data they contain. This allows the requested functionality to be added without violating Kustomize's core principle of supporting structured edits exclusively.\
For example, kustomize can't change the value "REPLACE_TARGET_HOSTNAME" in this yaml file straightforwardly.
apiVersion: v1
kind: ConfigMap
metadata:
name: target-configmap
data:
config.json: |-
{"config": {
"id": "42",
"hostname": "REPLACE_TARGET_HOSTNAME"
}}
This function has been wanted for a long time.
This function can be an alternative deprecated vars function in many use cases.
Goals:
<!-- List the specific goals of the proposal. What is it trying to achieve? How will we know that this has succeeded? -->Non-goals:
<!-- What is out of scope for this proposal? Listing non-goals helps to focus discussion and make progress. -->Do not provide Unstructured edits, because kustomize eschews parameterization.
Do not provide a way for targeting/merging values within strings from the patches field.
Do not provide a way for customizing field merge strategies for the embedded data.
I propose to extend the fieldPath and fieldPaths fields to replace the value in structured data with the replacements function.
This idea is extending how to select any field in replacement TargetSelector. If the source.fieldPath and targets.fieldPaths had extra values after a specific string literal in Yaml, Kustomize tries to parse that string as structure data and tries to drill down using that additional values.
## replacement
replacements:
- source:
kind: ConfigMap
name: source-configmap
fieldPath: data.HOSTNAME
targets:
- select:
kind: ConfigMap
name: target-configmap
fieldPaths:
- data.config\.json.config.hostname # A path after `config\.json` is pointing one place in the structured data.
Please check Story 1.
This Proposal is to add parameters for configMapGenerator and secretGenerator to allow the merging of two string literals when the behavior option is set to merge and string literals value is structured.
This idea is to add one parameter, mergeValues to GeneratorArgs used from configMapGenerator and secretGenerator.
The mergeValues option is a list that contains two parameters, key and format. One element corresponds to one string literal, key is used to select a string literal to merge, and format is used to designate a format for string literal which is JSON or YAML.
This merge operation will be implemented for a part of Overriding Base ConfigMap Values. It will execute to merge two string literal having same key name when merging two configMap or secret.\
configMapGenerator:
- name: demo-settings
behavior: merge # This function requires `behavior: merge`.
mergeValues:
- key: config.json # Key with a target to merge.
format: json # Setting structured data format MUST be YAML/JSON.
literals:
- config.json: |-
{
"config": {
"hostname": "REPLACE_TARGET_HOSTNAME",
"value": {
"foo": "bar"
}
}
}
Please check Story 2.
Scenario summary: Replacement the value inside for structured data(json) in the configMap.
<!-- A walkthrough of what it will look like for a user to take advantage of the new feature. Include the the steps the user will take and samples of the commands they'll run and config they'll use. -->kustomize patching overlay is very strong to manage common yaml when using many cluster.
But, if you want to set cluster specific change value in the json with configMap data field, you have to replacement whole json file.
apiVersion: v1
kind: ConfigMap
metadata:
name: target-configmap-dev
data:
config.json: |-
{"config": {
"id": "42",
"hostname": "dev.example.com
}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: target-configmap-prod
data:
config.json: |-
{"config": {
"id": "42",
"hostname": "prod.example.com"
}}
So if we can replacement this value in the substring formatted with json, we can easy to overlay this difference.
## source
apiVersion: v1
kind: ConfigMap
metadata:
name: source-configmap
data:
HOSTNAME: www.example.com
---
apiVersion: v1
kind: ConfigMap
metadata:
name: target-configmap
data:
config.json: |-
{"config": {
"id": "42",
"hostname": "REPLACE_TARGET_HOSTNAME"
}}
## replacement
replacements:
- source:
kind: ConfigMap
name: source-configmap
fieldPath: data.HOSTNAME
targets:
- select:
kind: ConfigMap
name: target-configmap
fieldPaths:
- data.config\.json.config.hostname
## expected
apiVersion: v1
kind: ConfigMap
metadata:
name: source-configmap
data:
HOSTNAME: www.example.com
---
apiVersion: v1
kind: ConfigMap
metadata:
name: target-configmap
data:
config.json: '{"config":{"hostname":"www.example.com","id":"42"}}'
Scenario summary: Merge the two values formatted by structured data(json) with configMapGenerator.
<!-- A walkthrough of what it will look like for a user to take advantage of the new feature. Include the the steps the user will take and samples of the commands they'll run and config they'll use. -->Many application needs to be set with json format file. And, That file is handled with configMap when the application is running on kubernetes.
So, If kustomize configMapGenerator can merge for json inside a configMap data value, A json format file will be simple and easy to handle.
# base/kustomization.yaml
configMapGenerator:
- name: demo
literals:
- config.json: |-
{
"config": {
"loglevel": debug,
"parameter": {
"foo": "bar"
}
}
}
# overlay/kustomization.yaml
resources:
- ../base
configMapGenerator:
- name: demo
behavior: merge
mergeValues:
- key: config.json # Key with a target to merge.
format: json # Setting structured data format MUST be YAML/JSON.
literals:
- config.json: |-
{
"config": {
"hostname": "www.example.com",
"parameter": {
"baz": "qux"
}
}
}
apiVersion: v1
data:
config.json: |-
{
"config": {
"loglevel": debug,
"hostname": "www.example.com",
"parameter": {
"foo": "bar",
"baz": "qux"
}
}
}
kind: ConfigMap
metadata:
name: demo-xxxxxxxxxx # name suffix hash
Scenario summary: Replacement the value inside for yaml in the configMap.
<!-- A walkthrough of what it will look like for a user to take advantage of the new feature. Include the the steps the user will take and samples of the commands they'll run and config they'll use. -->A few applications require to use yaml format config file, and some major cloudnative applications are using that.(ex, Prometheus,AlertManager)
A value you want to overlay in yaml is usually into a nested yaml structure. So if you can overlay value inside yaml, you won't need to copy a whole yaml file.
## source
apiVersion: v1
kind: ConfigMap
metadata:
name: environment-config
data:
env: dev
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
data:
prometheus.yml: |-
global:
external_labels:
prometheus_env: TARGET_ENVIROMENT
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
## replacement
replacements:
- source:
kind: ConfigMap
name: environment-config
fieldPath: data.env
targets:
- select:
kind: ConfigMap
name: prometheus-config
fieldPaths:
- data.prometheus\.yml.global.external_labels.prometheus_env
## expected
apiVersion: v1
kind: ConfigMap
metadata:
name: environment-config
data:
env: dev
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
data:
prometheus.yml: |-
global:
external_labels:
prometheus_env: dev
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
Scenario summary: Replacement the value inside for json in the annotations.
<!-- A walkthrough of what it will look like for a user to take advantage of the new feature. Include the the steps the user will take and samples of the commands they'll run and config they'll use. -->A few times, an application on your cluster requires to set json format config for the Annotations on kubernetes yaml resources. We need to overlay in this position value.
## source
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: debug-backend-config
spec:
securityPolicy:
name: "debug-security-policy"
---
apiVersion: v1
kind: Service
metadata:
name: appA-svc
annotations:
cloud-provider/backend-config: '{"ports": {"appA":"gke-default-backend-config"}}'
spec:
ports:
- name: appA
port: 1234
protocol: TCP
targetPort: 8080
## replacement
replacements:
- source:
kind: BackendConfig
name: debug-backend-config
fieldPath: metadata.name
targets:
- select:
kind: ConServicefigMap
name: appA-svc
fieldPaths:
- metadata.annotations.cloud-provider/backend-config.ports.appA
## expected
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: debug-backend-config
spec:
securityPolicy:
name: "debug-security-policy"
---
apiVersion: v1
kind: Service
metadata:
name: appA-svc
annotations:
cloud-provider/backend-config: '{"ports": {"appA":"debug-backend-config"}}'
spec:
ports:
- name: appA
port: 1234
protocol: TCP
targetPort: 8080
kubectl kustomize during alpha? Why or why not?