rfcs/2022-02-24-11552-dd-agent-style-secret-management.md
The Datadog Agent has a straightforward secret resolution facility to avoid having sensitive information stored directly in its config, it relies on a user-provided external program that is run to retrieve sensitive value from a third party system. This RFC aims to propose a similar mechanism for Vector.
The Datadog Agent documentation provide all user-relevant information to use that feature. It covers the specification for the user-provided executable that loads/decrypts secrets along with the agent configuration and the syntax to retrieve encrypted config value.
vector config (PR)and related Vector enterprise work.SECRET[<backend>.<secret_key>] to indicate Vector that a secret should
be retrieved for this option (the Datadog Agent uses ENC[secret_key], ENC stands for encrypted, but that can be
confusing, hence the current suggestion to use SECRET[<backend>.<secret_key>] instead).Datadog Secret API, as per the official doc:
Vector would run the user-provided executable and feed its standard input with the list of secrets to retrieve:
{"version": "1.0", "secrets": ["secret1", "secret2"]}
The version used by the Datadog agent is 1.0. The user provided executable should then reply on its standard output with a JSON formatted string:
{
"secret1": {"value": "secret_value", "error": null},
"secret2": {"value": null, "error": "could not fetch the secret"}
}
Note:
SECRET[<backend>.<secret_key>], dots are not allowed in <backend> but can be used in
<secret_key>, this means that SECRET[mul.ti.ple.dots] will cause Vector to query the backend named mul (it shall
be defined) for the secret key ti.ple.dotsNew top level options to be added will be sitting inside the secret namespace, this would lead to something like:
[secret.local_exec]
type = "exec"
path = "/path/to/the/command"
argument = "--config foo=bar"
timeout = 5
[secret.prod_vault]
type = "vault"
address = "https://vault.corp.tld/"
token = "SECRET[local_exec.vault_token]"
timeout = 5
[sources.system_logs]
type = "file"
includes = ["/var/log/system.log"]
[sinks.app_logs]
type = "datadog_logs"
default_api_key = "SECRET[prod_vault.dd_api_2022_02]"
inputs = ["system_logs"]
The first implementation would only support the exec backend but with extensibility point clearly identified to easily
implement additional backend if needed.
Overall the behaviour for corner cases should follow what's in place for environment variable interpolation as this is a very close feature. When a configuration reload happens, secrets shall also be refreshed.
A secret backend that would lie in ./src/config/secret.rs and would call the user provided executable for secrets,
cache secrets to avoid calling the backend for the same key multiple time. It would read the configuration file once to
get its config before further processing, ideally env var interpolating should be supported (this should not be a
problem). In ./src/config/builder.rs, load_builder_from_paths will still be returning a complete configuration with
placeholders replaced by secrets.
The ConfigBuilder struct will get a new secret field (typed to something like
IndexMap<ComponentKey,SecretBackend>), this means that load_builder_from_paths will then assert if this field is not
empty before returning to the caller, and if it, the config will be reloaded with this SecretBackend passed to
downstream callee and hook the secret interpolation around the same point as the environment variable
interpolation:
SECRET[<backend>.<key>] placeholders present in config will be collected.Note: As shown in the aforementioned config sample a secret backend shall itself be able to use other secret backend for its own initialisation.
The implementation should ease future extension and split the internal API queried by the interpolation logic and the
secret provider that may see other implementation like: exec (the one we will focus on first), vault,
k8s-config-map, aws-secretsmanager, etc.
ENC[secret_key], and another possible URI-like solution was mentioned (secret://<backend>/<key>).Note: doing nothing is not really an alternative here, as plain text secret in config is a strong blocker for some users.