designs/extensible-cli-and-scaffolding-plugins-phase-2.md
| Authors | Creation Date | Status | Extra |
|---|---|---|---|
| @rashmigottipati | Mar 9, 2021 | partial implemented | Plugins doc |
Plugin Phase 1.5 was designed and implemented to allow chaining of plugins. The purpose of Phase 2 plugins is to discover and use external plugins, also referred to as out-of-tree plugins (which can be implemented in any language). Phase 2 achieves both chaining and discovery of external plugins/source code not compiled with the kubebuilder CLI binary. By achieving this goal, we could (for example) externalize the optional declarative plugin which means that the CLI would still be able to use it, however, its source code would no longer be required to be inside of the Kubebuilder repository.
POC - Invoke an external python program that simulates a plugin from a go main and pass messages from kubebuilder to the plugin and vice-versa using stdin/stdout/stderr.
As a plugin developer, I would like to be able to provide external plugins path for the CLI to perform the scaffolds, so that I could take advantage of external initiatives which are implemented using Kubebuilder as a lib and following its standards but are not shipped with its CLI binaries.
As a Kubebuilder maintainer, I would like to support external plugins not maintained by the core project.
kubebuilder binary.kubebuilder is able to discover plugin binaries and run those plugins using the CLI.
Kubebuilder can use the external plugins as well as its own internal ones to do scaffolding.
kubebuilder should be able to show plugin specific information via the --help flag.
Support for standard streams i.e. stdin/stdout/stderr as the only IPC method between kubebuilder and plugins.
Kubebuilder library consumers can support chaining and discovery of out-of-tree plugins.
Addition of new arbitrary subcommands other than the subcommands that we already support i.e init, create api, and create webhook.
Discovering plugin binaries that are not locally present on the machine (i.e. binary exists in a remote repository).
Providing other options (other than standard streams such as stdin/stdout/stderr) for inter-process communication between kubebuilder and external plugins.
kubebuilder create api --plugins=myexternalplugin/v1
create api method.kubebuilder create api --plugins=myexternalplugin/v1,myotherexternalplugin/v2
create api method (by respecting the plugin chaining order, i.e. in the order of create api of v1 and then create api of v2 as specified in the layout field in the configuration).kubebuilder create api --plugins=myexternalplugin/v1 --help
kubebuilder binary).kubebuilder create api --plugins=go/v3,myexternalplugin/v2
go/v3 plugin, then pass those files to myexternalplugin/v2 as defined in its implementation of the create api method by respecting the plugin chaining order.The method kustomize uses to discover plugins, by following a GVK path scheme, is the most natural for this use case since plugins must have a group-like name and version.
Every plugin gets its own directory constructed using the plugin name and plugin version for the executable to be placed in and kubebuilder will search for a plugin binary with the name of the plugin in the ${name}/${version} directory of the plugin. This information (plugin name and plugin version) is obtained by kubebuilder via the value passed to the --plugins CLI flag. Once kubebuilder successfully locates the plugin, it will run the plugin using the CLI.
Every plugin gets its own directory as below.
On Linux:
$XDG_CONFIG_HOME/kubebuilder/plugins/${name}/${version}
The default value of XDG_CONFIG_HOME is $HOME/.config.
On OSX:
~/Library/Application Support/kubebuilder/plugins/${name}/${version}
Based on the above directory scheme, let's say that if the value passed to the --plugins CLI flag is myexternalplugin/v1:
kubebuilder will search for the myexternalplugin binary in $XDG_CONFIG_HOME/kubebuilder/plugins/myexternalplugin/v1, where the base of this path in is the binary name.myexternalplugin binary in $HOME/Library/Application Support/kubebuilder/plugins/myexternalplugin/v1.Note: If the name is ambiguous, then the qualified name myexternalplugin.my.domain would be used, so the path would be $XDG_CONFIG_HOME/kubebuilder/plugins/my/domain/myexternalplugin/v1 on Linux and $HOME/Library/Application Support/kubebuilder/plugins/my/domain/myexternalplugin/v1 on OSX.
kustomize which is popular and robust tool, follows this approach in which apiVersion and kind fields are used to locate the plugin.
This approach enforces naming constraints as the permitted character set must be directory name-compatible following naming rules for both Linux and OSX systems.
The one-plugin-per-directory requirement eases creation of a plugin bundle for sharing.
I propose we use our own plugin system that passes JSON blobs back and forth across stdin/stdout/stderr and make this the only option for now as it's a language-agnostic medium and it is easy to work with in most languages.
We came to the conclusion that a kubebuilder-specific plugin library should be written after evaluating plugin libraries such as the built-in go-plugin library and Hashicorp's plugin library:
In the future, if a need arises (for example, users are hitting performance issues), we can then explore the possibility of using the Hashicorp's go plugin library. From a design standpoint, to leave it architecturally open, I propose using a type field in the PROJECT file to potentially allow other plugin libraries in the future and make this a separate field in the PROJECT file per plugin; and this field determines how the universe will be passed for a given plugin. However, for the sake of simplicity in initial design and not to introduce any breaking changes as Project version 3 would suffice for our needs, this option is out of scope in this proposal.
Currently, the project configuration has two fields to store plugin specific information.
Layout field (of type []string) is used for plugin chain resolution on initialized projects. This will be the default if no plugins are specified for a subcommand.
Plugins field (of type map[string]interface{}) is used for option plugin configuration that stores configuration information of any plugin.
So, where should external plugins be defined in the configuration?
layout field.
myexternalplugin/v2 can be specified through the --plugins flag for every subcommand and also be defined in the project configuration in the layout field for plugin resolution.Example PROJECT file:
version: "3"
domain: testproject.org
layout:
- go.kubebuilder.io/v3
- myexternalplugin/v2
plugins:
myexternalplugin/v2:
resources:
- domain: testproject.org
group: crew
kind: Captain
version: v2
declarative.go.kubebuilder.io/v1:
resources:
- domain: testproject.org
group: crew
kind: FirstMate
version: v1
repo: github.com/test-inc/testproject
resources:
- group: crew
kind: Captain
version: v1
kubebuilder and external pluginsWhy do we need communication between kubebuilder and external plugins?
kubebuilder and the external plugin as they are two separate processes/binaries. kubebuilder needs to communicate the subcommand that the external plugin should run, and all the arguments received in the CLI request by the user. These arguments contain flags which will have to be directly passed to all plugins in the chain. Additionally, it's important to have context of all the files that were scaffolded until that point especially if there is more than one external plugin in the chain. kubebuilder attaches that information in the request, along with the command and arguments. For the external plugin, it would need to communicate the subcommand it ran and the updated file contents information that the external plugin scaffolded to kubebuilder. The external plugin would also need to provide its help text if requested by kubebuilder. As discussed earlier, standard streams seems to be a desirable IPC method of communication for the use-cases that Phase 2 is trying to solve that involves discovery and chaining of external plugins.How does kubebuilder communicate to external plugins?
stdin), standard output (stdout) and standard error (stderr) and they work well with chaining applications, meaning that output stream of one program can be redirected to the input stream of another.kubebuilder communicates to the plugins myfirstexternalplugin/v1 and mysecondexternalplugin/v1.kubebuilder and an external plugin?Message passing between kubebuilder and the external plugin will occur through a request / response mechanism. The PluginRequest will contain information that kubebuilder sends to the external plugin. The PluginResponse will contain information that kubebuilder receives from the external plugin.
The following scenarios show what kubebuilder will send/receive to the external plugin:
kubebuilder to external plugin:
kubebuilder constructs a PluginRequest that contains the Command (such as init, create api, or create webhook), Args containing all the raw flags from the CLI request and license boilerplate without comment delimiters, and an empty Universe that contains the current virtual state of file contents that are not written to disk yet. kubebuilder writes the PluginRequest through stdin.External plugin to kubebuilder:
PluginRequest through its stdin and processes the request based on the Command that was sent. If the Command doesn't match what the plugin supports, it writes back an error immediately without any further processing. If the Command matches what the plugin supports, it constructs a PluginResponse containing the Command that was executed by the plugin, and modified Universe based on the new files that were scaffolded by the external plugin, Error and ErrorMsg that add any error information, and writes the PluginResponse back to kubebuilder through stdout.Note: If --help flag is being passed from kubebuilder to the external plugin through PluginRequest, the plugin attaches its help text information in the Metadata field of the PluginResponse. Both PluginRequest and PluginResponse also contain APIVersion field to have compatible versioned schemas.
Handling plugin failures across the chain:
PluginResponse to kubebuilder and plugin chain execution will be halted, as one plugin may be dependent on the success of another. All the files that were scaffolded already until that point will not be written to disk to prevent a half committed state.PluginRequest holds all the information kubebuilder receives from the CLI and the plugins that were executed before it and the PluginRequest will be marshaled into a JSON and sent over stdin to the external plugin. PluginResponse is what the plugin constructs with the updated universe and sent back to kubebuilder. The following structs would be defined on the Kubebuilder side.
// PluginRequest contains all information kubebuilder received from the CLI
// and plugins executed before it.
type PluginRequest struct {
// Command contains the command to be executed by the plugin such as init, create api, etc.
Command string `json:"command"`
// APIVersion defines the versioned schema of the PluginRequest that is encoded and sent from Kubebuilder to plugin.
// Initially, this will be marked as alpha (v1alpha1).
APIVersion string `json:"apiVersion"`
// Args holds the plugin specific arguments that are received from the CLI which are to be passed down to the plugin.
Args []string `json:"args"`
// Universe represents the modified file contents that gets updated over a series of plugin runs
// across the plugin chain. Initially, it starts out as empty.
Universe map[string]string `json:"universe"`
}
// PluginResponse is returned to kubebuilder by the plugin and contains all files
// written by the plugin following a certain command.
type PluginResponse struct {
// Command holds the command that gets executed by the plugin such as init, create api, etc.
Command string `json:"command"`
// Metadata contains the plugin specific help text that the plugin returns to Kubebuilder when it receives
// `--help` flag from Kubebuilder.
Metadata plugin.SubcommandMetadata `json:"metadata"`
// APIVersion defines the versioned schema of the PluginResponse that will be written back to kubebuilder.
// Initially, this will be marked as alpha (v1alpha1).
APIVersion string `json:"apiVersion"`
// Universe in the PluginResponse represents the updated file contents that was written by the plugin.
Universe map[string]string `json:"universe"`
// Error is a boolean type that indicates whether there were any errors due to plugin failures.
Error bool `json:"error,omitempty"`
// ErrorMsg holds the specific error message of plugin failures.
ErrorMsg string `json:"error_msg,omitempty"`
}
The following function handles construction of the PluginRequest based on the information kubebuilder receives from the CLI and the request is marshaled into JSON. The command to run the external plugin by providing the plugin path will be invoked and kubebuilder will send the marshaled PluginRequest JSON to the plugin over stdin.
func (p *ExternalPlugin) runExternalProgram(req PluginRequest) (res PluginResponse, err error) {
pluginReq, err := json.Marshal(req)
if err != nil {
return res, err
}
cmd := exec.Command(p.Path)
cmd.Dir = p.DirContext
cmd.Stdin = bytes.NewBuffer(pluginReq)
cmd.Stderr = os.Stderr
out, err := cmd.Output()
if err != nil {
fmt.Fprint(os.Stdout, string(out))
return res, err
}
if json.Unmarshal(out, &res); err != nil {
return res, err
}
return res, nil
}
On the plugin side, the request JSON will be decoded and depending on what the Command in the PluginRequest is, the corresponding function to handle init or create api will be invoked thereby modifying the universe by writing the updated files to it. After init or create api functions execute successfully, the plugin will write back PluginResponse with updated universe and errors (if any) in JSON format through stdout to kubebuilder. PluginResponse also contains error fields Error and ErrorMsg that the plugin can utilize to add error context if any errors occur.
kubebuilder receives the command output and decodes into PluginResponse struct. This is how message passing will occur between kubebuilder and the external plugin. Refer to POC for specifics.
kubebuilder init --plugins=myexternalplugin/v1 --domain example.com
What happens when the above is invoked?
kubebuilder discovers myexternalplugin/v1 plugin binary and runs the plugin from the discovered path.
Send PluginRequest as a JSON over stdin to myexternalplugin plugin.
PluginRequest JSON:
{
"command":"init",
"args":["--domain","example.com"],
"universe":{}
}
myexternalplugin plugin parses the PluginRequest and based on the Command specified in the request i.e init, performs the necessary scaffolding.
myexternalplugin plugin constructs PluginResponse with modified Universe that contains the updated file contents and errors if any.
Plugin writes PluginResponse to stdout in a JSON format back to kubebuilder.
kubebuilder receives the command output containing the PluginResponse JSON which will be decoded into the PluginResponse struct.
kubebuilder writes the files in the universe to disk.
PluginResponse JSON:
{
"command": "init",
"universe": {
"LICENSE": "Apache 2.0 License\n",
"main.py": "..."
}
}
A user will provide a list of file paths for kubebuilder to discover the plugins in. We will define a variable KUBEBUILDER_PLUGINS_DIRS that will take a list of file paths to search for the plugin name. It will also have a default value to search in, in case no file paths are provided. It will search for the plugin name that was provided to the --plugins flag in the CLI. kubebuilder will recursively search for all file paths until the plugin name is found and returns the successful match, and if it doesn't exist, it returns an error message that the plugin is not found in the provided file paths. Also use the host system mechanism for PATH separation.
Alternatively, this could be handled in a way that helm kustomize plugin discovers the plugin based on the non-existence of a separator in the path provided, in which case kubebuilder will search in $PATH, otherwise resolve any relative paths to a fully qualified path.
Pros
This provides flexibility for the user to specify the file paths that the plugin would be placed in and kubebuilder could discover the binaries in those user specified file paths.
No constraints on plugin binary naming or directory placements from the Kubebuilder side.
Provides a default value for the plugin directory in case user wants to use that to drop their plugins.
Another approach is adding plugin executables with a prefix kubebuilder- followed by the plugin name to the PATH variable. This will enable kubebuilder to traverse through the PATH looking for the plugin executables starting with the prefix kubebuilder- and matching by the plugin name that was provided in the CLI. Furthermore, a check should be added to verify that the match is an executable or not and return an error if it's not an executable. This approach provides a lot of flexibility in terms of plugin discovery as all the user needs to do is to add the plugin executable to the PATH and kubebuilder will discover it.
Pros
kubectl and git follow the same approach for discovering plugins, so there's prior art.
There's a lot of flexibility in just dropping plugin binaries to PATH variable and enabling the discovery without having to enforce any other constraints on the placements of the plugins.
Cons
Enumerating the list of all available plugins might be a bit tough compared to having a single folder with the list of available plugins and having to enumerate those.
These plugin binaries cannot be run in a standalone manner outside of Kubebuilder, so may not be very ideal to add them to the PATH var.
Do we want to support the addition of new arbitrary subcommands other than the subcommands (init, create api, create webhook) that we already support?
Do we need to discover flags by calling the plugin binary or should we have users define them in the project configuration?
PluginResponse.What alternatives to stdin/stdout exist and why shouldn't we use them?
What happens when two plugins bind the same flag name? Will there be any conflicts?
How should we handle environment variables?
Should the API version be a part of the plugin request spec?
PluginRequest and PluginResponse so the initial schemas can be marked as v1alpha1.