docs/static/v0.8/project/contributing/contributing-cli-tests/index.html
Meshery CLI is the command line interface for Meshery. Meshery CLI, otherwise known as mesheryctl, is a client of Meshery Server’s REST API. It provides a way to interact with Meshery and perform various operations such as installing, configuring, and managing cloud native infrastructure.
This document is intended to help you contribute to the end-to-end tests for mesheryctl, the Meshery CLI. It is designed to be a guide for developers who are new to the project and want to contribute to the testing of mesheryctl.
The end-to-end tests for mesheryctl are designed to ensure that the CLI is working as expected and that it is compatible with the various cloud native infrastructure and public cloud services that Meshery supports. These tests are run automatically on every pull request to ensure that any changes made to the code do not break the existing functionality of the CLI.
mesheryctl commands.End-to-end testing of mesheryctl uses the Bash Automated Testing System (BATS) framework to define and execute CLI tests. Each test case is designed to mimic the experience that a Meshery CLI user might have while interacting with mesheryctl in their terminal of choice. In this sense, mesheryctl tests run end-to-end with each pull request submitted containing changes to either the /mesheryctl or the /server directories in the meshery/meshery repository, ensuring that changes included in those pull requests do not break the existing CLI functionality.
Before diving into mesheryctl’s testing environment, certain prerequisites must be met to ensure a smooth testing experience. These prerequisites include:
A working installation of Meshery CLI and Meshery Server.
A working installation of the BATS testing framework.
bash as shell terminal.[Optional] A working installation of a Kubernetes cluster (Minikube, Kind, etc.) for testing Kubernetes-related functionality.
jq and yq, tools for processing JSON and YAML inputs, respectively.
For Bats Core, always try to use a BATS-native OS whenever possible. This is because BATS Core does not support Windows. If you are using Windows, you can use WSL (Windows Subsystem for Linux) to run BATS Core. See the official BATS installation documentation for more information on how to install BATS Core on your system. Here are quick start steps.
brew install bats-core
npm install -g bats
Check out a copy of the Bats repository and install it to $HOME. This will place the bats executable in $HOME/bin, which will need to be added in $PATH.
git clone https://github.com/bats-core/bats-core.git
cd bats-core
./install.sh $HOME
Some tests could use bats libraires as helpers to create the tests suite.
There are needed dependencies to test whether the server is up and running. Resolve them by navigating to meshery/mesheryctl directory and then executing:
make e2e-libs
There are a few ways to set up the Meshery server, but for end-to-end testing, we aim to get as close to a production environment as possible. We know developers might need to make some tweaks for Server. Rebuilding the whole project can take time, and we don’t support hot reload because it’s more for development than for end-to-end testing.
make server
Be aware that some test cases require the availability of a Kubernetes cluster and one or more Meshery Adapters. In those cases, please refer to the installation guides) (like that of installing Meshery on Minikube).
To run the tests successfully, you need be logged in to your Meshery account. This is required to access the Meshery Server and perform operations on it. Whether using the local or a remote provider, you will need to generate a token from your user account to use while writing and executing tests.
Remote Provider Example
If you are using Layer5 Cloud as your remote provider, you can generate and download an API token from your user account for use while writing and executing tests.
mesheryctl system check
If you see this error message - Error: !! Authentication token not found. Please supply a valid user token. Login with mesheryctl system login, you will need to authenticate using the command:
mesheryctl system login
Once all prerequisites have been met, you are setup to run, add, update new test cases.To keep the development of tests consistents and maintainable, we have put some guidance on implementation process which are describe above.
The tests will be available inmesheryctl/tests/e2e folder with the following structure which is done to follow the code base structure of mesheryctl.
├── aaa-<command>
├── bbb-<command>
├── ...
├── yyy-<command>
├── zzz-<command>
├── helpers
├── README.md
├── run_tests.bash
├── setup_suite.bash
└── teardown_suite.bash
Bats will run the tests sequentially so we use a prefix code alphanumeric to be able to order tests as required when necessary. The prefix code is used to order the tests in a way that makes sense for the command under test. The prefix code is used to group the tests by command and subcommand. The prefix code is also used to order the tests within the command and subcommand. For example, you need to import a Meshery Model before being able to view the details of that model
Test Folder
Example: 011-design
Test File
Example: 05-list.bats
For consistency, we will keep the prefix 00- for the command under test in the folder and subcommands will start at 01-. Here an example with mesheryctl model command:
002-model/
├── 00-model.bats
└── 01-model-list.bats
Make sure you are in meshery/mesheryctl directory
Run all tests
# run tests building mesheryctl binary
make e2e
# run tests without buiding mesheryctl binary
make e2e-no-build
Run a specific commmand tests suite
make e2e-no-build BATS_FOLDER_PATTERN=<test folder name>
# Example to run mesheryctl model subcommand tests
make e2e-no-build BATS_FOLDER_PATTERN=002-model
Run a specific test file
make e2e-no-build BATS_FOLDER_PATTERN=<test folder name> BATS_FILE_PATTERN=<test command name>
# Example to run mesheryctl model genereate tests
make e2e-no-build BATS_FOLDER_PATTERN=002-model BATS_FILE_PATTERN=06-model-generate
More on running tests locally
Breaking down the execution of make e2e, two commands are executed before subsequent commands are run;
make: which builds the binary ande2e-libs: which gets needed dependencies for running the tests.These steps can become redundant if you’re not running the end-to-end tests for the first time.
It is important to point out that there are other ways to run end-to-end tests locally. To use these, ensure you are in the meshery/mesheryctl/tests/e2e directory.
Run tests with already built binary
This excludes the need to build the binary everytime there is an attempt to run all the tests.
bash run_tests_local.sh
NB: This works if there is an existing mesheryctl binary. If there isn’t, the binary will be built.
Enforce rebuilding the binary
This involves parsing a flag for the binary to be built whether it exists or not. This comes in handy when you have local changes and possibly will like to test.
bash run_tests_local.sh -b
Refer to Meshery Test Plan for test scenarios.
To filter and view only CLI-related test cases using the Sheet Views feature:
End-to-End Tests:
mesheryctl CLI) down to the underlying systems (Meshery server, Kubernetes, etc.).For mesheryctl, E2E tests will focus on verifying that CLI commands perform their intended actions against a running Meshery instance in context of how users experience the CLI , this means we are focusing on UX. This might involve deploying applications, managing connections, interacting with Meshery features, and verifying the expected outcomes.
We will exclusively use the Bats Core framework and its built-in functionalities for writing E2E tests. This ensures consistency and leverages a well-established testing tool for shell scripts.
Key Principles:
mesheryctl: The tests should primarily interact with the mesheryctl CLI.assert, assert_success, assert_failure, assert_output, etc.) to verify expected outcomes.setup() and teardown() functions to prepare the testing environment and clean up afterwards..bash file within the helpers folder. Each helper script/function should have a clear description of its purpose within the file itself. Avoid inline custom scripting within the test files.It must follow this naming convention
<mesheryctl command> [subcommand] <execution context> <expected result>
Example:
@test "mesheryctl model view without model name should display an error" {
... test implementation ...
}
If a command requries a specific id, name or any predefined value ensure that the data is created by your test or another test beforehand. Do not rely on external or uncontrolled data as it will lead to unexpected results.
Example:
In the following example, we must have create a model with the name model-test before creating or running the following test
@test "mesheryctl model view providing a model name should display model information" {
run MESHERYCTL_BIN model view model-test
... ...
}
Official documentation is available at https://bats-core.readthedocs.io/en/stable/
Github organization https://github.com/bats-core contains bats-core repository and also bats libraries repositoires
Basic Test Structure: A Bats test file consists of one or more test cases defined using the @test keyword.
Interacting with mesheryctl: Execute mesheryctl commands within your test cases using the run command. Capture the output and exit status for assertions.
Assertions: Use Bats Core’s assertion functions to validate the results of your mesheryctl commands. Refer to the Bats Core documentation for a complete list of assertions.
Setup and Teardown:
Test Data: If your tests require specific input files (e.g., Kubernetes manifests), store them in a relevant directory (e.g., tests/e2e/fixtures).
Error Handling: Consider how your tests will handle errors. Use appropriate assertions to check for expected failures and provide informative error messages.
While implementing new E2E tests, you might discover bugs in mesheryctl. It is crucial to report these findings properly:
Create a New Issue: Navigate to the meshery/meshery repository on GitHub and click “New issue.”
Use the mesheryctl Bug Report Template: Look for and select the “mesheryctl Bug Report” issue template. This template provides a structured format for reporting bugs related to the CLI.
Provide Detailed Information: Fill out the sections of the bug report template with as much detail as possible, including:
Link to the Test: If the bug was discovered while writing a specific test, mention the test file and test case in the issue.