docs/static/v0.8/project/contributing/contributing-ui-tests/index.html
To automate functional integration and end-to-end testing Meshery uses Playwright as one of the tools to automate browser testing. End-to-end tests run with each pull request to ensure that the changes do not break the existing functionality.
Before diving into Meshery’s testing environment, certain prerequisites are necessary:
To run the tests successfully, three environment variables must be configured:
• REMOTE_PROVIDER_USER_EMAIL (Required): The email associated with your account within your provider.
• REMOTE_PROVIDER_USER_PASSWORD (Required): The password associated with your account within your provider.
• PROVIDER_TOKEN (Optional): Your provider token, can be generated from an account registered within your provider
In the case you are using Layer5 Cloud as a remote provider, you can generate a token from your user account to use while writing and executing tests.
During the setup phase, Playwright utilizes these environment variables to log in and store credentials securely in the playwright/.auth directory. To protect sensitive data, the .gitignore file is configured to exclude the .env file and any JSON files within the /playwright/.auth directory from the GitHub repository.
There are several tools to help you to working with environment variables locally for each project such as direnv, it can work across multiple shell such as Bash, Powershell, Oh my zsh, Fish, etc
There are a few ways to set up the Meshery UI and server, but for end-to-end testing, we aim to get as close to a production environment as possible. We know developers might need to make some tweaks for UI and Server. Rebuilding the whole project can take time, and we don’t support hot reload because it’s more for development than for end-to-end testing.
Some test cases required you to have kubernetes cluster and build meshery adapter as well, be aware of that. Which is out of scope for this documentation
This approach is very quick to build, but also dependent on your operating system, so you need to have all dependencies necessary to be able compile and running the server.
make ui-build
make build-server
make server-binary
There is also Meshery CLI which can help you run the UI and Server, for more detail, you go to Meshery CLI documentation
Alternatively, a Docker-based setup can be utilized, simplifying the process, and ensuring consistency across different environments. It is closer to the production environment than the native solution but slower in terms of build time.
make docker-testing-env-build
make docker-testing-env
For Playwrights, always try to use a native OS whenever possible. The Docker-based approach is intended only for unsupported OSes and is generally not recommended because it runs on top of Ubuntu images, which can be redundant if you already using Ubuntu or Windows.
Setup playwright:
make ui-test-setup
Run the all project and test cases:
make ui-test
The first step is to pull the docker image from Azure Container Registry where the playwright stores their image using this command:
docker pull mcr.microsoft.com/playwright:<version>-<base-image>
Make sure the version you are using matches the version of @playwright/test in the package.json dev dependencies
Here is the example of pulling playwright v1.44.0 with Ubuntu 22.04 LTS
docker pull mcr.microsoft.com/playwright:v1.44.0-jammy
Starting up playwright docker server:
docker run --rm --network host --init -it mcr.microsoft.com/playwright:v1.44.0-jammy /bin/sh -c "cd /home/pwuser && npx -y [email protected] run-server --port 8080"
Keep in mind this is just for development purposes inside your local system and don’t try to expose your container network to the host system using --network host on production or CI
In the last step go to ui folder,
cd ui;
There are several options we can use to run the test cases, in CLI:
To run playwright UI mode using the browser, you can add --ui in the cli, for example:
npx playwright test --ui
If you are using playwright from docker, you can use --ui-port=<playwright-docker-server>, for example:
npx playwright test --ui-port=8080
To run playwright for specific project only, for example meshery-provider, you can run this command:
npx playwright test --ui --project=chromium-meshery-provider
To run specific test, you can add the test file location, for example:
npx playwright test --ui --project=chromium-meshery-provider tests/e2e/service-mesh-performance.spec.js
For more detail, you can read the Playwright Cli docs
By default our test cases is running against both Meshery and Local Provider, we are utilizing playwright feature such as:
provider directly from the test, and make sure the test is wrapped using:import { expect, test } from './fixtures/project';
test('Random test', async ({ provider }) => {
if (provider === "Meshery") {
// Run this for testing Meshery provider
}
if (provider === "None") {
// Run this for testing Local provider
}
});
After merging a pull request, ensuring test stability across CI/CD runs is crucial. A test may pass locally but fail in the CI/CD environment due to differences in execution conditions. While one approach is to mark all new tests as @unstable by default, a more effective strategy is to apply the @unstable tag only if a test exhibits intermittent failures or flaky behavior.
After merging into the master branch, monitor the GitHub workflow that executes the new test case and assess its stability. If the test fails, raise another PR to mark it as @unstable and communicate this to the team. For example:
import { expect, test } from './fixtures/project';
test('Random test', { tag: '@unstable' }, async ({ provider }) => {
// Test cases here
// ...
});
End-to-end test results are stored as artifacts on every PR in Github Actions. In case you need to debug a failed test:
Meshery UI and Server / UI end-to-end testsplaywright-reportWatch the training session on Playwright testing and trace debugging.
Refer to Meshery Test Plan for test scenarios.
To filter and view only UI-related tests using the Sheet Views feature: