Integration tests

This document provides information about the integration testing framework used in this project.

Overview

The integration tests are designed to validate the end-to-end functionality of Gemini CLI. They execute the built binary in a controlled environment and verify that it behaves as expected when interacting with the file system.

These tests are located in the integration-tests directory and are run using a custom test runner.

Building the tests

Prior to running any integration tests, you need to create a release bundle that you want to actually test:

bash

npm run bundle

You must re-run this command after making any changes to the CLI source code, but not after making changes to tests.

Running the tests

The integration tests are not run as part of the default npm run test command. They must be run explicitly using the npm run test:integration:all script.

The integration tests can also be run using the following shortcut:

bash

npm run test:e2e

Running a specific set of tests

To run a subset of test files, you can use npm run <integration test command> <file_name1> .... where <integration test command> is either test:e2e or test:integration* and <file_name> is any of the .test.js files in the integration-tests/ directory. For example, the following command runs list_directory.test.js and write_file.test.js:

bash

npm run test:e2e list_directory write_file

Running a single test by name

To run a single test by its name, use the --test-name-pattern flag:

bash

npm run test:e2e -- --test-name-pattern "reads a file"

Regenerating model responses

Some integration tests use faked out model responses, which may need to be regenerated from time to time as the implementations change.

To regenerate these golden files, set the REGENERATE_MODEL_GOLDENS environment variable to "true" when running the tests, for example:

WARNING: If running locally you should review these updated responses for any information about yourself or your system that gemini may have included in these responses.

bash

REGENERATE_MODEL_GOLDENS="true" npm run test:e2e

WARNING: Make sure you run await rig.cleanup() at the end of your test, else the golden files will not be updated.

Deflaking a test

Before adding a new integration test, you should test it at least 5 times with the deflake script or workflow to make sure that it is not flaky.

Deflake script

bash

npm run deflake -- --runs=5 --command="npm run test:e2e -- -- --test-name-pattern '<your-new-test-name>'"

Deflake workflow

bash

gh workflow run deflake.yml --ref <your-branch> -f test_name_pattern="<your-test-name-pattern>"

Running all tests

To run the entire suite of integration tests, use the following command:

bash

npm run test:integration:all

Sandbox matrix

The all command will run tests for no sandboxing, docker and podman. Each individual type can be run using the following commands:

bash

npm run test:integration:sandbox:none

bash

npm run test:integration:sandbox:docker

bash

npm run test:integration:sandbox:podman

Memory regression tests

Memory regression tests are designed to detect heap growth and leaks across key CLI scenarios. They are located in the memory-tests directory.

These tests are distinct from standard integration tests because they measure memory usage and compare it against committed baselines.

Running memory tests

Memory tests are not run as part of the default npm run test or npm run test:e2e commands. They are run nightly in CI but can be run manually:

bash

npm run test:memory

Updating baselines

If you intentionally change behavior that affects memory usage, you may need to update the baselines. Set the UPDATE_MEMORY_BASELINES environment variable to true:

bash

UPDATE_MEMORY_BASELINES=true npm run test:memory

This will run the tests, take median snapshots, and overwrite memory-tests/baselines.json. You should review the changes and commit the updated baseline file.

How it works

The harness (MemoryTestHarness in packages/test-utils):

Forces garbage collection multiple times to reduce noise.
Takes median snapshots to filter spikes.
Compares against baselines with a 10% tolerance.
Can analyze sustained leaks across 3 snapshots using analyzeSnapshots().

Performance regression tests

Performance regression tests are designed to detect wall-clock time, CPU usage, and event loop delay regressions across key CLI scenarios. They are located in the perf-tests directory.

These tests are distinct from standard integration tests because they measure performance metrics and compare it against committed baselines.

Running performance tests

Performance tests are not run as part of the default npm run test or npm run test:e2e commands. They are run nightly in CI but can be run manually:

bash

npm run test:perf

Updating baselines

If you intentionally change behavior that affects performance, you may need to update the baselines. Set the UPDATE_PERF_BASELINES environment variable to true:

bash

UPDATE_PERF_BASELINES=true npm run test:perf

This will run the tests multiple times (with warmup), apply IQR outlier filtering, and overwrite perf-tests/baselines.json. You should review the changes and commit the updated baseline file.

How it works

The harness (PerfTestHarness in packages/test-utils):

Measures wall-clock time using performance.now().
Measures CPU usage using process.cpuUsage().
Monitors event loop delay using perf_hooks.monitorEventLoopDelay().
Applies IQR (Interquartile Range) filtering to remove outlier samples.
Compares against baselines with a 15% tolerance.

Diagnostics

The integration test runner provides several options for diagnostics to help track down test failures.

Keeping test output

You can preserve the temporary files created during a test run for inspection. This is useful for debugging issues with file system operations.

To keep the test output set the KEEP_OUTPUT environment variable to true.

bash

KEEP_OUTPUT=true npm run test:integration:sandbox:none

When output is kept, the test runner will print the path to the unique directory for the test run.

Verbose output

For more detailed debugging, set the VERBOSE environment variable to true.

bash

VERBOSE=true npm run test:integration:sandbox:none

When using VERBOSE=true and KEEP_OUTPUT=true in the same command, the output is streamed to the console and also saved to a log file within the test's temporary directory.

The verbose output is formatted to clearly identify the source of the logs:

--- TEST: <log dir>:<test-name> ---
... output from the gemini command ...
--- END TEST: <log dir>:<test-name> ---

Linting and formatting

To ensure code quality and consistency, the integration test files are linted as part of the main build process. You can also manually run the linter and auto-fixer.

Running the linter

To check for linting errors, run the following command:

bash

npm run lint

You can include the :fix flag in the command to automatically fix any fixable linting errors:

bash

npm run lint:fix

Directory structure

The integration tests create a unique directory for each test run inside the .integration-tests directory. Within this directory, a subdirectory is created for each test file, and within that, a subdirectory is created for each individual test case.

This structure makes it easy to locate the artifacts for a specific test run, file, or case.

.integration-tests/
└── <run-id>/
    └── <test-file-name>.test.js/
        └── <test-case-name>/
            ├── output.log
            └── ...other test artifacts...

Continuous integration

To ensure the integration tests are always run, a GitHub Actions workflow is defined in .github/workflows/chained_e2e.yml. This workflow automatically runs the integrations tests for pull requests against the main branch, or when a pull request is added to a merge queue.

The workflow runs the tests in different sandboxing environments to ensure Gemini CLI is tested across each:

sandbox:none: Runs the tests without any sandboxing.
sandbox:docker: Runs the tests in a Docker container.
sandbox:podman: Runs the tests in a Podman container.