DEVELOPMENT.md
This guide covers everything needed to set up a local Great Expectations development environment: forking and cloning the repository, creating a virtual environment, installing Great Expectations and its extra requirements, configuring backends for testing, running the test suite, and testing performance.
For the contribution journey itself — proposing a change, claiming an issue, submitting a pull request, and what happens during review — see CONTRIBUTING.md. This guide only covers environment setup and test mechanics.
A GitHub account.
A working version of Git on your computer. See Getting Started - Installing Git.
A new SSH (Secure Shell Protocol) key. See Generating a new SSH key and adding it to the ssh-agent.
The latest Python version installed and configured. See Python downloads.
Open a browser and go to the Great Expectations repository.
Click Fork and then Create Fork.
Click Code and then select the HTTPS or SSH tabs.
Copy the URL, open a Git terminal, and then run the following command:
git clone <url>
Run the following command to specify a new remote upstream repository that will be synced with the fork:
git remote add upstream [email protected]:great-expectations/great_expectations.git
Run the following command to create a branch for your changes:
git checkout -b <branch-name>
A virtual environment allows you to install an independent set of Python packages to their own site directory, isolated from the base/system install of Python.
Great Expectations requires a Python version from 3.10 to 3.13.
Run the following command to create a virtual environment named gx_dev:
python3 -m venv <path_to_environments_folder>/gx_dev
Run the following command to activate the virtual environment:
source <path_to_environments_folder>/gx_dev/bin/activate
Run the following command to upgrade pip, the "package installer" for Python:
pip install --upgrade pip
Run the following command to create a virtual environment named gx_dev:
conda create --name gx_dev
Run the following command to activate the virtual environment:
conda activate gx_dev
Run the following command from the root of the great_expectations repository to install Great Expectations in editable mode, with extra requirements for testing:
pip install -c constraints-dev.txt -e ".[test]"
To specify other dependencies, add a comma after test and enter the dependency name(s). For example, ".[test, postgresql, trino]".
The supported extra dependencies include: arrow, athena, aws_secrets, azure, azure_secrets, bigquery, clickhouse, cloud, dremio, excel, gcp, hive, sql-server, mysql, postgresql, redshift, s3, snowflake, spark, teradata, test, trino, vertica.
Check below to see if any of your desired dependencies need system packages installed, before pip install.
Optional. If you're using Amazon Redshift (redshift) or PostgreSQL (postgresql), run one of the following commands to install the pg_config executable, which is required to install the psycopg2-binary Python package:
sudo apt-get install -y libpq-dev
or
brew install postgresql
Optional. If you're using Microsoft SQL Server or Dremio (dremio), run one of the following commands to install unixodbc, which is required to install the pyodbc Python package:
sudo apt-get install -y unixodbc-dev
or
brew install unixodbc
If your Mac computer has an Apple Silicon chip, you might need to
export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/[your version]/lib"
export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/[your version]/include"`
python -m pip install --force-reinstall --no-binary :all: pyodbc
python -c "import pyodbc; print(pyodbc.version)"
install the ODBC 17 driver: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15
see the following resources for more information:
Add ulimit -n 4096 to the ~/.zshrc or ~/.bashrc files to prevent OSError: [Errno 24] Too many open files errors.
Run the following command to confirm pandas and SQLAlchemy with SQLite tests are passing:
ulimit -n 4096
pytest -v
Some Great Expectations features require specific backends for local testing. This section is optional — skip it if your change doesn't need a specific backend.
CD to assets/docker/postgresql in your great_expectations repository, and then and run the following command:
docker-compose up -d
Run the following command to verify the PostgreSQL container is running:
docker-compose ps
The command should return results similar to the following example:
Name Command State Ports
———————————————————————————————————————————
postgresql_travis_db_1 docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
Run the following command to run tests on the PostgreSQL container:
pytest -v --postgresql
When you finish testing, run the following command to shut down the PostgreSQL container:
docker-compose down
Errors similar to the following are returned when you try to start the PostgreSQL container and another service is using port 5432:
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: database "test_ci" does not exist
(Background on this error at: http://sqlalche.me/e/e3q8)
To resolve these errors, configure Docker to run on another port and confirm the server details are correct.
If another service is using port 3306, Docker might start the container but silently fail to set up the port.
CD to assets/docker/mysql in your great_expectations repository, and then and run the following command:
docker-compose up -d
Run the following command to verify the MySQL container is running:
docker-compose ps
The command should return results similar to the following example:
Name Command State Ports
------------------------------------------------------------------------------------------
mysql_mysql_db_1 docker-entrypoint.sh mysqld Up 0.0.0.0:3306->3306/tcp, 33060/tcp
Run the following command to run tests on the MySQL container:
pytest -v --mysql
When you finish testing, run the following command to shut down the MySQL container:
docker-compose down
Use the following information to use Spark for code testing.
Java. See Java downloads.
The PATH or JAVA_HOME environment variables set correctly. See Setting Java variables in Windows or Setting Java variables in Linux.
On Mac, run the following commands to set the PATH and JAVA_HOME environment variables:
export JAVA_HOME=`/usr/libexec/java_home`
export PATH=$PATH:$JAVA_HOME/bin
When you install PySpark, Spark is also installed. See Spark Overview.
Run the following command to install PySpark and Apache Spark:
pip install pyspark
Great Expectations production code must be thoroughly tested, and you must perform unit testing on all branches of every method, including likely error states. Most new feature contributions should include multiple unit tests. Contributions that modify or extend existing features should include a test of the new behavior.
Additionally, all PRs impacting core behavior -- including changes to implementations of Expectations, ValidationDefinitions, Checkpoints, and Datasources -- must have integration test coverage in tests/integration/data_sources_and_expectations.
To perform unit testing, run pytest in the great_expectations directory root. By default, tests are run against pandas and sqlite. You can use pytest flags to test additional backends like postgresql, spark, and sql-server. For example, to run a test against PostgreSQL backend, you run pytest --postgresql.
The following are the supported pytest flags for general testing:
--spark: Execute tests against Spark backend.--postgresql: Execute tests against PostgreSQL.--mysql: Execute tests against MySql.--sql-server: Execute tests against Microsoft SQL Server.--bigquery: Execute tests against Google BigQuery (requires additional set up).--aws: Execute tests against AWS resources such as Amazon S3, Amazon Redshift, and Athena (requires additional setup).To skip all local backend tests (except pandas), run pytest --no-sqlalchemy.
Testing can generate warning messages. These warnings are often caused by dependencies such as pandas or SQLAlchemy. Run pytest --no-sqlalchemy --disable-pytest-warnings to suppress these warnings.
All tests in Great Expectations must include one marker from the REQUIRED_MARKERS list. To view the list of defined markers, see tests/conftest.py.
To verify each test is marked, run invoke marker-coverage if invoke is installed, or run pytest --verify-marker-coverage-and-exit.
When verification fails, a list of unmarked tests and the required markers appears.
In your project, create a BigQuery dataset named test_ci and set the dataset default table expiration to .1 day.
Run the following command to test your project with the GE_TEST_BIGQUERY_PROJECT and GE_TEST_BIGQUERY_DATASET environment variables:
GE_TEST_BIGQUERY_PROJECT=<YOUR_GOOGLE_CLOUD_PROJECT>
GE_TEST_BIGQUERY_DATASET=test_ci
pytest tests/test_definitions/test_expectations_cfe.py --bigquery
Test the performance of code changes to determine they perform as expected. BigQuery is required to complete performance testing.
Run the following command to set up the data for testing:
GE_TEST_BIGQUERY_PEFORMANCE_DATASET=<YOUR_GCP_PROJECT> tests/performance/setup_bigquery_tables_for_performance_test.sh
Run the following command to start the performance test:
pytest tests/performance/test_bigquery_benchmarks.py \
--bigquery --performance-tests \
-k 'test_taxi_trips_benchmark[1-True-V3]' \
--benchmark-json=tests/performance/results/`date "+%H%M"`_${USER}.json \
-rP -vv
Some benchmarks take significant time to complete. In the previous example, only the test_taxi_trips_benchmark[1-True-V3] benchmark runs. The output should appear similar to the following:
--------------------------------------------------- benchmark: 1 tests ------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------
test_taxi_trips_benchmark[1-True-V3] 5.0488 5.0488 5.0488 0.0000 5.0488 0.0000 0;0 0.1981 1 1
-----------------------------------------------------------------------------------------------------------------------------
Run the following command to compare the test results:
$ py.test-benchmark compare --group-by name tests/performance/results/initial_baseline.json tests/performance/results/*${USER}.json
The output should appear similar to the following:
---------------------------------------------------------------------------- benchmark 'test_taxi_trips_benchmark[1-True-V3]': 2 tests ---------------------------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_taxi_trips_benchmark[1-True-V3] (initial_base) 5.0488 (1.0) 5.0488 (1.0) 5.0488 (1.0) 0.0000 (1.0) 5.0488 (1.0) 0.0000 (1.0) 0;0 0.1981 (1.0) 1 1
test_taxi_trips_benchmark[1-True-V3] (2114_work) 6.4675 (1.28) 6.4675 (1.28) 6.4675 (1.28) 0.0000 (1.0) 6.4675 (1.28) 0.0000 (1.0) 0;0 0.1546 (0.78) 1 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Optional. If your change is intended to improve performance, run the following command to generate the benchmark results that confirm the performance improvement:
$ tests/performance/run_benchmark_multiple_times.sh minimal_multithreading
The name for the tests should include the first argument provided to the script. In the previous example, this was tests/performance/results/minimal_multithreading_*.json.
Before submitting a pull request, make sure that your code passes the lint check. Run:
black .
ruff . --fix
You can find more information on developing Great Expectations docs in /docs/docusaurus/README.md. To get a version of the docs deployed locally, run:
invoke docs
The website should then be available at:
open http://localhost:3000/docs
Generates a glossary page in the docs:
python3 ./build_glossary_page.py
Run this command from the scripts directory.