ContextQMD
Back to Spark

Spark Docker Integration Tests

connector/docker-integration-tests/README.md

4.1.14.4 KB
Original Source

Running the Docker integration tests

Note that the integration test framework is using thirdparty Docker images to test functionalities of integration with other data sources, not running Spark itself in Docker containers, which is actually done by the Kubernetes Integration Tests module.

In order to run the Docker integration tests, the Docker engine needs to be installed and started on the machine. Additionally, the environment variable ENABLE_DOCKER_INTEGRATION_TESTS=1 shall be specified to enable the Docker integration tests.

export ENABLE_DOCKER_INTEGRATION_TESTS=1

or,

ENABLE_DOCKER_INTEGRATION_TESTS=1 ./build/sbt -Pdocker-integration-tests "docker-integration-tests/test"

Running an individual Docker integration test

Testing the whole module of Docker Integration Tests might be time-consuming because of image pulling and the container bootstrapping. To run an individual Docker integration test, use the following command:

./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly <test class name>"

Using a custom Docker image

Besides the default Docker images, the integration tests can be run with custom Docker images. For example,

ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:23.9-slim ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"

The following environment variables can be used to specify the custom Docker images for different databases:

  • DB2_DOCKER_IMAGE_NAME
  • MARIADB_DOCKER_IMAGE_NAME
  • MSSQLSERVER_DOCKER_IMAGE_NAME
  • MYSQL_DOCKER_IMAGE_NAME
  • ORACLE_DOCKER_IMAGE_NAME
  • POSTGRES_DOCKER_IMAGE_NAME
  • STARROCKS_DOCKER_IMAGE_NAME

Using a custom Docker context

In certain scenarios, you may want to use a different docker context/endpoint instead of the default provided by Docker Desktop.

docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                 KUBERNETES ENDPOINT   ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux       moby                Docker Desktop                            unix:///Users/.../docker.sock

docker context use desktop-linux

Then you can run the integration tests as usual targeting the Docker endpoint named desktop-linux.

This is useful when the upstream Docker image can not starton the Docker Desktop. For example, when the image you use only supports x86_64 architecture, but you are running on an Apple Silicon aarch64 machine, for which case you may want to use a custom context that is able to mock a x86_64 architecture, such as colima. After colima installed, you can start a runtime with x86_64 support and run the integration tests as follows:

colima start --arch x86_64 --memory 8 --network-address
docker context colima
./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"

Available Properties

The following are the available properties that can be passed to optimize testing experience.

./build/sbt -Pdocker-integration-tests \
            -Dspark.test.docker.keepContainer=true \
            "testOnly *MariaDBKrbIntegrationSuite"
<table> <tr> <th>Property</th> <th>Description</th> <th>Default</th> </tr> <tr> <td><code>spark.test.docker.keepContainer</code></td> <td> When true, the Docker container used for the test will keep running after all tests finished in a single test file. </td> <td>false</td> </tr> <tr> <td><code>spark.test.docker.removePulledImage</code></td> <td> When true, the Docker image used for the test will not be preserved after all tests finished in a single test file. </td> <td>true</td> </tr> <tr> <td><code>spark.test.docker.imagePullTimeout</code></td> <td> Timeout for pulling the Docker image before the tests start. </td> <td>5min</td> </tr> <tr> <td><code>spark.test.docker.startContainerTimeout</code></td> <td> Timeout for container to spin up. </td> <td>5min</td> </tr> <tr> <td><code>spark.test.docker.connectionTimeout</code></td> <td> Timeout for connecting the inner service in the container, such as JDBC services. </td> <td>5min(might get overridden by some inherits)</td> </tr> </table>