ContextQMD
Back to Nifi

Integration Tests

nifi-extension-bundles/nifi-elasticsearch-bundle/README.md

latest4.6 KB
Original Source
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

Integration Tests

Overview

The integration tests use Testcontainers to provide a sane default for developers who have installed Docker. Testcontainers support can be disabled by setting the system property elasticsearch.testcontainers.enabled to something other than true. If Testcontainers are disabled, the endpoint will need to be configured. It can be set manually with the system property elasticsearch.endpoint. The default value is http://localhost:9200.

Maven Profiles

  • integration-tests
  • elasticsearch6
  • elasticsearch7

Configurable System Properties

  • elasticsearch.endpoint - Manually configure the endpoint root for a non-Docker version of Elasticsearch,
  • elasticsearch.testcontainers.enabled - Set to anything other than true to disable Testcontainers and use a non-Docker version of Elasticsearch.
  • elasticsearch.elastic_user.password - Set the Elasticsearch elastic user's password. When Testcontainers are enabled, this sets up the Docker container and the rest clients for accessing it within the tests. When Testcontainers are disabled, it needs to be set to whatever password is used on the external Elasticsearch node or cluster.

Maven Run Examples

Elasticsearch 9.X is the current default version of Elasticsearch when Testcontainers are used. An example run of the integration tests with Elasticsearch 7 support would be like this:

mvn -Pintegration-tests,elasticsearch7 --fail-at-end clean install

An example using a non-Docker version of Elasticsearch:

mvn -Pintegration-tests --fail-at-end -Delasticsearch.testcontainers.enabled=false -Delasticsearch.elastic_user.password=s3cret1234 clean install

Bash Script Example

Execute the following script from the nifi-elasticsearch-bundle directory:

bash
mvn --fail-at-end -Pcontrib-check clean install

# blank entry to run the default integration-tests profile, i.e. Elasticsearch 8
es_versions=(elasticsearch8 elasticsearch7 " ")
it_modules=(nifi-elasticsearch-client-service nifi-elasticsearch-restapi-processors)
for v in "${es_versions[@]}"; do
    for m in "${it_modules[@]}"; do
        pushd "${m}"
        if ! mvn -P "integration-tests,${v}" --fail-at-end failsafe:integration-test failsafe:verify; then
            echo; echo; echo "Integration Tests failed for ${v} in ${m}, see Maven logs for details"
            exit 1
        fi
        popd
    done
done

Modules with Integration Tests (using Testcontainers)

Running on Mac

Testcontainers support for "Mac OS X - Docker for Mac" is currently best-efforts.

It may be necessary to do the following to run the tests successfully:

  • Link the Docker Unix Socket to the standard location

    sh
    sudo ln -s $HOME/.docker/run/docker.sock /var/run/docker.sock

  • Set the TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE environment variable

    sh
    export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock

Running in IDEs

  • Edit "Run Configurations" to:
    • enable Testcontainers via the system property, i.e. -Delasticsearch.testcontainers.enabled=true
    • set the TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE environment variable if required

Misc

Integration Tests with Testcontainers currently only uses the amd64 Docker Images.

Elasticsearch images (e.g. 7.x and above) appear to work with arm64 machines (e.g. Mac M1 and later). Settings have been altered for the Elasticsearch containers in order to try and enable them on different architectures, but there may still be some inconsistencies.

Explicit arm64 architecture support may be added in future where the Elasticsearch images exist.