Back to Datahub

DataHub MetadataAuditEvent (MAE) Consumer Docker Image

docker/datahub-mae-consumer/README.md

1.6.01.1 KB
Original Source

DataHub MetadataAuditEvent (MAE) Consumer Docker Image

Refer to DataHub MAE Consumer Job to have a quick understanding of the architecture and responsibility of this service for the DataHub.

By-query RequestOptions (delete/update-by-query, etc.) use the same tuning as GMS: ELASTICSEARCH_BULK_BY_QUERY_SLOW_OPERATION_TIMEOUT_SECONDSelasticsearch.bulkProcessor.slowByQueryOperationTimeoutSeconds. MAE-specific Elasticsearch settings are MAE_ELASTICSEARCH_SOCKET_TIMEOUT and MAE_ELASTICSEARCH_CONNECTION_REQUEST_TIMEOUT only (maeConsumer.elasticsearch), merged with global RestClient timeouts when MAE_CONSUMER_ENABLED=true. ELASTICSEARCH_BUILD_INDICES_SLOW_OPERATION_TIMEOUT_SECONDS is for system-update / build-indices jobs only, not the bulk processor. docker.env sets longer RestClient values where appropriate.