ContextQMD
Back to Clickhouse

system.iceberg_metadata_log

docs/en/operations/system-tables/iceberg_metadata_log.md

26.4.1.1-new5.5 KB
Original Source

import SystemTableCloud from '@site/docs/_snippets/_system_table_cloud.md';

system.iceberg_metadata_log

The system.iceberg_metadata_log table records metadata access and parsing events for Iceberg tables read by ClickHouse. It provides detailed information about each metadata file or entry processed, which is useful for debugging, auditing, and understanding Iceberg table structure evolution.

Purpose {#purpose}

This table logs every metadata file and entry read from Iceberg tables, including root metadata files, manifest lists, and manifest entries. It helps users trace how ClickHouse interprets Iceberg table metadata and diagnose issues related to schema evolution, file resolution, or query planning.

:::note This table is primarily intended for debugging purposes. :::

Columns {#columns}

NameTypeDescription
event_dateDateDate of the log entry.
event_timeDateTimeTimestamp of the event.
query_idStringQuery ID that triggered the metadata read.
content_typeEnum8Type of metadata content (see below).
table_pathStringPath to the Iceberg table.
file_pathStringPath to the root metadata JSON file, Avro manifest list, or manifest file.
contentStringContent in JSON format (raw metadata from .json, Avro metadata, or Avro entry).
row_in_fileNullable(UInt64)Row number in the file, if applicable. Present for ManifestListEntry and ManifestFileEntry content types.
pruning_statusNullable(Enum8)Pruning status for the entry. 'NotPruned', 'PartitionPruned', 'MinMaxIndexPruned'. Pay attention that partition pruning is done before minmax pruning so 'PartitionPruned' means that the entry was pruned by partition filter and minmax pruning was not even attempted. Present for ManifestFileEntry content type.

content_type values {#content-type-values}

  • None: No content.
  • Metadata: Root metadata file.
  • ManifestListMetadata: Manifest list metadata.
  • ManifestListEntry: Entry in a manifest list.
  • ManifestFileMetadata: Manifest file metadata.
  • ManifestFileEntry: Entry in a manifest file.

Controlling log verbosity {#controlling-log-verbosity}

You can control which metadata events are logged using the iceberg_metadata_log_level setting.

To log all metadata used in the current query:

sql
SELECT * FROM my_iceberg_table SETTINGS iceberg_metadata_log_level = 'manifest_file_entry';

SYSTEM FLUSH LOGS iceberg_metadata_log;

SELECT content_type, file_path, row_in_file
FROM system.iceberg_metadata_log
WHERE query_id = '{previous_query_id}';

To log only the root metadata JSON file used in the current query:

sql
SELECT * FROM my_iceberg_table SETTINGS iceberg_metadata_log_level = 'metadata';

SYSTEM FLUSH LOGS iceberg_metadata_log;

SELECT content_type, file_path, row_in_file
FROM system.iceberg_metadata_log
WHERE query_id = '{previous_query_id}';

See more information in the description of the iceberg_metadata_log_level setting.

Good To Know {#good-to-know}

  • Use iceberg_metadata_log_level at the query level only when you need to investigate your Iceberg table in detail. Otherwise, you may populate the log table with excessive metadata and experience performance degradation.
  • The table contains duplicate entries, as it is intended primarily for debugging and does not guarantee uniqueness per entity. Separate rows store content and pruning status because they are collected at different moments in a program. Content is collected when the metadata is read, pruning status is collected when the metadata is checked for pruning. Never rely on the table itself for deduplication.
  • If you use a content_type more verbose than ManifestListMetadata, the Iceberg metadata cache is disabled for manifest lists.
  • Similarly, if you use a content_type more verbose than ManifestFileMetadata, the Iceberg metadata cache is disabled for manifest files.
  • If the SELECT query was cancelled or failed, the log table may still contain entries for metadata processed before the failure but will not contain information about metadata entities that were not processed.

See also {#see-also}