apps/opik-documentation/documentation/fern/docs-v2/self-host/troubleshooting.mdx
This guide covers common troubleshooting scenarios for self-hosted Opik deployments.
Opik requires ClickHouse to be configured with cluster macros, even for single-node deployments. Opik migrations use the ON CLUSTER '{cluster}' clause to ensure DDL operations execute consistently across all nodes in a cluster.
If the {cluster} macro is not configured in your ClickHouse instance, migrations will fail with the following error:
Code: 139. DB::Exception: No macro 'cluster' in config.
Symptoms:
Code: 139. DB::Exception: No macro 'cluster' in config.Opik Helm Chart and Docker Compose deployments automatically configure the required cluster macros. If you're using Opik's provided deployment configurations, you should not encounter this issue.
If you're running your own ClickHouse instance (not using Opik's Helm chart or Docker Compose), you need to configure the cluster macros yourself.
Add the {cluster} macro to your ClickHouse configuration file. The location depends on your ClickHouse installation:
For standard ClickHouse installations:
Add the macros to /etc/clickhouse-server/config.d/macros.xml (or your equivalent config directory):
<clickhouse>
<macros>
<cluster>single_node_cluster</cluster>
<shard>1</shard>
<replica>clickhouse</replica>
</macros>
</clickhouse>
After adding the configuration, restart ClickHouse for the changes to take effect:
You can verify the macro is configured by connecting to ClickHouse and running:
SELECT * FROM system.macros WHERE macro = 'cluster';
You should see a row with the macro name and value.
After restarting ClickHouse, retry the backend deployment or migration. The backend should automatically retry after ClickHouse is ready.
A brand-new Opik install on a ClickHouse cluster with 2 or more replicas cannot complete its analytics migrations. The backend never starts (CrashLoopBackOff), and the migration step fails with errors such as:
Code: 60. DB::Exception: Could not find table: <table_name>. (UNKNOWN_TABLE)
Code: 81. DB::Exception: Database opik does not exist. (UNKNOWN_DATABASE)
opik is the default analytics database name (ANALYTICS_DB_DATABASE_NAME); substitute your own if you overrode it.
Symptoms:
opik database that is missing on one replicaopik database and tables while another has few or noneOpik's earliest analytics migrations predate cluster-aware DDL — they do not use ON CLUSTER '{cluster}' — so they create the opik database and base tables on a single replica only. Later, cluster-aware migrations fan out to every replica and fail on the one that never received that base schema. The ClickHouse operator only copies schema to a replica when it joins the cluster (a scale-up), not during a migration.
Install High-Availability deployments in two phases so the schema exists before the extra replicas join:
<Steps> <Step> Install (or reset to) `clickhouse.replicasCount: 1`. </Step> <Step> Wait until `opik-backend` is `Ready` — all migrations are applied on the single replica. </Step> <Step> Raise `clickhouse.replicasCount` (e.g. to `2`) and upgrade. The operator provisions the new replica and copies the fully-migrated schema to it. </Step> </Steps> <Callout intent="info"> Single-replica installs and upgrades of already-migrated multi-replica clusters are unaffected. Cluster-aware migrations use `ON CLUSTER '{cluster}'`, so once the base schema is present on all replicas, later migrations stay consistent automatically. </Callout> <Callout intent="warning"> If a fresh multi-replica attempt was torn down, the dropped tables can leave orphaned `/clickhouse/tables/...` ZooKeeper paths that cause `REPLICA_ALREADY_EXISTS` on a retry. Since a fresh install has no data to preserve, recovery only needs those orphaned paths cleared before re-running — the full replica-restore rebuild in [ClickHouse Zookeeper Metadata Loss](#clickhouse-zookeeper-metadata-loss) is for recovering existing data and isn't needed here. </Callout>If Zookeeper loses the metadata paths for ClickHouse tables, you will see coordination exceptions in the ClickHouse logs and potentially in the opik-backend service logs. These errors indicate that Zookeeper cannot find table metadata paths.
Symptoms:
Error messages appearing in ClickHouse logs and propagating to opik-backend service:
Code: 999. Coordination::Exception: Coordination error: No node, path /clickhouse/tables/0/default/DATABASECHANGELOG/log. (KEEPER_EXCEPTION)
This indicates that Zookeeper has lost the metadata paths for one or more ClickHouse tables.
Follow these steps to restore ClickHouse table metadata in Zookeeper:
If only some table paths are missing in Zookeeper, you'll need to delete the existing paths manually. Connect to the Zookeeper pod and use the Zookeeper CLI:
# Connect to Zookeeper pod
kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181
# Delete all ClickHouse table paths
deleteall /clickhouse/tables
Restart the ClickHouse pods so they become aware that Zookeeper no longer has the metadata:
kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
Connect to the first ClickHouse replica and restore the replica definitions for each table:
# Connect to the first ClickHouse replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client
Run the SYSTEM RESTORE REPLICA command for each table:
-- Restore system tables
SYSTEM RESTORE REPLICA default.DATABASECHANGELOG;
SYSTEM RESTORE REPLICA default.DATABASECHANGELOGLOCK;
-- Verify your Opik database name
SHOW DATABASES;
-- List all Opik tables (replace 'opik' with your actual schema name if different)
USE opik;
SHOW TABLES;
-- Restore each Opik table
SYSTEM RESTORE REPLICA opik.attachments;
SYSTEM RESTORE REPLICA opik.automation_rule_evaluator_logs;
SYSTEM RESTORE REPLICA opik.comments;
SYSTEM RESTORE REPLICA opik.dataset_items;
SYSTEM RESTORE REPLICA opik.experiment_items;
SYSTEM RESTORE REPLICA opik.experiments;
SYSTEM RESTORE REPLICA opik.feedback_scores;
SYSTEM RESTORE REPLICA opik.guardrails;
SYSTEM RESTORE REPLICA opik.optimizations;
SYSTEM RESTORE REPLICA opik.project_configurations;
SYSTEM RESTORE REPLICA opik.spans;
SYSTEM RESTORE REPLICA opik.traces;
SYSTEM RESTORE REPLICA opik.trace_threads;
SYSTEM RESTORE REPLICA opik.workspace_configurations;
Restart ClickHouse again to ensure it:
kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
After the restart completes, verify that the replica status is healthy:
-- Check table creation
SHOW CREATE TABLE opik.attachments;
-- Verify replica status
SELECT table, is_readonly, replica_is_active, zookeeper_exception
FROM system.replicas;
Expected Results:
is_readonly = 0 (table is writable)replica_is_active = 1 (replica is active)zookeeper_exception = '' (no exceptions)You can also verify from the Zookeeper side:
# Connect to Zookeeper CLI
kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181
# List tables (example path - adjust for your database name)
ls /clickhouse/tables/0/<database_name>/<table_name>
TOO_MANY_PARTS Errors and Stuck MergesUnder sustained high-volume ingestion, span/trace batch inserts may start failing with HTTP 500s while ClickHouse rejects new parts:
Code: 252. DB::Exception: Too many parts (N) in table 'opik.spans'.
Merges are processing significantly slower than inserts:
While executing WaitForAsyncInsert. (TOO_MANY_PARTS)
Clients calling POST /api/v1/private/spans/batch (or /traces/batch) receive 500s. The active part count for the table has exceeded ClickHouse's parts_to_throw_insert threshold (default 3000) and is not draining.
Symptoms:
TOO_MANY_PARTS (Code: 252) in the opik-backend logsspans and/or tracesConnectionRequestTimeoutException), amplifying the impact-- Active parts per node and table: is the count high and growing?
SELECT hostName() AS host, table, count() AS active_parts
FROM clusterAllReplicas('{cluster}', system.parts)
WHERE active AND database = '<database_name>'
GROUP BY host, table ORDER BY active_parts DESC;
-- Is a merge stuck? Look for a queue entry with a rising num_tries and a repeating exception.
SELECT hostName() AS host, database, table, type, num_tries, new_part_name, last_exception
FROM clusterAllReplicas('{cluster}', system.replication_queue)
WHERE database = '<database_name>'
ORDER BY num_tries DESC;
-- How many merges are actually running per node vs. the pool size?
SELECT hostName() AS host, count() AS running_merges
FROM clusterAllReplicas('{cluster}', system.merges)
GROUP BY host;
-- Replica health (rules out the ZooKeeper-metadata-loss scenario below).
SELECT hostName() AS host, table, is_readonly, replica_is_active, zookeeper_exception
FROM clusterAllReplicas('{cluster}', system.replicas);
running_merges is healthy, but parts are created faster than they merge. system.replication_queue shows no entry stuck with a high num_tries. This is driven by async-insert flush frequency under high concurrency.running_merges is near zero (despite a large background_pool_size) and one replication_queue entry has a high, climbing num_tries with a repeating last_exception — commonly Code: 76 ... CANNOT_OPEN_FILE on corrupt/missing source-part files. That one poison entry jams the scheduler so nothing else merges and parts cannot drain, regardless of CPU or disk.Opik applies its async-insert settings on the ClickHouse connection (via custom_http_params in ANALYTICS_DB_QUERY_PARAMETERS), so they apply to every insert. The shipped defaults favor freshness over batching:
async_insert=1
wait_for_async_insert=1
async_insert_busy_timeout_min_ms=100
async_insert_busy_timeout_max_ms=250
async_insert_use_adaptive_busy_timeout=1
With async_insert=1, each server-side flush becomes a new part. A short busy-timeout window (100–250 ms) means frequent flushes, and under high concurrency this creates many small parts that merges must keep up with.
To reduce fragmentation for high-volume deployments, widen the flush window (fewer, larger parts) via these opik-backend environment variables:
ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MAX_MS=2000 # ceiling of the adaptive flush window (ms); e.g. 1000–3000
ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MIN_MS=1000 # floor of the adaptive flush window (ms); keep below max
ANALYTICS_DB_ASYNC_INSERT_MAX_DATA_SIZE=52428800 # buffered bytes that force a flush; larger = fewer parts
Widening the flush window trades a little ingestion latency and buffer memory — rows become queryable up to max_ms later — for far fewer parts, a good trade for high-volume observability data. Under sustained high load, flushes are size-triggered anyway, so most of the added latency falls on quieter periods. After changing these, restart the opik-backend and watch the active part count and system.asynchronous_inserts, dialing max_ms between 1000–3000 ms to taste. New inserts will fragment less; an existing backlog still needs to merge down (it will, once the insert rate no longer outpaces merges).
Work from least to most invasive: the threshold bump (step 1) and diagnosis (steps 2–3) are non-destructive; DETACH (step 4) is recoverable but data-affecting; the ZooKeeper edit (step 6) is the last resort. Run the commands against the table your diagnosis flagged — replace <table> with spans or traces.
```sql
ALTER TABLE <database_name>.<table> ON CLUSTER '{cluster}' MODIFY SETTING parts_to_throw_insert = 20000, parts_to_delay_insert = 20000;
```
On a Distributed deployment, target the underlying per-shard ReplacingMergeTree table, not the Distributed proxy. This is a safe, reversible stopgap — no data is deleted — so revert to the defaults once the backlog has drained.
```sql
ALTER TABLE <database_name>.<table> DETACH PART 'all_1_100_5';
-- repeat for each corrupt part named in the exception
```
<Callout intent="warning">
Data-affecting, expert operation — do it with ClickHouse expertise on hand. On a `ReplicatedMergeTree` the queued merge entry references its source parts by name, so `DETACH` does **not** by itself remove that entry: it only helps when the part is intact elsewhere and can be re-fetched. If the part is corrupt on **all** replicas there is nothing to re-fetch — the entry keeps retrying and you must proceed to step 6 to clear it (and removing the part also drops its rows, a small bounded window for observability data). Always `DETACH` (recoverable), never `DROP`.
</Callout>
```
/clickhouse/tables/<shard>/<database_name>/<table>/replicas/<replica>/queue/queue-XXXXXXXXXX
```
<Callout intent="warning">
Editing ZooKeeper directly is dangerous and can corrupt replication if the wrong node is removed. Delete only the exact stuck entry, on all replicas, and prefer the `DETACH PART` path above whenever possible.
</Callout>
async_insert_busy_timeout_max_ms (Cause A above) before scaling ingestion up, and monitor part counts.system.replicas) and system.replication_queue is clean before ramping ingestion back up.system.replication_queue entries with a rising num_tries, so a stuck merge is caught before it becomes TOO_MANY_PARTS.Connect directly to ClickHouse pods for diagnostics:
# Connect to first replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client
# Connect to second replica (if running multiple replicas)
kubectl exec -it chi-opik-clickhouse-cluster-0-1-0 -- clickhouse-client
Connect directly to Zookeeper pods:
# Connect to Zookeeper pod
kubectl exec -it cometml-production-opik-zookeeper-0 -- bash
# Run Zookeeper client commands
zkCli.sh -server localhost:2181
Common Zookeeper commands:
# List tables in Zookeeper
kubectl exec -it cometml-production-opik-zookeeper-0 -- \
zkCli.sh -server localhost:2181 ls /clickhouse/tables/0/opik
# Remove a specific table from Zookeeper
kubectl exec -it cometml-production-opik-zookeeper-0 -- \
zkCli.sh -server localhost:2181 \
deleteall /clickhouse/tables/0/opik/optimizations
To avoid Zookeeper metadata loss issues:
Regular Backups: Implement regular backups of ClickHouse data. See the Advanced ClickHouse Backup guide for details.
Monitoring: Set up monitoring for Zookeeper health and ClickHouse replica status. Alert on zookeeper_exception in system.replicas.
Resource Allocation: Ensure Zookeeper has adequate resources (CPU, memory, disk) to maintain metadata reliably.
Persistent Storage: Use persistent volumes for Zookeeper to prevent data loss during pod restarts.
Replica Validation: Regularly check replica status with the diagnostic queries above.
If you continue to experience issues after following this guide: