ContextQMD
Back to Opik

Troubleshooting

apps/opik-documentation/documentation/fern/docs-v2/self-host/troubleshooting.mdx

2.2.2-609311.3 KB
Original Source

This guide covers common troubleshooting scenarios for self-hosted Opik deployments.

Common Issues

ClickHouse Migration Failures: Missing Cluster Macro

Problem Description

Opik requires ClickHouse to be configured with cluster macros, even for single-node deployments. Opik migrations use the ON CLUSTER '{cluster}' clause to ensure DDL operations execute consistently across all nodes in a cluster.

If the {cluster} macro is not configured in your ClickHouse instance, migrations will fail with the following error:

Code: 139. DB::Exception: No macro 'cluster' in config.

Symptoms:

  • Backend fails to start or enters CrashLoopBackOff state
  • Migration errors appear in backend logs
  • Error message: Code: 139. DB::Exception: No macro 'cluster' in config.

Automatic Configuration

Opik Helm Chart and Docker Compose deployments automatically configure the required cluster macros. If you're using Opik's provided deployment configurations, you should not encounter this issue.

Manual Configuration Required

If you're running your own ClickHouse instance (not using Opik's Helm chart or Docker Compose), you need to configure the cluster macros yourself.

1. Add Cluster Macro Configuration

Add the {cluster} macro to your ClickHouse configuration file. The location depends on your ClickHouse installation:

For standard ClickHouse installations:

Add the macros to /etc/clickhouse-server/config.d/macros.xml (or your equivalent config directory):

xml
<clickhouse>
    <macros>
        <cluster>single_node_cluster</cluster>
        <shard>1</shard>
        <replica>clickhouse</replica>
    </macros>
</clickhouse>
<Callout intent="info"> **Note**: For single-node setups, you can use any value for `<cluster>`. The value `single_node_cluster` is just an example. For multi-node clusters, use your actual cluster name that matches your `<remote_servers>` configuration. </Callout>
2. Restart ClickHouse

After adding the configuration, restart ClickHouse for the changes to take effect:

3. Verify Configuration

You can verify the macro is configured by connecting to ClickHouse and running:

sql
SELECT * FROM system.macros WHERE macro = 'cluster';

You should see a row with the macro name and value.

4. Retry Migrations

After restarting ClickHouse, retry the backend deployment or migration. The backend should automatically retry after ClickHouse is ready.

Fresh Multi-Replica Install Migration Failures

Problem Description

A brand-new Opik install on a ClickHouse cluster with 2 or more replicas cannot complete its analytics migrations. The backend never starts (CrashLoopBackOff), and the migration step fails with errors such as:

Code: 60. DB::Exception: Could not find table: <table_name>. (UNKNOWN_TABLE)
Code: 81. DB::Exception: Database opik does not exist. (UNKNOWN_DATABASE)

opik is the default analytics database name (ANALYTICS_DB_DATABASE_NAME); substitute your own if you overrode it.

Symptoms:

  • Fresh install only (no existing data); backend in CrashLoopBackOff
  • Migration errors reference a table or the opik database that is missing on one replica
  • One replica holds the opik database and tables while another has few or none

Cause

Opik's earliest analytics migrations predate cluster-aware DDL — they do not use ON CLUSTER '{cluster}' — so they create the opik database and base tables on a single replica only. Later, cluster-aware migrations fan out to every replica and fail on the one that never received that base schema. The ClickHouse operator only copies schema to a replica when it joins the cluster (a scale-up), not during a migration.

Resolution

Install High-Availability deployments in two phases so the schema exists before the extra replicas join:

<Steps> <Step> Install (or reset to) `clickhouse.replicasCount: 1`. </Step> <Step> Wait until `opik-backend` is `Ready` — all migrations are applied on the single replica. </Step> <Step> Raise `clickhouse.replicasCount` (e.g. to `2`) and upgrade. The operator provisions the new replica and copies the fully-migrated schema to it. </Step> </Steps> <Callout intent="info"> Single-replica installs and upgrades of already-migrated multi-replica clusters are unaffected. Cluster-aware migrations use `ON CLUSTER '{cluster}'`, so once the base schema is present on all replicas, later migrations stay consistent automatically. </Callout> <Callout intent="warning"> If a fresh multi-replica attempt was torn down, the dropped tables can leave orphaned `/clickhouse/tables/...` ZooKeeper paths that cause `REPLICA_ALREADY_EXISTS` on a retry. Since a fresh install has no data to preserve, recovery only needs those orphaned paths cleared before re-running — the full replica-restore rebuild in [ClickHouse Zookeeper Metadata Loss](#clickhouse-zookeeper-metadata-loss) is for recovering existing data and isn't needed here. </Callout>

ClickHouse Zookeeper Metadata Loss

Problem Description

If Zookeeper loses the metadata paths for ClickHouse tables, you will see coordination exceptions in the ClickHouse logs and potentially in the opik-backend service logs. These errors indicate that Zookeeper cannot find table metadata paths.

Symptoms:

Error messages appearing in ClickHouse logs and propagating to opik-backend service:

Code: 999. Coordination::Exception: Coordination error: No node, path /clickhouse/tables/0/default/DATABASECHANGELOG/log. (KEEPER_EXCEPTION)

This indicates that Zookeeper has lost the metadata paths for one or more ClickHouse tables.

Resolution Steps

Follow these steps to restore ClickHouse table metadata in Zookeeper:

1. Clean Zookeeper Paths (If Needed)

If only some table paths are missing in Zookeeper, you'll need to delete the existing paths manually. Connect to the Zookeeper pod and use the Zookeeper CLI:

bash
# Connect to Zookeeper pod
kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181

# Delete all ClickHouse table paths
deleteall /clickhouse/tables
<Callout intent="warning"> **Warning**: This operation removes all table metadata from Zookeeper. Proceed with caution. </Callout>
2. Restart ClickHouse

Restart the ClickHouse pods so they become aware that Zookeeper no longer has the metadata:

bash
kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
3. Restore Replica Definitions

Connect to the first ClickHouse replica and restore the replica definitions for each table:

bash
# Connect to the first ClickHouse replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client
<Callout intent="warning"> **Important**: The Opik schema name is typically `opik` but may vary depending on your installation. Before proceeding, verify your schema name by running `SHOW DATABASES;` in ClickHouse and identifying the Opik database. Use that database name in all subsequent commands. </Callout>

Run the SYSTEM RESTORE REPLICA command for each table:

sql
-- Restore system tables
SYSTEM RESTORE REPLICA default.DATABASECHANGELOG;
SYSTEM RESTORE REPLICA default.DATABASECHANGELOGLOCK;

-- Verify your Opik database name
SHOW DATABASES;

-- List all Opik tables (replace 'opik' with your actual schema name if different)
USE opik;
SHOW TABLES;

-- Restore each Opik table
SYSTEM RESTORE REPLICA opik.attachments;
SYSTEM RESTORE REPLICA opik.automation_rule_evaluator_logs;
SYSTEM RESTORE REPLICA opik.comments;
SYSTEM RESTORE REPLICA opik.dataset_items;
SYSTEM RESTORE REPLICA opik.experiment_items;
SYSTEM RESTORE REPLICA opik.experiments;
SYSTEM RESTORE REPLICA opik.feedback_scores;
SYSTEM RESTORE REPLICA opik.guardrails;
SYSTEM RESTORE REPLICA opik.optimizations;
SYSTEM RESTORE REPLICA opik.project_configurations;
SYSTEM RESTORE REPLICA opik.spans;
SYSTEM RESTORE REPLICA opik.traces;
SYSTEM RESTORE REPLICA opik.trace_threads;
SYSTEM RESTORE REPLICA opik.workspace_configurations;
<Callout intent="info"> **Note**: The exact list of tables may vary depending on your Opik version. Use the `SHOW DATABASES;` or `\d` command to list all tables in your database and restore each one. </Callout>
4. Restart ClickHouse Again

Restart ClickHouse again to ensure it:

  • Re-establishes connections to Zookeeper
  • Verifies and synchronizes the newly restored metadata
  • Automatically resumes normal replication operations
bash
kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
5. Validate the Recovery

After the restart completes, verify that the replica status is healthy:

sql
-- Check table creation
SHOW CREATE TABLE opik.attachments;

-- Verify replica status
SELECT table, is_readonly, replica_is_active, zookeeper_exception
FROM system.replicas;

Expected Results:

  • is_readonly = 0 (table is writable)
  • replica_is_active = 1 (replica is active)
  • zookeeper_exception = '' (no exceptions)

You can also verify from the Zookeeper side:

bash
# Connect to Zookeeper CLI
kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181

# List tables (example path - adjust for your database name)
ls /clickhouse/tables/0/<database_name>/<table_name>

Diagnostic Commands

Connecting to ClickHouse

Connect directly to ClickHouse pods for diagnostics:

bash
# Connect to first replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client

# Connect to second replica (if running multiple replicas)
kubectl exec -it chi-opik-clickhouse-cluster-0-1-0 -- clickhouse-client

Connecting to Zookeeper

Connect directly to Zookeeper pods:

bash
# Connect to Zookeeper pod
kubectl exec -it cometml-production-opik-zookeeper-0 -- bash

# Run Zookeeper client commands
zkCli.sh -server localhost:2181

Common Zookeeper commands:

bash
# List tables in Zookeeper
kubectl exec -it cometml-production-opik-zookeeper-0 -- \
  zkCli.sh -server localhost:2181 ls /clickhouse/tables/0/opik

# Remove a specific table from Zookeeper
kubectl exec -it cometml-production-opik-zookeeper-0 -- \
  zkCli.sh -server localhost:2181 \
  deleteall /clickhouse/tables/0/opik/optimizations

Prevention and Best Practices

To avoid Zookeeper metadata loss issues:

  1. Regular Backups: Implement regular backups of ClickHouse data. See the Advanced ClickHouse Backup guide for details.

  2. Monitoring: Set up monitoring for Zookeeper health and ClickHouse replica status. Alert on zookeeper_exception in system.replicas.

  3. Resource Allocation: Ensure Zookeeper has adequate resources (CPU, memory, disk) to maintain metadata reliably.

  4. Persistent Storage: Use persistent volumes for Zookeeper to prevent data loss during pod restarts.

  5. Replica Validation: Regularly check replica status with the diagnostic queries above.

Getting Help

If you continue to experience issues after following this guide:

  1. Check the Opik GitHub Issues for similar problems
  2. Review ClickHouse and Zookeeper logs for additional error details
  3. Open a new issue on GitHub with:
    • Opik versions:
      • Backend version (opik-backend)
      • Frontend version (opik-frontend)
      • Helm chart version (if deployed via Helm)
    • ClickHouse version
    • Zookeeper version
    • Error logs from all services (ClickHouse, Zookeeper, opik-backend)
    • Steps taken to reproduce the issue