docs/content/en/guides/troubleshooting/meshery-operator-meshsync.md
{{% alert color="info" title="What is Meshery Operator?" %}} <a href='{{< ref "concepts/architecture/operator/index.md" >}}'>Meshery Operator</a> controls and monitors the lifecycle of components deployed inside Meshery-managed Kubernetes clusters. Learn more about <a href='{{< ref "concepts/_index.md" >}}'>Meshery's architecture</a>. {{% /alert %}}
This guide offers comprehensive troubleshooting instructions for [Meshery Operator]({{< ref "concepts/architecture/operator/index.md" >}}) and its custom controllers, [MeshSync]({{< ref "concepts/architecture/meshsync.md" >}}) and [Broker]({{< ref "concepts/architecture/broker/index.md" >}}). Follow the steps outlined in this document to ensure a smooth Meshery deployment.
First, understand the Meshery Operator Deployment Scenarios and the Status of Meshery Operator, MeshSync, and Meshery Broker to identify the deployment model fitting of your environment. Then, follow the guidance under the respective scenario to troubleshoot accordingly.
{{% alert color="dark" title="Meshery Error Code Reference" %}} Have specific error with an error code? See the <a href='{{< ref "reference/references/error-codes.md" >}}'>Meshery Error Code Reference</a> for probable cause and suggested remediations. {{% /alert %}}
Each Meshery Operator controller offers a health status that you can use to understand its current health in your deployment. These statuses are computed by Meshery Server from what it observes of the Operator, MeshSync, and Broker; their meanings are described below.
Any of the three can instead report UNKOWN (spelled that way on the wire). It is not a health state: it means Meshery made no observation of that controller on this cluster, so it is asserting nothing about it. All three report it at once when the connection's kubeconfig could not be read or its Kubernetes client could not be created, since nothing about the cluster was observable. The cards stay visible on purpose - the reason is in the connection's Diagnostics and in the events feed, not in the status itself.
Because Meshery is versatile in its deployment models, there are different scenarios in which you may need to troubleshoot the health of Meshery Operator. Identify the deployment model fitting your environment and follow the guidance under the respective scenario to troubleshoot accordingly.
Whether using [mesheryctl system start]({{< ref "installation/_index.md" >}}), [helm install]({{< ref "installation/kubernetes/helm.md" >}}) or make run-local, Meshery Server will automatically connect to any available Kubernetes clusters found in your kubeconfig (under $HOME/.kube/config). Once connected, operator, broker(NATS) and meshsync will automatically get deployed in the same clusters.
If everything is fine, by viewing the connection in Meshery UI, MeshSync should be in CONNECTED: state. Otherwise, check the Operator's pod logs:
kubectl logs <meshery-operator-pod> -n meshery
In this model Meshery Server must both reach and authenticate to the in-cluster Meshery Broker (NATS):
ClusterIP only, which is not reachable from outside the cluster. When Meshery Server runs out-of-cluster it automatically establishes a self-healing port-forward to the Broker's NATS pod through the Kubernetes API server (like kubectl port-forward, using the credentials Meshery already holds) and connects over it - no manual step required. This is on by default out-of-cluster, skipped automatically in-cluster, and can be disabled with MESHERY_MANAGED_BROKER_PORTFORWARD=false (after which you must provide your own path, e.g. kubectl port-forward -n meshery svc/meshery-nats 4222:4222, or expose the Broker via NodePort/LoadBalancer).meshery-nats-auth); Meshery Server reads that token and presents it automatically. Without it the Broker rejects the connection with an authorization violation.For the full walkthrough of the Kubernetes connection lifecycle, its components, and these connectivity behaviors, see [Kubernetes Connection Lifecycle]({{< ref "guides/infrastructure-management/kubernetes-connection-lifecycle.md" >}}).
Some common failure situations that Meshery users might face are described below.
ClusterIP-only and unreachable, or Meshery is not presenting the NATS token. See Out-of-Cluster Deployment; the connection's Diagnostics will report broker_unreachable with remediation.meshery-nats (Broker) pod is in CrashLoopBackOff and never becomes ready.
nats.conf unquoted (token: $NATS_TOKEN). When the generated token happens to look like a number, the NATS config parser rejects it and the pod crash-loops. Confirm with kubectl logs -n meshery meshery-nats-0 -c nats (look for a variable reference for 'NATS_TOKEN' ... could not be parsed error). The fix belongs in the Operator (quote it: token: "$NATS_TOKEN"); redeploying often generates a token that parses.meshery-operator pod never becomes ready after connecting a cluster. Its kube-rbac-proxy container is in ImagePullBackOff, and/or its manager container crash-loops with open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory.
Signature. After adding a Kubernetes connection, the operator never reaches DEPLOYED. Both of these appear together:
kubectl -n meshery get pods
# meshery-operator-... 1/2 ImagePullBackOff (kube-rbac-proxy)
# CrashLoopBackOff (manager)
kubectl -n meshery describe pod -l app=meshery-operator | grep -A2 kube-rbac-proxy
# Failed to pull image "registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.16.0"
# ...older charts name the same sidecar under the other registry:
# Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.16.0"
kubectl -n meshery logs deploy/meshery-operator -c manager
# open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory
Cause. Both symptoms come from one thing: an old meshery-operator Helm chart, from before v1.0.51.
kube-rbac-proxy sidecar next to the manager. Charts around v0.8.214 and above - including the chart a helm-installed Meshery Server asks for, which is the usual case - name it registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.16.0; older charts such as v0.8.180 name the same image as gcr.io/kubebuilder/kube-rbac-proxy:v0.16.0. Match on either. When that image does not pull, the container sits in ImagePullBackOff and the Pod never becomes ready.
gcr.io copy of v0.16.0 is gone (the repository's tag list is empty and the tag returns 404), while the registry.k8s.io copy still resolves from an unrestricted network. So if your cluster reports the pull failing on registry.k8s.io, treat it as something about that cluster's access to the registry rather than as the image having been withdrawn. The remedy below does not depend on which it is.ENABLE_WEBHOOKS on the manager and mount no serving certificate. Current operator images treat an unset ENABLE_WEBHOOKS as enabled, so the manager looks for a certificate that the old chart never created and crash-loops.v1.0.51 is the oldest published chart confirmed to render without the sidecar and with ENABLE_WEBHOOKS=false - the conversion webhook opt-in, off by default - and every chart above it does the same. Both conditions behind the symptoms - the sidecar, and the absent ENABLE_WEBHOOKS - were confirmed by rendering the published archives of v1.0.40 and of the contiguous run v1.0.41 through v1.0.50 (v1.0.47 was never published); charts older than v1.0.40 were not rendered, so treat "older than v1.0.51" as the boundary rather than a claim about any specific ancient release.
Remedy. Get a chart at or above v1.0.51 into the cluster.
If Meshery Server deployed the operator for you (the usual case: you added a kubeconfig and Meshery installed the operator), simply upgrade Meshery Server. The version a Server derives from its own release is floored: the Server resolves it against what the repository actually publishes and raises anything older than v1.0.51 to the oldest published chart at or above that boundary, telling you it did so in the events feed. Then reconnect the cluster.
The floor covers that derived version and nothing else, so two cases still leave an old chart in place:
operator.version you set is honored as written and is never raised, so a connection pinned below v1.0.51 keeps reinstalling that chart until you change or clear the pin. See Choosing the chart version yourself.v1.0.51, the newest published chart is deployed anyway, and the events feed says so and warns that the Operator may not become ready.If you installed the operator chart yourself with Helm:
helm repo add meshery https://meshery.github.io/meshery.io/charts
helm repo update
helm upgrade --install meshery-operator meshery/meshery-operator \
--namespace meshery --create-namespace
helm repo update matters on its own: a stale local repository cache will happily reinstall the same broken chart.
If the operator Pod is still wedged after the upgrade, delete it so the new spec takes effect immediately:
kubectl -n meshery rollout restart deploy/meshery-operator
The chart version and the operator image tag are different numbers; the image tag is what tells you which operator is actually running:
# The operator image actually running in the cluster
kubectl -n meshery get deploy meshery-operator \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
# The Helm chart release that installed it (chart version and app version)
helm -n meshery list --filter meshery-operator
A kube-rbac-proxy container in the output of the following command means the chart predates v1.0.51, whatever version it claims:
kubectl -n meshery get deploy meshery-operator \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
Meshery Server normally deploys the operator chart that matches its own release, falling back to the newest published chart when that one is not published yet (chart publishing trails Meshery Server releases). To pin a specific chart version for one connection, set operator.version in the connection's controllers configuration.
The value must be a chart version that the repository publishes, for example v1.0.64 (the leading v is optional - 1.0.64 names the same chart). A moving tag such as stable-latest, or a version that is not published, is rejected with a visible error rather than being silently replaced. Clear the field to go back to tracking the Meshery Server release.
A pin is deliberate, so it also escapes the minimum-version floor that guards the derived version: operator.version below v1.0.51 is installed as written rather than raised. Every published chart checked below that boundary leaves the Operator Pod unable to become ready (see Meshery Operator will not start), so choosing a chart that deploys is yours to get right.
Release candidates are the one thing Meshery will never pick for you: when it falls back to the newest published chart, or raises an old one to the oldest chart known to deploy, it skips any version carrying a prerelease suffix such as v1.0.66-rc.1. Naming a prerelease in operator.version deploys it exactly as asked.
If the chart repository cannot be reached, Meshery still reports the operator's status and image version for an operator that is already installed - only installing or upgrading it is withheld, and the reason appears in the connection's diagnostics and in the events feed.
A version that cannot be resolved - an operator.version naming something the repository does not publish, or a chart repository Meshery could not read at all - stops the install before Helm is called. Nothing partial is applied to the cluster.
The failure is visible, not silent. The Operator's status card carries the error, and the connection's Diagnostics report operator_deploy_failed with the underlying cause; the same cause appears in the events feed. The card does not vanish and the operator does not sit at a blank status while Meshery quietly gives up.
Retrying is the remedy for a transient outage. Meshery re-resolves the version on a user-initiated deploy, so a chart repository that was briefly unreachable needs no reconnect and no restart: redeploy the Operator from the connection's actions and it self-heals. Check first that Meshery Server has outbound access to https://meshery.github.io/meshery.io/charts ([egress requirements]({{< ref "installation/production/networking-and-connectivity.md#egress-requirements" >}})).
If the error names your own pin, the pin is the problem - correct operator.version to a published version or clear it to go back to tracking the Meshery Server release. An explicit pin is never quietly swapped for a working one.
Meshery Operator, MeshSync, and Broker are crucial components in a Meshery deployment. Meshery can function without them, but some functions of Meshery will be disable / unusable. Whether Meshery Operator is initially deployed via mesheryctl command or via Meshery Server, you can monitor the health of the Meshery Operator deployment using either the CLI or UI clients.
The following commands are available to troubleshoot Meshery Operator, MeshSync, and Broker.
Meshery Server and Adapters
mesheryctl system status - Displays the status of Meshery Server and Meshery Adapters.Meshery Operator, MeshSync, and Broker
mesheryctl system check - Displays the status of Meshery Operator, MeshSync, and Broker.Based on discussed scenarios, the UI exposes tools to perform the following actions:
Click a Kubernetes connection's row in the Connections table to open its detail view. Below the Operator / MeshSync / Broker status chips, a Diagnostics section lists actionable problems and remediation, derived from the live controller status and Meshery's actual Broker connection. Every code it reports, and the remedy for each, is listed in [Diagnostics]({{< ref "guides/infrastructure-management/kubernetes-connection-lifecycle.md#diagnostics" >}}).
The same data is available at GET /api/system/controllers/diagnostics?connectionId=<id>.
Initiate a synthetic check to verify a fully functional Operator deployment, testing MeshSync/Broker connectivity.
When the CLI and UI clients don't explain why data is missing or stale, inspect the MeshSync pod directly.
Read MeshSync logs (enable debug logging for detail):
kubectl -n meshery logs deploy/meshery-meshsync
# For verbose output, set DEBUG=true on the Deployment and let it restart:
kubectl -n meshery set env deploy/meshery-meshsync DEBUG=true
Check liveness and readiness (MeshSync serves these on port 11000):
kubectl -n meshery port-forward deploy/meshery-meshsync 11000:11000 &
curl -sS http://127.0.0.1:11000/healthz # liveness
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:11000/readyz # 200 == connected to Broker
{{% alert color="info" title="What readiness does and does not mean" %}} <code>/readyz</code> returns <code>200</code> once MeshSync has connected to the Broker, <strong>not</strong> once its informer caches have finished priming. Immediately after a (re)start MeshSync may report ready while its cluster snapshot is still filling in. If Meshery shows a partial cluster right after a restart, give discovery a moment or trigger a fresh discovery with <code>kubectl -n meshery rollout restart deploy/meshery-meshsync</code>. {{% /alert %}}
Verify the Broker is reachable from MeshSync. On startup MeshSync runs a connectivity test against the Broker's monitoring endpoint (http://<broker-host>:8222/connz) before opening its NATS client; a failure here appears in the MeshSync logs and blocks readiness. Confirm the BROKER_URL value and that the Broker Service is reachable:
kubectl -n meshery get deploy meshery-meshsync \
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="BROKER_URL")].value}{"\n"}'
caBundle fields), this can cause repeated re-discovery and transient load or gaps. If you observe this, scope discovery with a whitelist (see the [MeshSync configuration FAQ]({{< ref "concepts/architecture/meshsync.md#meshsync-faqs" >}})).secrets.v1., and the Secret objects it forwards to Meshery Server include their data and stringData payload. Those Secret contents are therefore transmitted over the Broker and persisted in the Meshery Database. In security-sensitive environments, either blacklist secrets.v1. (or use a whitelist that omits it) to keep Secrets out of discovery entirely, or set MESHSYNC_REDACT_SECRETS=true on the MeshSync Deployment to keep discovering Secrets while replacing their values with [REDACTED] (keys are preserved). See [Redacting Secret contents]({{< ref "guides/infrastructure-management/configuring-operator-meshsync-broker.md#redacting-secret-contents" >}}).kubectl -n meshery rollout restart deploy/meshery-meshsync or reset the Meshery Database from the UI.{{< related-discussions tag="meshery" >}}