Deploy Loki meta-monitoring

The primary method for collecting and monitoring a Loki cluster is to use the Kubernetes Monitoring Helm chart. This chart provides a comprehensive monitoring solution for Kubernetes clusters and includes direct integrations for monitoring the full LGTM (Loki, Grafana, Tempo, and Mimir) stack. This procedure will walk you through deploying the Kubernetes Monitoring Helm chart to monitor your Loki cluster.

{{< admonition type="note" >}} We recommend running a production cluster of Loki in distributed mode using Kubernetes. This procedure assumes you have a running Kubernetes cluster and a running Loki deployment. There are other methods for deploying Loki, such as using Docker or VM installations. meta-monitoring is still possible when using these deployment methods but not covered in this procedure. {{< /admonition >}}

Prerequisites

kubectl
Helm 3 or above. See Installing Helm.
A running Kubernetes cluster with a running Loki deployment.
A Grafana Cloud account or a separate LGTM stack for monitoring.

Preparing your environment

Before deploying the Kubernetes Monitoring Helm chart, you need to set up several components in your environment.

Add the Grafana Helm repository:

bash

helm repo add grafana https://grafana.github.io/helm-charts

Update your Helm repositories:
bash
```
helm repo update
```
Create a namespace for the monitoring stack:
bash
```
kubectl create namespace meta
```

Authentication

The Kubernetes Monitoring Helm chart requires a Grafana Cloud account or a separate LGTM stack for monitoring. You will need to provide the necessary credentials to the Helm chart to authenticate with your Grafana Cloud account or LGTM stack. In this procedure, we will use Grafana Cloud as an example.

Create a new Cloud Access Policy in Grafana Cloud.
1. Sign into Grafana Cloud.
2. In the main menu, select Security > Access Policies.
3. Click Create access policy.
4. Give the policy a Name and select the following permissions:
  - Metrics: Write
  - Logs: Write
5. Click Create.
6. Click Add Token. Give the token a name and click Create. Save the token for later use.
Collect url and username for Prometheus and Loki.
1. Navigate to the Grafana Cloud Portal Overview page.
2. Click the Details button for your Prometheus instance.
  1. From the Sending metrics using Grafana Alloy section, collect the instance username and url.
  2. Navigate back to the Overview page.
3. Click the Details button for your Loki instance.
  1. From the Sending Logs to Grafana Cloud using Grafana Alloy section, collect the instance username and url.
  2. Navigate back to the Overview page.

Create the Kubernetes Secrets with the collected credentials from Grafana Cloud.

bash

kubectl create secret generic metrics -n meta \
 --from-literal=username=<PROMETHEUS-USER> \
 --from-literal=password=<CLOUD-TOKEN>

kubectl create secret generic logs -n meta \
 --from-literal=username=<LOKI-USER> \
 --from-literal=password=<CLOUD-TOKEN>

Note that the Kubernetes Monitoring Helm supports many different authentication methods based upon your requirements including:

Bearer Tokens
OAuth2
SigV4
External Secrets

For further information on how to configure these methods, see the Kubernetes Monitoring Helm examples.

Deploy the Kubernetes Monitoring Helm chart

Now that you have prepared your environment and collected the necessary credentials, you can deploy the Kubernetes Monitoring Helm chart to monitor your Loki cluster. To do this we need to copy a values.yaml file to our local machine and modify it to include the necessary configuration.

Download the values.yaml file from the Kubernetes Monitoring Helm chart repository:

bash

curl -O https://raw.githubusercontent.com/grafana/loki/main/production/helm/meta-monitoring/values.yaml

Open the values.yaml file in a text editor of your choosing and add the Prometheus and Loki endpoints.

yaml

destinations:
  - name: prometheus
    type: prometheus
    url: https://<PROMETHEUS-ENDPOINT>/api/prom/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
        create: false
        name: metrics
        namespace: meta

  - name: loki
    type: loki
    url: https://<LOKI-ENDPOINT>/loki/api/v1/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
        create: false
        name: logs
        namespace: meta

(Optional) Update the cluster name to a human-readable name to identify your cluster in Grafana Cloud.

yaml

# Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster.
 cluster:
     name: loki-meta-monitoring-cluster

The default values file assumes that you have deployed Loki in the loki namespace and will deploy the Kubernetes monitoring stack in the meta namespace. If you have deployed Loki in a different namespace, you will need to update namespaces in the values.yaml file to match the namespace where Loki is deployed. Here is an example:
yaml
```
namespaces:
    - loki
```
Note if you would like to collect from all available namespaces, you can remove the namespaces key from the values.yaml file.
Deploy the Kubernetes Monitoring Helm chart using the modified values.yaml file:
bash
```
helm install meta-loki grafana/k8s-monitoring \
 --namespace meta \
 -f values.yaml
```

Verify that the Kubernetes Monitoring Helm chart has been deployed successfully:

bash

 kubectl get pods -n meta

You should see a list of pods running in the meta namespace.

console

NAME                                           READY   STATUS    RESTARTS ...        
meta-loki-alloy-singleton-6d7f8d8b86-sg4wx     2/2     Running   0        ...       
meta-loki-kube-state-metrics-64bdcfcbd-5snqz   1/1     Running   0        ...       
meta-loki-node-exporter-855l5                  1/1     Running   0        ...       
meta-loki-node-exporter-b976b                  1/1     Running   0        ...       
meta-loki-node-exporter-vsm4s                  1/1     Running   0        ...

Next Steps

You have successfully deployed the Kubernetes Monitoring Helm chart to monitor your Loki cluster. You can now move onto the next step of deploying the Loki mixin to visualize the metrics and logs from your Loki cluster. For more information, see Install the Loki Mixin.