⚠️ Repo Archive Notice

As of Nov 13, 2020, charts in this repo will no longer be updated. For more information, see the Helm Charts Deprecation and Archive Notice, and Update.

SumoLogicFluentd

Sumo Logic is a hosted logging platform.

DEPRECATION NOTICE

This chart is deprecated and no longer supported.

Introduction

This chart adds the Sumo Logic FluentD Plugin to all nodes in your cluster as a DaemonSet. The image supports fluentd file and systemd log sources.

After you have installed the chart, each pod, deployment, etc. can be optionally configured to specify its log format, source category, source name, or exclude itself from SumoLogic.

Configure an individual pod

annotations:
  sumologic.com/format: "text"
  sumologic.com/sourceCategory: "mywebsite/nginx"
  sumologic.com/sourceName: "mywebsite_nginx"

Prevent an individual pod from logging

annotations:
  sumologic.com/exclude: "true"

Prerequisites

• Kubernetes 1.10+ with Beta APIs enabled. However, certain configuration parameters may require a more recent version of Kubernetes. Such parameters will specify the minimum Kubernetes version required in the parameter description.

Installing the Chart

To install the chart with the release name my-release, create your Sumo Logic HTTP Collector and run:

$ helm install --name my-release \
    --set sumologic.collectorUrl=YOUR-URL-HERE stable/sumologic-fluentd

After a few minutes, you should see logs available in Sumo Logic.

Tip: List all releases using helm list

Uninstalling the Chart

To uninstall/delete the my-release deployment:

$ helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The following table lists the configurable parameters of the sumologic-fluentd chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

$ helm install --name my-release \
    --set sumologic.collectorUrl=YOUR-URL-HERE \
    stable/sumologic-fluentd

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install --name my-release -f values.yaml stable/sumologic-fluentd

Tip: You can use the default values.yaml

Persistence

By default, the fluentd position files will be written to an ephemeral emptyDir. Each time the pods die, new position files will be created, all of the logs in the cluster will be sent to sumologic again. To avoid unnecessary re-transmissions, pos directories can be maintained as a hostPath. Create a directory, on each of the nodes, and point persistence.hostPath at that directory.

$ helm install --name my-release \
    --set sumologic.collectorUrl=URL,persistence.hostPath=/var/run/fluentd \
    stable/sumologic-fluentd

RBAC

By default the chart will not install the associated RBAC rolebinding, using beta annotations.

To determine if your cluster supports this running the following:

$ kubectl api-versions | grep rbac

You also need to have the following parameter on the api server. See the following document for how to enable RBAC

--authorization-mode=RBAC

If the output contains "beta" or both "alpha" and "beta" you can enable rbac.

Enable RBAC role/rolebinding creation

To enable the creation of RBAC resources, do the following

$ helm install --name my-release stable/sumologic-fluentd --set rbac.create=true

Excluding and Including data

You have several options controlling the filtering of data that gets sent to Sumo Logic.

Excluding data using environment variables

There are several environment variables that can exclude data. The following table show which environment variables affect which Fluentd sources.

Excluding data using annotations

You can also use the sumologic.com/exclude annotation to exclude data from Sumo. This data is sent to FluentD, but not to Sumo Logic.

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    app: mywebsite
  template:
    metadata:
      name: nginx
      labels:
        app: mywebsite
      annotations:
        sumologic.com/format: "text"
        sumologic.com/sourceCategory: "mywebsite/nginx"
        sumologic.com/sourceName: "mywebsite_nginx"
        sumologic.com/exclude: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Include excluded using annotations

If you excluded a whole namespace, but still need one or few pods to be still included for shipping to Sumo Logic, you can use the sumologic.com/include annotation to include data to Sumo. It takes precedence over the exclusion described above.

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    app: mywebsite
  template:
    metadata:
      name: nginx
      labels:
        app: mywebsite
      annotations:
        sumologic.com/format: "text"
        sumologic.com/sourceCategory: "mywebsite/nginx"
        sumologic.com/sourceName: "mywebsite_nginx"
        sumologic.com/include: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

FluentD stops processing logs

When dealing with large volumes of data (TB's from what we have seen), FluentD may stop processing logs, but continue to run. This issue seems to be caused by the scalability of the inotify process that is packaged with the FluentD in_tail plugin. If you encounter this situation, setting the ENABLE_STAT_WATCHER to false should resolve this issue.