ContextQMD
Back to Logstash

Tutorial: Extend Elastic {{integrations}} with the {{ls}} elastic_integration filter[ea-integrations-tutorial]

docs/reference/ea-integration-tutorial.md

9.4.07.6 KB
Original Source

Tutorial: Extend Elastic {{integrations}} with the {{ls}} elastic_integration filter[ea-integrations-tutorial]

You can use {{ls}} to transform events collected by {{agent}}s and paired with an Elastic integration.

You get the benefits of Elastic integrations--such as the simplicity of ingesting data from a wide variety of data sources and ensuring compliance with the Elastic Common Schema (ECS)--combined with the extra processing power of {{ls}}.

This new functionality is made possible by the elastic_integration filter plugin. When you include the elastic_integration filter in your configuration, {{ls}} reads certain field values generated by the {{agent}}, and uses them to apply the transformations from Elastic integrations. This ability allows you to further process events in the Logstash pipeline before sending them to their configured destinations.

This tutorial walks you through adding the Crowdstrike integration, and sending the data to {{ech}}, {{serverless-full}}, or self-managed {{es}}.

Prerequisites [ea-integrations-prereqs]

You need:

:::{note} Even though the focus of this tutorial is {{fleet}}-managed agents, you can use the elastic_integration filter and this general approach with self-managed agents. :::

Process overview [ea-integrations-process-overview]

Configure {{fleet}} to send data from {{agent}} to {{ls}} [ea-integrations-fleet]

  1. For {{fleet}}-managed agents, go to {{kib}} and navigate to Fleet > Settings.
  2. Create a new output and specify {{ls}} as the output type.
  3. Add the {{ls}} hosts (domain or IP address/s) that the {{agent}} should send data to.
  4. Add the client SSL certificate and the Client SSL certificate key to the configuration.
  5. Click Save and apply settings in the bottom right-hand corner of the page.

Create an {{agent}} policy with the necessary integrations [ea-integrations-create-policy]

  1. In {{kib}} navigate to Fleet > Agent policies, and select Create agent policy.

  2. Give this policy a name, and then select Advanced options.

  3. Change the Output for integrations setting to the {{ls}} output you created.

  4. Click Create agent policy.

  5. Select the policy name, and click Add integration.

    This step takes you to the {{integrations}} browser, where you can select an integration that has everything necessary to integrate the data source with your other data in the {{stack}}.

    We'll use Crowdstrike as our example in this tutorial.

  6. On the Crowdstrike integration overview page, click Add Crowdstrike to configure the integration.

  7. Configure the integration to collect the data you need. On step 2 at the bottom of the page (Where to add this integration?), make sure that the “Existing hosts” option is selected and the Agent policy selected is the {{ls}} policy that we created for our {{ls}} output. This policy should be selected by default.

  8. Click Save and continue.

    You have the option to add the {{agent}} to your hosts. If you haven't already, install the {{agent}} on the host where you want to collect data.

Configure {{ls}} to use the elastic_integration filter plugin [ea-integrations-pipeline]

  1. Create a new {{ls}} pipeline.
  2. Be sure to include these plugins:

Note that every event sent from the {{agent}} to {{ls}} contains specific meta-fields. {{ls}} expects events to contain a top-level data_stream field with type, dataset, and namespace sub-fields.

{{ls}} uses this information and its connection to {{es}} to determine which integrations to apply to the event before sending the event to its destination output. {{ls}} frequently synchronizes with {{es}} to ensure that it has the most recent versions of the enabled integrations.

Sample configuration: output to Elastic Cloud [ea-integrations-ech-sample]

This sample illustrates using the elastic_agent input and the elastic_integration filter for processing in {{ls}}, and then sending the output to {{ech}} or {{serverless-full}}.

Check out the elastic_integration filter for the full list of configuration options.

txt
input {
  elastic_agent { port => 5055 }
}

filter {
  elastic_integration {
    cloud_id => "your-cloud:id"
    api_key => "your-api-key"
  }
}

output {
  stdout {}
  elasticsearch {
    cloud_id => "your-cloud:id"
    api_key => "your-api-key"
  }
}

All processing occurs in {{ls}} before events are forwarded to {{ech}} or {{serverless-full}}.

Sample configuration: output to self-managed {{es}} [ea-integrations-es-sample]

This sample illustrates using the elastic_agent input and the elastic_integration filter for processing in {{ls}}, and then sending the output to {{es}}.

Check out the elastic_integration filter for the full list of configuration options.

Check out minimum required privileges for more info.

txt
input {
  elastic_agent { port => 5055 }
}

filter {
  elastic_integration {
    hosts => ["{es-host}:9200"]
    ssl_enabled => true
    ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem"
    username => "elastic" 
    password => "changeme"
  }
}

output {
  stdout {
    codec => rubydebug # debugs datastream inputs
  }
  ## add elasticsearch
  elasticsearch {
    hosts => ["{es-host}:9200"]
    user => "elastic"
    password => "changeme"
    ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem"
  }
}

:::{note} The user credentials that you specify in the elastic_integration filter must have sufficient privileges to get information about {{es}} and the integrations that you are using. :::

If your {{agent}} and {{ls}} pipelines are configured correctly, events go to {{ls}} for processing before {{ls}} forwards them on to {{es}}.

If you encounter problems, check out the Troubleshooting section in the elastic_integration filter docs.