CloudWatch Logs large query

Overview

This example shows how to use AWS SDKs to perform a query on Amazon CloudWatch Logs and get more than the maximum number of 10,000 logs back.

The CloudWatch Logs API is capped at 10,000 records for requests that read or write. This example breaks down one query into multiple queries if more than the maximum number of records are returned from the query.

The following components are used in this example:

Amazon CloudWatch Logs hosts the logs that are queried using the Amazon CloudWatch Logs API.

Supporting infrastructure

An AWS CloudFormation template exists in the resources folder. The template contains two resources: a CloudWatch log group, and a CloudWatch log stream that will be used as the default log group and stream for the queries made in the example. We recommend deploying a stack with this template.

Prerequisites

If you need to, install or update the latest version of the AWS CLI.

Deploy resources

To deploy the stack using the template, run the following command:

aws cloudformation deploy --template-file stack.yaml --stack-name LargeQueryStack

Destroy resources

To destroy the stack, run the following command:

aws cloudformation delete-stack --stack-name LargeQueryStack

Sample logs

A lot of logs are needed to make a robust example. If you happen to have a log group with over 10,000 logs at the ready, great! If not, there are two resources that can help:

Resources

make-log-files.sh will create 50,000 logs and divide them among 5 files of 10,000 logs each (the maximum for each call to 'PutLogEvents'). Two timestamps will output to the console. These timestamps can be used to configure the query. Five minutes of logs, starting at the time of execution, will be created. Wait at least five minutes after running this script before attempting to query.
put-log-events.sh will use the AWS CLI to put the created files from Step 1 into the log group/stream created by the CloudFormation template.

Implementations

This example is implemented in the following languages:

Additional reading

CloudWatch Logs Insights query syntax