CloudWatch Logs large query

Overview

This example shows how to use AWS SDKs to perform a query on Amazon CloudWatch Logs and get more than the maximum number of 10,000 logs back.

The CloudWatch Logs API is capped at 10,000 records for requests that read or write. GetLogEvents returns tokens for pagination, but GetQueryResults does not. This example breaks down one query into multiple queries if more than the maximum number of records are returned from the query.

The following components are used in this example:

• Amazon CloudWatch Logs hosts the logs that are queried using the Amazon CloudWatch Logs API.

⚠ Important

• Running this code might result in charges to your AWS account.
• Running the tests might result in charges to your AWS account.
• We recommend that you grant your code least privilege. At most, grant only the minimum permissions required to perform the task. For more information, see Grant least privilege.
• This code is not tested in every AWS Region. For more information, see AWS Regional Services.

Scenario

Prerequisites

For general prerequisites, see the README in the python folder.

To run this example, you need a CloudWatch log group that contains over 10,000 logs. You can create one yourself, or you can follow the steps in the Infrastructure and data section. These steps require you to install or update the latest version of the AWS CLI

Infrastructure and data

Use the following steps to create the necessary resources in AWS CloudFormation and use the AWS CLI to upload the necessary logs.

1. In your local terminal, change directories to resources.
2. Run aws cloudformation deploy --template-file stack.yaml --stack-name CloudWatchLargeQuery
3. Run ./make-log-files.sh. This will output two timestamps for use in the following step.
4. Run export QUERY_START_DATE=<QUERY_START_DATE>. Replace <QUERY_START_DATE> with the output from the previous step. Repeat this for QUERY_END_DATE.
5. Optional: Run export QUERY_LOG_GROUP=<QUERY_LOG_GROUP>. Replace <QUERY_LOG_GROUP> with your preferred log group.
6. Run ./put-log-events.sh.
7. Wait five minutes for logs to settle and to make sure you're not querying for logs that exist in the future.

Run the scenario

1. python exec.py

Additional reading

• CloudWatch Logs Insights query syntax
• CloudWatch Logs billing and cost

---

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: Apache-2.0