Smart Retry + Spot Preemption handling

Overview

Smart retry handles automatic retries for tasks/tests in two different scenarios:

• When a test or task fails
• When a test or task is cancelled or never run due to spot preemption

The basic idea is that when a Buildkite step fails, or is cancelled due to the spot instance being terminated, we need to retry the step, but skip all of the successful work we did in the previous test.

• If a non-spot failure passes on a retry, it was a flaky failure.
• If a spot failure succeeds on the retry, the original failure can be completely ignored.

There are a lot of edge cases. You could have a scenario like:

• Attempt #1 - Task A and Task B fail before the spot preemption. Task C and Task D are cancelled due to Spot preemption
• Attempt #2 - Task A succeeds, Task B and C are cancelled due to spot preemption
• Attempt #3 - Task B,D suceed and C fails
• Attempt #4 - Task C fails, Task D succeeds

All 4 attempts have to be thought of as a single, combined run to understand what the end result is. The end result that is relevant to the user is:

• Task A and Task B are flaky
• Task C is a real, consistent failure
• Task D is successful

Process

Here's the overall process that makes this work, ignoring spot preemptions for now:

• For any given run, if it is a retry, download the task-status from the previous step, and have Gradle skip all tests/tasks that completed successfully across previous attempts.
• As Gradle executes tasks and tests, keep track of their status (not started, started, finished successfully, finished with an error)
• After everything is finished executing, have Gradle write a task-status file with the final status of all tests and tasks.
• Combine this file with the task-status from the previous attempt, if there was one, so that each attempt has a cumulative view of all the previous attempts.
• Upload this file as a Buildkite artifact (which the next attempt, if there is one, will download)
• Automatically retry this step if there were any failures for tests that haven't executed twice

Pruning granularity

InternalTestRerunPlugin prunes previously successful work at three levels, from coarsest to finest:

• Task: a task listed in successfulTasks is disabled outright with onlyIf(false).
• Suite: a class listed in successfulSuites is excluded with a Class.* filter pattern.
• Individual test: a Class#method reference listed in successfulTests is excluded with a filter pattern for that method.

Individual test pruning needs care for suites run by the randomized runner with a @ParametersFactory, because those report one test per parameter, for example test {yaml=analysis-common/30_tokenizers/letter}. RandomizedRunner#applyFilters keeps a test candidate as soon as either its parameterized description or its bare method description is allowed to run, so excluding only the parameterized name has no effect at all. The plugin therefore emits the bare Class.method pattern in addition to the parameterized one. That does not take the method's other parameters down with it: those still carry a parameterized name that no pattern matches, which is enough to keep them running. MutedTestsBuildService does the same thing for muted-tests.yml, and the two mechanisms contribute patterns to the same filter.

Ordered suites

Individual test pruning assumes tests are independent. Upgrade suites are not: they are parameterized over upgrade phases and an early phase sets up the cluster state a later phase asserts on, so skipping a phase that already passed makes the later phase fail.

Projects hosting such suites opt out in their build.gradle:

smartRetry.pruneIndividualTests.set(false)

Task and suite pruning stay active for those projects, since both either skip a suite in its entirety or not at all.

Spot Preemptions

Here's how the above process additionally handles spot preemptions:

• GCP instances have an internal, HTTP-based metadata server you can query from the instance. There is a metadata item that gets set when the instance has been preempted. 30 seconds after this flag is set, the instance will be terminated.
• Have Gradle poll this metadata item every 1s checking for preemption status
• If the preemption flag flips to true, Gradle should immediately stop executing tests/tasks.
• Mark all tests/tasks that were executing at the time of cancellation as "cancelled", then write the task-status file as usual.
• Tag the buildscan as PREEMPTED and also add a value for the preemption time. Upload the buildscan as normal.
• Combine/upload task-status as normal
• Exit with a special exit code ($GCP_PREEMPTION_EXIT_CODE - currently 47) to mark this step as "Preempted"
• Automatically retry this step based on that exit code

In the subsequent retry, any tests/tasks that didn't fully complete before the preemption (or never ran at all), will execute.

Relevant Source

• build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ci/GcpPreemptionWatchdog.java
  • Polls instance metadata service to watch for preemption status
• build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ci/PreemptionBuildCanceller.java
  • Handles cancelling the build when the instance gets preempted. Cancels running tasks, child processes, etc.
• build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ci/TaskStatusReport.java
  • Writes the task-status report at the end of execution
• build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ci/TaskStatusTrackerPlugin.java
  • Tracks status of tasks as they start/finish
• build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rerun/InternalTestRerunPlugin.java
  • Handles filtering out successful tests/tasks from retry runs
• .buildkite/scripts/smart-retry
  • All of the non-Gradle bits. Downloading previous reports, combining reports, putting together the retry data to pass to Gradle, summarizing overall status at the end of execution for the user.