Reproducing Crashes from ClusterFuzz

Introduction

This guide provides step-by-step instructions for reproducing crashes found by ClusterFuzz. Reproducing a crash locally is the first step toward debugging and fixing it.

*** note Requirements: For Windows, you must convert the forward slashes (/) to backslashes (\) in the commands below and use set command instead of export to set the environment variable (step 4). Note that these commands are intended to be used with cmd.exe, not PowerShell. Also, you may find these tips on how to debug an ASAN instrumented binary helpful.

[TOC]

Which Workflow Should I Use?

The first step is to identify how the crash was found. Look at the ClusterFuzz bug report:

• If the report specifies a "Fuzz Target" and mentions a fuzzing engine (like libFuzzer, Centipede, etc.), you should follow the steps under Reproducing a Crash from a Fuzzing Engine.

• Otherwise, the crash was likely caused by a blackbox fuzzer that produced a file to crash a larger application (like content_shell). You should follow the steps under Reproducing a Crash from a Blackbox Fuzzer.

Reproducing a Crash from a Fuzzing Engine

The majority of the bugs reported by ClusterFuzz have the Reproducible label. This workflow applies to those cases.

1. Download the testcase from ClusterFuzz. If you are CCed on an issue filed by ClusterFuzz, a link to it is next to "Reproducer testcase" in the bug description.

   For the rest of this section, we call the path of this file: $TESTCASE_PATH and the fuzz target you want to reproduce a crash on: $FUZZER_NAME (provided as "Fuzz Target" in the bug description).

2. Generate gn build configuration:

   Copy

   gn args out/fuzz
   

   This will open up an editor. Copy the gn configuration parameters from the values provided in GN Config section in the ClusterFuzz testcase report.

3. Build the fuzzer:

   Copy

   autoninja -C out/fuzz $FUZZER_NAME
   

4. Set the *SAN_OPTIONS environment variable as provided in the Crash Stacktrace section in the testcase report. Here is an example value of ASAN_OPTIONS that is similar to its value on ClusterFuzz:

   Copy

   export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0
   

   Note: The sanitizer options may reference suppressions files. You can find those at go/crfuzz-clusterfuzz-suppressions (google-internal).

5. Run the fuzz target with the downloaded testcase. The -runs=100 flag helps ensure reproduction even if the crash has minor flakiness.

   Copy

   out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH
   

File a bug if you run into any issues.

Reproducing a Crash from a Blackbox Fuzzer

1. Get info from the report.

   • Download the Reproducer testcase. We'll call its path $TESTCASE_PATH.
   • Identify the target application from the Job Type (e.g., linux_asan_content_shell_drt means the target is content_shell).

2. Configure and build the application:

   Generate the build configuration using the arguments from the “GN Config” section.

   Copy

   gn args out/fuzz
   

   Build the target application binary (e.g., content_shell).

   Copy

   autoninja -C out/fuzz content_shell
   

3. Set environment variables:

   Export all variables listed under [Environment] in the stack trace of the report.

   Copy

   export ASAN_OPTIONS=...
   # Also export any other variables listed.
   

4. Construct and run the command:

   In the stack trace, find the line beginning with Command line:. Construct your local command by replacing the executable path (e.g., /proc/self/exe or /mnt/.../content_shell) with the path to your built binary (e.g., out/fuzz/content_shell), copying all the flags, and appending the $TESTCASE_PATH at the end.

   Copy

   # Example command based on a content_shell crash
   out/fuzz/content_shell \
   --run-web-tests \
   --disable-in-process-stack-traces \
   --autoplay-policy=no-user-gesture-required \
   --lang=en-US \
   $TESTCASE_PATH
   

Crashes reported as Unreproducible

ClusterFuzz generally does not report issues that it cannot reliably reproduce, unless the following condition is met. If a certain crash is occurring often enough, such a crash might be reported with Unreproducible label and an explicit clarification that there is no convenient way to reproduce it. There are two ways to work with such crashes.

Option 1: Attempt a Speculative Fix

Try a speculative fix based on the stacktrace. Once the fix is landed, wait a couple days and then check Crash Statistics section on the ClusterFuzz testcase report page. If the fix works out, you will see that the crash is not happening anymore. If the crash does not occur again for a little while, ClusterFuzz will automatically close the issue as Verified.

Option 2: Replay the Fuzzing Session (libFuzzer Only)

Try to reproduce the whole fuzzing session. This workflow is very similar to the one described above for reproducing a crash from a fuzzing engine. The only differences are:

• Instead of downloading a single testcase, you need to download corpus backup. This can be done using the following command:

  Copy

  gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip .
  

• Alternatively, you can navigate to the following URL in your browser and download the latest.zip file:

  Copy

  https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME
  

• Create an empty directory and unpack the corpus into it.

• Follow steps 2-4 in the reproducing a crash from a fuzzing engine section above.

• On step 5, use the following command:

  Copy

  out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP
  

• Wait and hope that the fuzzer will crash.

Waiting for a crash to occur may take some time (up to 1hr), but if it happens, you will be able to test the fix locally and/or somehow debug the issue.

Symbolizing stack traces

Stack traces from ASAN builds are not symbolized by default. However, you can symbolize them by piping the output into:

src/tools/valgrind/asan/asan_symbolize.py

Minimizing a crash input (optional)

ClusterFuzz does crash input minimization automatically, and a typical crash report has two testcases available for downloading:

• An original testcase that has triggered the crash;
• A minimized testcase that is smaller than the original but triggers the same crash.

If you would like to further minimize a testcase, run the fuzz target with the two additional arguments:

• -minimize_crash=1
• -exact_artifact_path=<output_filename_for_minimized_testcase>

The full command would be:

out/fuzz/$FUZZER_NAME -minimize_crash=1 -exact_artifact_path=<minimized_testcase_path> $TESTCASE_PATH

This might be useful for large testcases that make it hard to identify a root cause of a crash. You can leave the minimization running locally for a while (e.g. overnight) for better results.