ContextQMD
Back to Trivy

Troubleshooting

docs/guide/references/troubleshooting.md

0.70.012.3 KB
Original Source

Troubleshooting

Scan

Timeout

!!! error bash $ trivy image ... ... analyze error: timeout: context deadline exceeded

Your scan may time out. Java takes a particularly long time to scan. Try increasing the value of the --timeout option such as --timeout 15m.

Unable to initialize an image scanner

!!! error bash $ trivy image ... ... 2024-01-19T08:15:33.288Z FATAL image scan error: scan error: unable to initialize a scanner: unable to initialize an image scanner: 4 errors occurred: * docker error: unable to inspect the image (ContainerImageName): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? * containerd error: containerd socket not found: /run/containerd/containerd.sock * podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory * remote error: GET https://index.docker.io/v2/ContainerImageName: MANIFEST_UNKNOWN: manifest unknown; unknown tag=0.1

It means Trivy is unable to find the container image in the following places:

  • Docker Engine
  • containerd
  • Podman
  • A remote registry

Please see error messages for details of each error.

Common mistakes include the following, depending on where you are pulling images from:

Common

  • Typos in the image name
    • Common mistake :)
  • Forgetting to specify the registry
    • By default, it is considered to be Docker Hub ( index.docker.io ).

Docker Engine

  • Incorrect Docker host
    • If the Docker daemon's socket path is not /var/run/docker.sock, you need to specify the --docker-host flag or the DOCKER_HOST environment variable. The same applies when using TCP; you must specify the correct host address.

containerd

  • Incorrect containerd address
    • If you are using a non-default path, you need to specify the CONTAINERD_ADDRESS environment variable. Please refer to this documentation.
  • Incorrect namespace
    • If you are using a non-default namespace, you need to specify the CONTAINERD_NAMESPACE environment variable. Please refer to this documentation.

Podman

  • Podman socket configuration

Container Registry

  • Unauthenticated
    • If you are using a private container registry, you need to authenticate. Please refer to this documentation.
  • Using a proxy
    • If you are using a proxy within your network, you need to correctly set the HTTP_PROXY, HTTPS_PROXY, etc., environment variables.
  • Use of a self-signed certificate in the registry
    • Because certificate verification will fail, you need to either trust that certificate or use the --insecure flag (not recommended in production).

Certification

!!! error Error: x509: certificate signed by unknown authority

TRIVY_INSECURE can be used to allow insecure connections to a container registry when using SSL.

$ TRIVY_INSECURE=true trivy image [YOUR_IMAGE]

If you need to trust a custom CA certificate, you can provide a PEM-encoded bundle.

=== "Unix (except macOS)"

You can specify the location of your certificate using the `SSL_CERT_FILE` or `SSL_CERT_DIR` environment variables.

```bash
$ SSL_CERT_FILE=/path/to/ca.pem trivy image [YOUR_IMAGE]
```

```bash
$ SSL_CERT_DIR=/path/to/certs trivy image [YOUR_IMAGE]
```

=== "All systems"

Use the `--cacert` flag to point Trivy to a PEM-encoded CA certificate file, regardless of the operating system.

```bash
$ trivy image --cacert /path/to/ca.pem [YOUR_IMAGE]
```

GitHub Rate limiting

Trivy uses GitHub API for VEX repositories.

!!! error bash $ trivy image --vex repo ... ... API rate limit exceeded for xxx.xxx.xxx.xxx.

Specify GITHUB_TOKEN for authentication

$ GITHUB_TOKEN=XXXXXXXXXX trivy image --vex repo [YOUR_IMAGE]

!!! note GITHUB_TOKEN doesn't help with the rate limit for the vulnerability database and other assets. See https://github.com/aquasecurity/trivy/discussions/8009

Unable to open JAR files

!!! error bash $ trivy image ... ... failed to analyze file: failed to analyze usr/lib/jvm/java-1.8-openjdk/lib/tools.jar: unable to open usr/lib/jvm/java-1.8-openjdk/lib/tools.jar: failed to open: unable to read the file: stream error: stream ID 9; PROTOCOL_ERROR; received from peer

Currently, we're investigating this issue. As a temporary mitigation, you may be able to avoid this issue by downloading the Java DB in advance.

shell
$ trivy image --download-java-db-only
2023-02-01T16:57:04.322+0900    INFO    Downloading the Java DB...
$ trivy image [YOUR_JAVA_IMAGE]

Cache lock errors

!!! error cache may be in use by another process

Trivy's vulnerability database is opened in read-only mode, so it does not cause lock issues. Lock errors occur only when using filesystem cache for scan cache storage.

Filesystem cache uses BoltDB internally, which creates file locks to prevent data corruption. As stated in the BoltDB documentation:

Please note that Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it.

Reference: BoltDB README

If you're using memory cache (default for some commands like fs, rootfs, config, and sbom) or external cache (Redis), you will not encounter lock errors. Lock issues only occur when using filesystem cache with multiple concurrent processes. See Cache Backend for more details.

These errors occur when:

  • Multiple Trivy processes try to use the same filesystem cache directory simultaneously
  • A previous Trivy process did not shut down cleanly
  • Trivy server is running with filesystem cache and holding a lock on the cache

Solutions

Solution 1: Use memory cache or Redis cache (Recommended)

Memory cache is the default for some commands (e.g., fs, rootfs, config, sbom). For other commands like image scanning, you can use --cache-backend memory to enable concurrent execution:

bash
$ trivy image --cache-backend memory debian:11 &
$ trivy image --cache-backend memory debian:12 &

Note that memory cache does not persist scan results, so subsequent scans will take longer as layers need to be scanned again each time.

For server mode or persistent cache with concurrent access, use Redis cache:

bash
$ trivy server --cache-backend redis://localhost:6379

Solution 2: Terminate conflicting processes

If you need to use filesystem cache, check for running Trivy processes and terminate them:

bash
$ ps aux | grep trivy
$ kill [process_id]

Solution 3: Use different cache directories

If you must run multiple Trivy processes with filesystem cache, specify different cache directories for each process:

bash
$ trivy image --cache-dir /tmp/trivy-cache-1 debian:11 &
$ trivy image --cache-dir /tmp/trivy-cache-2 debian:12 &

Note that each cache directory will download its own copy of the vulnerability database and other scan assets, which will increase network traffic and storage usage.

Multiple Trivy servers

!!! error $ trivy image --server http://xxx.com:xxxx test-image ... - twirp error internal: failed scan, test-image: failed to apply layers: layer cache missing: sha256:***** To run multiple Trivy servers, you need to use Redis as the cache backend so that those servers can share the cache. Follow this instruction to do so.

Problems with /tmp on remote Git repository scans

!!! error FATAL repository scan error: scan error: unable to initialize a scanner: unable to initialize a filesystem scanner: git clone error: write /tmp/fanal-remote...

Trivy clones remote Git repositories under the /tmp directory before scanning them. If /tmp doesn't work for you, you can change it by setting the TMPDIR environment variable.

Try:

$ TMPDIR=/my/custom/path trivy repo ...

Running out of space during image scans

!!! error bash image scan failed: failed to copy the image: write /tmp/fanal-3323732142: no space left on device

Trivy uses a temporary directory during image scans. The directory path would be determined as follows:

  • On Unix systems: Use $TMPDIR if non-empty, else /tmp.
  • On Windows: Uses GetTempPath, returning the first non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory.

See this documentation for more details.

If the image is large or the temporary directory has insufficient space, the scan will fail. You can configure the directory path to redirect Trivy to a directory with adequate storage. On Unix systems, you can set the $TMPDIR environment variable.

$ TMPDIR=/my/custom/path trivy image ...

When scanning images from a container registry, Trivy processes each layer by streaming, loading only the necessary files for the scan into memory and discarding unnecessary files. If a layer contains large files that are necessary for the scan (such as JAR files or binary files), Trivy saves them to a temporary directory (e.g. $TMPDIR) on local storage to avoid increased memory consumption. Although these files are deleted after the scan is complete, they can temporarily increase disk consumption and potentially exhaust storage. In such cases, there are currently three workarounds:

  1. Use a temporary directory with sufficient capacity

    This is the same as explained above.

  2. Specify a small value for --parallel

    By default, multiple layers are processed in parallel. If each layer contains large files, disk space may be consumed rapidly. By specifying a small value such as --parallel 1, parallelism is reduced, which can mitigate the issue.

  3. Specify --skip-files or --skip-dirs

    If the container image contains large files that do not need to be scanned, you can skip their processing by specifying --skip-files or --skip-dirs. For more details, please refer to this documentation.

DB

Old DB schema

!!! error --skip-update cannot be specified with the old DB schema.

Trivy v0.23.0 or later requires Trivy DB v2. Please update your local database or follow the instruction of air-gapped environment.

Error downloading vulnerability DB

!!! error FATAL failed to download vulnerability DB

If Trivy is running behind corporate firewall, refer to the necessary connectivity requirements as described here.

Denied

!!! error GET https://ghcr.io/token?scope=repository%3Aaquasecurity%2Ftrivy-db%3Apull&service=ghcr.io: DENIED: denied

Your local GHCR (GitHub Container Registry) token might be expired. Please remove the token and try downloading the DB again.

shell
docker logout ghcr.io

or

shell
unset GITHUB_TOKEN

Debugging

HTTP Request/Response Tracing

For debugging network issues, connection problems, or authentication failures, you can enable HTTP request/response tracing using the --trace-http flag.

!!! danger "Security Warning" While Trivy attempts to redact known sensitive information such as authentication headers and common secrets, the --trace-http flag may still expose sensitive data in HTTP requests and responses.

**Never use this flag in production environments or CI/CD pipelines.**
This flag is automatically disabled in CI environments for security.
bash
# Enable HTTP tracing for debugging registry issues
$ trivy image --trace-http registry.example.com/my-image:latest

# HTTP tracing with other debugging options
$ trivy image --trace-http --debug --insecure my-image:tag

Others

Unknown error

Try again after running trivy clean --all:

$ trivy clean --all