E2E Failure Troubleshooting Guide

This guide describes how to investigate kops E2E CI job failures, identify root causes, and find correlations across groups of failing jobs.

E2E Job Structure

Job Naming Conventions

Periodic jobs follow these naming patterns:

• e2e-kops-grid-{networking}-{distro}-k{k8s_version}-ko{kops_version} — grid matrix jobs testing combinations of networking, distro, and version
• e2e-kops-{cloud}-{feature/scenario} — feature-specific tests (e.g., load balancer controller, nftables, upgrades)
• ci-kubernetes-kops-{cloud}-{scenario} — upstream Kubernetes CI jobs that use kops

Presubmit jobs use the pattern:

• pull-kops-{description} — triggered by pull requests

Job Metadata Dimensions

Each job is annotated with test.kops.k8s.io/* keys in its Prow config that describe its test parameters:

These dimensions are critical for correlation analysis (see below).

Job Definitions

Job configs are auto-generated YAML files located in the test-infra repository at config/jobs/kubernetes/kops/:

• kops-periodics-*.yaml — periodic job definitions (grid, conformance, distros, gce, misc, network-plugins, nftables, pipeline, upgrades, versions, ai-conformance)
• kops-presubmits*.yaml — presubmit job definitions (main, ai-conformance, branch, distros, e2e, network-plugins, scale)
• kops-presets*.yaml — shared credential and config presets

Periodic vs Presubmit Jobs

Artifact Locations

GCS Bucket

All job artifacts are stored in the kubernetes-ci-logs GCS bucket.

Finding the latest build:

https://storage.googleapis.com/kubernetes-ci-logs/logs/{job-name}/latest-build.txt

Build artifacts base URL:

# Periodic jobs:
https://storage.googleapis.com/kubernetes-ci-logs/logs/{job-name}/{build-id}/

# Presubmit (PR) jobs:
https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/directory/{job-name}/{build-id}/

Prow UI

Each build is viewable at:

# Periodic:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/{job-name}/{build-id}

# Presubmit:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/{job-name}/{build-id}

Testgrid Dashboards

• Periodic jobs: https://testgrid.k8s.io/sig-cluster-lifecycle-kops
• Presubmit jobs: https://testgrid.k8s.io/kops-presubmits

Failures Dashboard

The failures dashboard at https://storage.googleapis.com/k8s-metrics/failures-latest.html tracks all consecutively failing Prow jobs. It loads data from failures-latest.json in the same bucket. Each entry contains the job name and consecutive failing days. The dashboard classifies severity as: Critical (365+ days), Severe (180-364), Warning (90-179), Moderate (30-89), Low (1-29).

Artifact Structure

Each build directory contains the following hierarchy:

{build-id}/
├── build-log.txt              # Overall test runner stdout/stderr
├── finished.json              # Build result: {"result": "SUCCESS"|"FAILURE", ...}
├── started.json               # Build start metadata
└── artifacts/
    ├── junit_*.xml            # JUnit test results
    ├── toolbox-dump.yaml      # kops toolbox dump output
    ├── cluster.yaml           # kops get cluster -o yaml
    ├── instancegroups.yaml    # kops get instancegroups -o yaml
    ├── cluster-info/          # Kubernetes API resource dumps and pod logs
    │   ├── nodes.yaml         # All Node objects (status, conditions, capacity, addresses)
    │   ├── {resource}.yaml    # Other cluster-scoped resources (namespaces, clusterroles, etc.)
    │   └── {namespace}/       # Per-namespace resources and pod logs
    │       ├── events.yaml    # Events in this namespace (warnings, errors, scheduling decisions)
    │       ├── pods.yaml      # All Pod objects in this namespace
    │       ├── services.yaml  # All Services
    │       ├── deployments.apps.yaml  # Deployments, DaemonSets, etc.
    │       ├── configmaps.yaml
    │       └── {pod-name}/    # Container logs for each pod
    │           └── {container-name}.log        # Current container logs
    │           └── {container-name}.previous.log  # Previous (crashed) container logs
    └── {node-name}/           # Per-node directory (one per dumped VM)
        ├── journal.log        # Full systemd journal (journalctl --output=short-precise)
        ├── kern.log           # Kernel log (journalctl -k)
        ├── kubelet.log        # Kubelet service log
        ├── containerd.log     # Container runtime log
        ├── kube-proxy.log     # kube-proxy log
        ├── kops-configuration.log  # kops node configuration service (nodeup)
        ├── protokube.log      # protokube service
        ├── node-problem-detector.log
        ├── docker.log         # Docker service (if applicable)
        │
        │   # Control plane nodes only:
        ├── kube-apiserver.log
        ├── kube-controller-manager.log
        ├── kube-scheduler.log (from /var/log/)
        ├── etcd.log
        ├── etcd-events.log
        ├── etcd-cilium.log    # (if Cilium networking)
        ├── kops-controller.log
        │
        │   # Networking state:
        ├── iptables-nat.log   # iptables -t nat --list-rules
        ├── iptables-filter.log # iptables -t filter --list-rules
        ├── nftables-ruleset.log # nft list ruleset
        ├── ip-routes.log      # ip route show table all
        ├── ip-rules.log       # ip rule list
        ├── ip-link.log        # ip -s link
        ├── netstat.log        # ss -s (socket statistics)
        │
        │   # System state:
        ├── etchosts           # /etc/hosts
        ├── sysctls            # sysctl -a
        ├── kubelet.conf       # /var/lib/kubelet/kubelet.conf
        ├── modules            # /proc/modules (loaded kernel modules)
        ├── cloud-init-output.log
        │
        │   # AWS-specific (if applicable):
        ├── aws-routed-eni_ipamd.log   # AWS VPC CNI IPAMD
        ├── aws-routed-eni_plugin.log  # AWS VPC CNI plugin
        │
        │   # Pod logs collected via kubectl:
        ├── external-dns.log   # external-dns pod logs
        └── dns-controller.log # dns-controller pod logs

Node Dump Priority

Nodes are dumped in this priority order (from pkg/dump/dumper.go):

1. Control plane nodes — always dumped first
2. Unregistered nodes — cloud instances that didn't join the Kubernetes cluster (indicates bootstrap failure)
3. Regular worker nodes — up to the --max-nodes limit

Failure Categories

Failures can be classified by examining JUnit XML results:

build — Infrastructure failure

No JUnit test results found at all. The test infrastructure failed before any tests ran — the build, compilation, or cluster setup tooling broke before producing test output.

How to identify: No junit_*.xml files in the artifacts directory.

cluster_up — Cluster creation failed

JUnit contains a failure for a test named "Up" or a test name containing "cluster". The kops cluster failed to create or validate.

How to identify: Parse JUnit XML, look for <testcase> elements with <failure> children where the test name is "Up" or contains "cluster".

e2e_test — Individual test failures

The cluster came up successfully but specific e2e/conformance/feature tests failed. This is the most common failure mode for grid jobs.

How to identify: JUnit XML contains test failures for specific test cases that are not cluster creation.

Diagnostic Process

Step 1: Identify the failure type

1. Fetch finished.json to confirm the build failed
2. Fetch JUnit XML files (artifacts/junit_*.xml) and parse for <failure> elements
3. Classify the failure as build, cluster_up, or e2e_test

Step 2: Read the build log

Start with build-log.txt — this is the overall test runner output and shows the high-level flow of what happened. Search for error messages, timeouts, and validation failures.

Step 3: Check Kubernetes API resource dumps

The cluster-info/ directory contains dumps of all Kubernetes API objects. These are essential for understanding cluster state at the time of failure:

• cluster-info/nodes.yaml — Check Node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable). Nodes showing NotReady or NetworkUnavailable conditions point to kubelet or CNI issues. Also check node addresses and capacity.
• cluster-info/{namespace}/events.yaml — Check cluster-info/kube-system/events.yaml first for system-level issues, then other namespaces as needed. Search for Warning events that reveal scheduling failures, image pull errors, failed mounts, unhealthy pods, and node lifecycle issues. Look for patterns like FailedScheduling, FailedCreatePodSandBox, Unhealthy, BackOff, NodeNotReady.
• cluster-info/kube-system/pods.yaml — Check the status of system pods. Look for pods not in Running phase, containers in CrashLoopBackOff or Waiting state, and pods with restart counts > 0.

Step 4: Check pod container logs

Pod container logs in cluster-info/{namespace}/{pod-name}/{container-name}.log are critical for diagnosing addon and system component failures:

• CNI plugin pods — For networking failures, check the logs of the CNI addon pods (e.g., cilium, calico, aws-node for VPC CNI, kube-router). These reveal why pod networking may not be functional. Look for errors during initialization, connectivity checks, or IPAM allocation.
• CrashLooping pods — Any pod with CrashLoopBackOff status (visible in pods.yaml) should have its container logs examined. Check both {container}.log (current) and {container}.previous.log (previous crash) for the root cause.
• CoreDNS / kube-dns — DNS failures cascade into many other errors. Check CoreDNS pod logs for upstream resolution failures or crash loops.
• kube-proxy — Networking issues at the Service level may appear in kube-proxy logs.
• Cloud controller manager — Node registration and load balancer issues may appear in cloud-controller-manager logs.

Step 5: Check node-level logs

Use this priority order for investigating per-node logs in the {node-name}/ directories:

1. journal.log — Full systemd journal with kubelet, networking, and service errors
2. kubelet.log — Kubelet-specific issues (pod scheduling, volume mounts, node registration)
3. kube-apiserver.log, kube-controller-manager.log — Control plane issues (control plane nodes only)
4. Other service logs — containerd.log, kube-proxy.log, protokube.log, kops-configuration.log

Step 6: Check networking state

For networking-related failures, examine per-node networking dumps:

• ip-routes.log — Are expected routes present?
• ip-link.log — Are network interfaces up? Are CNI interfaces (e.g., cilium_, cali, eni*) present?
• iptables-nat.log / iptables-filter.log / nftables-ruleset.log — Are firewall rules correct?
• CNI-specific logs on nodes (e.g., aws-routed-eni_ipamd.log for AWS VPC CNI)

Cross-reference node networking state with CNI pod logs from cluster-info/ to correlate node-level symptoms with CNI control plane errors.

Step 7: Check cluster configuration

• cluster.yaml — Was the cluster configured correctly?
• instancegroups.yaml — Are instance groups as expected?
• toolbox-dump.yaml — Cloud resources dump (EC2 instances, security groups, load balancers, VPCs, subnets)

Error Signal Patterns

When scanning log files, search for these error patterns to quickly identify root causes:

For each signal found, note the source file and surrounding context (2-3 lines before and after the match) to understand the root cause.

Correlation Analysis

To identify systemic issues affecting multiple jobs, group failing jobs by their metadata dimensions and look for shared failure patterns.

Process

1. Collect metadata for each failing job from its Prow config annotations (test.kops.k8s.io/*)
2. Group jobs by each dimension (cloud, distro, networking, k8s_version, kops_version)
3. Within each group, count how many jobs share the same failure category and error signal type
4. Flag systemic patterns when 3 or more jobs with the same dimension value share the same failure pattern, especially if they represent more than half the group

Examples of Systemic Patterns

• All jobs with a specific networking plugin failing with network_connectivity errors → likely a bug or breaking change in that CNI plugin
• All jobs with a specific distro failing with config_error → likely a package version incompatibility on that OS
• All jobs with a specific k8s version failing with validation_failure → likely a regression in that Kubernetes release affecting kops
• All jobs with a specific kops version failing → likely a kops regression in that release branch

Cross-referencing with Testgrid

Use Testgrid dashboards to see the historical failure trend:

• Check when failures started to identify the triggering change
• Compare with other jobs in the same dimension to confirm correlation
• Look at the git history around the failure start date for relevant changes

Key Source Files