Documentation/contributor-guide/prow_jobs.md
Prow is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. Prow provides GitHub automation through policy enforcement and chat-ops via /command interactions on pull requests (e.g., /test, /approve, /retest), enabling contributors to trigger jobs and manage workflows directly from GitHub comments.
When a user comments /ok-to-testor /retest, on a Pull Request, GitHub sends a webhook to Prow's Kubernetes cluster. Visit this site to further understand the lifecycle of a Prow job.
This is where you can find all etcd Prow jobs status
etcd's CI is managed by kubernetes/test-infra, running Prow.
When a pull request is submitted, or a /command is issued, the CI of etcd which managed by kubernetes/test-infra uses Prow to run the tests. You can view all supported Prow commands.
The jobs configuration for etcd. Please see ProwJob docs for more info.
There are 3 different job types:
As an example, pull-etcd-e2e-amd64 is one of the presubmits. The job automatically runs end-to-end (e2e) tests on the amd64 architecture for every pull request to the etcd repository targeting the main, release-3.6, release-3.5, or release-3.4 branches. This is an example to its dashboard result graph.
Refer to the test-infra Job Types documentation to learn more about them.
These tests can be triggered when you leave a comment, like /ok-to-test (only triggered by an etcd-io member) or /retest, in PR example. /ok-to-test allows Prow to run tests on a pull request from a first-time contributor. /retest tells Prow to rerun any failed or flaky joobs on the pull request, useful if a previous test failed due to a transient issue.
You can find all supported commands.
Test-infra's Prow exposes Grafana dashboards to provide visibility into build resource usage (CPU, memory, number of running builds, etc.) for the Prow build cluster’s Kubernetes jobs. It is scoped via organization, repository, build identifier and time range filters.
It is useful for a few reasons:
Tuning resources: By drilling into each build-run, you can determine realistic memory & CPU requests and limits for that job‑type. This helps avoid waste or avoid failed builds hitting resource limits.
Spotting anomalies: If one build suddenly used 8 GiB while normally this job uses 1 GiB, it may indicate a regression or mis‑configuration.
Capacity planning: Seeing typical and peak usage helps cluster operators plan node sizes, scheduling, concurrency of builds, etc.
Debugging performance issues: A build with unexpectedly high CPU or memory might be stuck, looping, or consuming resources inefficiently.
Shows the number of builds that are in Running vs Pending states over time. Use it to track build backlog or concurrency — e.g., if the “Pending” line rises, builds may be waiting for resources. If the “Running” line fluctuates a lot or remains at some steady value, you can infer how many builds typically run in parallel.
Shows memory usage over time for each build ID (each build listed in the legend at the bottom). The y‑axis shows memory use (e.g., in MiB / GiB). Use this to spot builds with unusually high memory usage — a spike indicates one build consumed many resources.
Similar to the memory panel but shows CPU usage per build over time. Spikes in CPU usage may indicate heavy compute jobs, inefficiencies, or need for resource tuning.
Green line (“used”): how much memory this build’s pod was using at each time point. Orange/Yellow line (“requested”): how much memory was requested (i.e., Kubernetes requests.memory) for that pod. Red line (“limit”): how much memory was limited (i.e., Kubernetes limits.memory) for that pod. Y‑axis: shows memory (GiB, MiB) over the build runtime.
X‑axis: time of day/date. If the green “used” line is close to or hits the red “limit”, it means the build came close to its memory cap (risking OOM). If “used” is much lower than “requested”, you may be over‑allocating memory (waste). If the “requested” line is much higher than “used”, it suggests the job’s request could be tuned downward.
Similar structure: green = actual usage, orange/yellow = requested CPU, red = CPU limit (if set). Y‑axis often in number of CPU cores or fraction thereof (e.g., 1.0 = one core). A green line with spikes may show bursts of CPU usage (e.g., build or compile phases) while idle periods show low usage. If CPU usage consistently saturates the limit, the job may be throttled or delayed. If usage is consistently far below request, tuning may reduce cost.
Static check:
Tests:
Robustness:
Integration:
Some Prow components expose Prometheus metrics that can be used for monitoring and alerting. You can find metrics like the number of PRs in each Tide pool, a histogram of the number of PRs in each merge and various other metrics to this site.