docs/releases/1.36-NOTES.md
⚠ kOps 1.36 has not been released yet! ⚠
This is a document to gather the release notes prior to the release.
kOps 1.36 adds Kubernetes 1.36 support, completes the move away from the in-tree cloud providers, refreshes the bundled components (containerd v2.2, etcd-manager, AWS Load Balancer Controller, EBS CSI driver, Cilium options, CoreDNS, etc.), reworks how the kops-channels addon manager is built and deployed, introduces a hybrid bootstrap path for gossip clusters, expands Azure and Hetzner support, and lays the groundwork for Linode (Akamai) as a new cloud provider.
kube-apiserver: align deleteCollectionWorkers, MaxRequestInflight and compactionInterval with kube-up defaults for large clusters (#18081, #18086)kubelet: migrate deprecated CLI flags into the kubelet configuration file passed via --config (#18280)kubelet: add new tunables (kubeAPIQPS/kubeAPIBurst, nodeAllocatableUpdatePeriodSeconds and friends) and only set nodeAllocatableUpdatePeriodSeconds on Kubernetes 1.35+ (#18085, #18153, #18305)kube-proxy: bind-mount the kubeconfig directory instead of the file so kubeconfig rotations survive (#18344)kube-scheduler: documentation clarification for KubeSchedulerConfiguration (#18151)kops-controller: validate instance group names (#18391)kops-controller: delete the legacy node reconciler and route DigitalOcean through the non-legacy Identifier (#18227)spec.containerd.configAdditions on a cluster or instance group must update those entries to match the TOML v3 schema before upgrading._default directory (#18291)ctr import from nodeup and drop the containerized-mounter Archive task (#18278, #18277)kops toolbox dump for troubleshooting (#18313)cilium-operator on control-plane nodes (#18375)cilium-config entries via Cilium.ExtraConfig (#18285)EnableHostFirewall field to CiliumNetworkingSpec (#18152)bpf-lb-sock and bpf-lb-sock-hostns-only flags (#18375)k8s-connectivity in the liveness probe (#18237)priorityClassName configurable and default Provider when ExternalDNS is partially set (#18298, #18302)kOps now supports running workloads under gVisor (runsc) as a containerd runtime handler. When enabled, kOps installs the runsc runtime via apt, configures containerd with a runsc runtime handler, and creates a gvisor RuntimeClass. Pods targeting the RuntimeClass are automatically scheduled to nodes with gVisor enabled via the kops.k8s.io/gvisor node label.
gVisor is configured per worker instance group. Cluster-level gVisor configuration is rejected, and instance group-level configuration is accepted only for workers:
spec:
containerd:
gvisor:
enabled: true
platform: systrap # default; also supports kvm
This feature is currently supported on Debian-family distributions only (Debian, Ubuntu).
cloud-provider-aws dependency from kOps (#18336)ALBTargetControlConfig CRD (#18222, #18233)elasticloadbalancing:SetRulePriorities and ec2:DescribeSubnets permissions (#17999)MutableCSINodeAllocatableCount for Kubernetes 1.35+ (#18220)kops-controller is now served over HTTPS with a /healthz target on the API NLB target group (#18236, #18174)NodeTerminationHandler:
onDemandAllocationStrategy to the ASG and propagate taints without value as ASG tags (#18342, #18343)instanceRequirements: add excludedInstanceTypes and fix a memory-assignment bug (#18113, #18123)nm-cloud-setup on RHEL 9 for AWS VPC CNI (#18264)HeadBucket to resolve the S3 bucket region, pass the VFS scheme and provider-specific options to the S3 client, and silence warnings when the S3 provider has no supported checksum (#18335, #18129, #18132, #18128)az role assignment delete or the cluster is deleted.cloud-controller-manager for node lifecycle and load-balancer support (#18197)/etc/kubernetes/azure.json for the cloud config, load CCM/CSI config from a Secret via a new azure-cloud-config addon and stop writing the cloud-config file on nodes (#18345)azureblob:// URLs (#18260)RoleAssignment after the VM scale set, handle missing resource groups in disk listing, fix nil pointer dereferences and ordering issues (#18196, #18186, #18191, #18184, #18185)LoadBalancer task dependencies on PublicIP, and use a larger default VM SKU (Standard_D4ls_v6) in tests (#18157, #18154, #18194)regenerate.sh for addons (#18225)kops-controller to the GCE internal load balancer and expose it on the internal LB for gossip clusters (#18169, #18307)kops-controller on GCE (#18171)role=apiserver on GCE with dns=none and fix instance tags for role=apiserver (#18159, #18175)InstanceManagers/InstanceTemplates deletion to complete and include MIG scaling errors when instances are not found (#18013, #18247)N4A instance type (#18330)cloud-provider-gcp dependency and switch clouddns to a forked gcetokensource (#18274)apiserver role e2e template, move it to dns=none, add e2e templates that combine internal load balancer + Cilium etcd, and reject role=apiserver + dns=none only on GCE when not supported (#18147, #18146, #18162, #18214)hcloud/node-group label, which the Cluster Autoscaler Hetzner provider uses for node group membership. Servers created by earlier kOps versions receive the label when they are replaced.minSize of an instance group no longer removes surplus servers; minSize now acts as a floor. Remove specific servers with kops delete instance, or rely on Cluster Autoscaler scale-down.hcloud-cloud-controller-manager to v1.31.0 (#18281, #18317)hcloud-csi-driver to v2.20.2 and reorder the CSI driver Deployment before the DaemonSet (#18318)s-2vcpu-4gb-amd (#18227)Identifier interface and tag droplets with the instance group role (#18227)floatingip TODO from kOps 1.21 (#18314)EtcdEvents HTTP from main etcd cluster traffic in scalability tests (#18370)kops-channels addon manager is now built as a kops-managed image and rendered as a static pod on control-plane nodes; protokube no longer applies channels or labels control-plane nodes (#18215, #18328, #18373, #18374)kops-channels gains --node-name, --interval, multi-URL apply support, and quicker retries until the first reconcile succeeds (#18328)kops-channels: move the node labeler from protokube (#18215)kops-channels: fix region detection and the discovery cache permission noise (#18390)channels binary from the kOps release artifacts (#18374)addons: render addons as fi-tasks so addon templates can reference the finalized task graph, and drop the legacy 9.99.0 version shim and the deprecated master nodeAffinity term (#18215, #18257)dns=none and logs a deprecation warning if gossip is requested (#18245)nf_tables module and install iptables-nft on RHEL 10+ (#18179)nf_conntrack kernel module on Rocky 9 (#17968)ImageVolume tests on Debian 11 and prevent cloud-ifupdown-helper from hijacking CNI veths on Debian 11 (#18261)--node-os-arch=arm64 for Rocky 10 (#18192)gcr.io/distroless/static as the base image and strip release binaries by default (#18403, #18263)kops toolbox; switch to a forked helmstrvals (#18272)structured-merge-diff from v4 to v6 (#18273)hashicorp/memberlist to v0.5.4 (#18230)kops reconcile cluster accepts --use-kubeconfig to reuse an existing kubeconfig instead of regenerating it (#18126)kops accepts --node-volume-type flags on create cluster (#18145)kops create instancegroup aligns the node label across create cluster and create instancegroup (#18341)kops get assets: fix lookup when spec.dnsZone is a DNS name (#18384)kops update cluster: reject non-http(s) URLs for assets.fileRepository (#18340)kops update cluster: validate assets.fileRepository (#18340)kops upgrade-ab: allow kOps downgrades for upgrade-AB scenarios (#18219)kops toolbox dump: time out per-node log dumping after one minute, improve reliability and skip not-found nodes on GCE (#18349, #18311, #18049)nodeup: add experimental hybrid-bootstrap workers, skip protokube/channels assets on workers, populate DefaultMachineType for Cilium-ENI clusters, and use shared system-component env vars for kops-channels (#18245, #18358, #18365)nodeup: fix protokube skip on hybrid-bootstrap workers (#18378)kops-controller/nodeup: ensure files have the desired permissions before close and rename, and fix PrivateKey.WriteTo returning zero length (#18379)dns-controller: honor klog -stderrthreshold even when -logtostderr is true (#18231)HasHighlyAvailableControlPlane to use AllInstanceGroups when an instance-group filter is in use (#17740)kops panic on send to a closed results channel (#18326)AssetBuilder is now concurrency-safe (#18181)KOPS_BASE_URL image version (#18200)fs.inotify.max_user_watches sysctl setting (#17556)actions/upload-artifact, actions/setup-go and actions/dependency-review-action to their latest releases, pinned by commit SHA (#18114)shipbot with a gh-based script for promoting binaries (#18095)kops-channels image build and CI push, and run make apimachinery updates as needed (#18328)gomod: tidy and verify all modules dynamically (#18401)tests/ai-conformance covering accelerator metrics, gang scheduling with Kueue, observability and AI-service metrics, pod and cluster autoscaling, GPU operator integration with Cilium gateway API, and more. The suite is intended for kOps CI/E2E and is not yet exposed as a user-facing feature (#18056, #18075, #18055, #18053, #18100, #18112, #18063, #18064, #18071, #18074, #18073, #17992, #18001, #18005, #18006, #18008, #18009, #18018, #18066, #18067, #18076, #18077, #18084, #18087, #18101, #18108)channels binary is no longer distributed; the kops-channels addon manager now runs as a static pod on control-plane nodes (#18374)9.99.0 addons-version shim has been removed; addons set up by kOps versions prior to 1.22 must be re-applied before upgrading (#18257)cloud-provider-aws and cloud-provider-gcp dependencies have been dropped from kOps; external cloud providers are required (already mandatory for Kubernetes 1.33+) (#18336, #18274)azureblob://{container}/{key} URL form (paired with the AZURE_STORAGE_ACCOUNT environment variable) is no longer accepted; state-store paths must use the new azureblob://{account}/{container}/{key} form (#18260)spec.containerd.configAdditions must update those entries to the containerd config TOML v3 schema before upgrading to kOps 1.36 (#18291)authorization field omitted (kops create -f / kops replace -f) now defaults to RBAC instead of AlwaysAllow, matching kops create cluster. This takes effect when such a manifest is (re-)applied; clusters whose stored spec already sets authorization are unchanged. Set spec.authorization.alwaysAllow: {} to keep AlwaysAllow.Support for Kubernetes version 1.30 is removed in kOps 1.36.
Support for Kubernetes version 1.31 is deprecated and will be removed in kOps 1.37.
Support for Amazon Linux 2, Ubuntu 20.04 and Debian 10 is removed in kOps 1.36.
Support for etcd 3.4 is removed in kOps 1.36.
The standalone channels binary is no longer distributed in kOps 1.36; kops-channels runs as a static pod on the control plane.
Support for AWS Classic Load Balancer (CLB) for the API, deprecated since kOps 1.26, will be rejected for new clusters in kOps 1.37 and fully removed (existing clusters must migrate) in kOps 1.38. See the CLB to NLB migration guide for the upgrade procedure.
Support for gossip DNS is deprecated. kOps 1.37 will reject new gossip DNS clusters, and kOps 1.38 will require existing gossip DNS clusters to migrate before upgrading. This affects clusters whose name ends in .k8s.local and that were not created with --dns=none; clusters using --dns=none, even with a .k8s.local name, are not affected. Existing gossip DNS clusters should migrate to --dns=none or a hosted DNS zone. kOps 1.36 introduces hybrid bootstrap to make that migration easier.