• v1.11.10
  • Downloads for v1.11.10
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.9
    • Other notable changes
• v1.11.9
  • Downloads for v1.11.9
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.8
    • Other notable changes
• v1.11.8
  • Downloads for v1.11.8
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.7
    • Other notable changes
• v1.11.7
  • Downloads for v1.11.7
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.6
    • Other notable changes
• v1.11.6
  • Downloads for v1.11.6
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.5
    • Other notable changes
• v1.11.5
  • Downloads for v1.11.5
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.4
    • Other notable changes
• v1.11.4
  • Downloads for v1.11.4
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.3
    • Other notable changes
• v1.11.3
  • Downloads for v1.11.3
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.2
    • Action Required
    • Other notable changes
• v1.11.2
  • Downloads for v1.11.2
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.1
    • Action Required
    • Other notable changes
• v1.11.1
  • Downloads for v1.11.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0
    • Action Required
    • Other notable changes
• v1.11.0
  • Downloads for v1.11.0
    • Client Binaries
    • Server Binaries
    • Node Binaries
• Kubernetes 1.11 Release Notes
  • Urgent Upgrade Notes
    • (No, really, you MUST do this before you upgrade)
  • Major Themes
    • SIG API Machinery
    • SIG Auth
    • SIG CLI
    • SIG Cluster Lifecycle
    • SIG Instrumentation
    • SIG Network
    • SIG Node
    • SIG OpenStack
    • SIG Scheduling
    • SIG Storage
    • SIG Windows
  • Known Issues
  • Before Upgrading
    • New Deprecations
    • Removed Deprecations
    • Graduated to Stable/GA
    • Graduated to Beta
    • New alpha features
  • Other Notable Changes
    • SIG API Machinery
    • SIG Apps
    • SIG Auth
    • SIG Autoscaling
    • SIG Azure
    • SIG CLI
    • SIG Cluster Lifecycle
    • SIG GCP
    • SIG Instrumentation
    • SIG Network
    • SIG Node
    • SIG OpenStack
    • SIG Scheduling
    • SIG Storage
    • SIG vSphere
    • SIG Windows
    • Additional changes
  • External Dependencies
  • Bug Fixes
    • General Fixes and Reliability
  • Non-user-facing changes
• v1.11.0-rc.3
  • Downloads for v1.11.0-rc.3
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0-rc.2
    • Other notable changes
• v1.11.0-rc.2
  • Downloads for v1.11.0-rc.2
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0-rc.1
    • Other notable changes
• v1.11.0-rc.1
  • Downloads for v1.11.0-rc.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0-beta.2
    • Action Required
    • Other notable changes
• v1.11.0-beta.2
  • Downloads for v1.11.0-beta.2
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0-beta.1
    • Action Required
    • Other notable changes
• v1.11.0-beta.1
  • Downloads for v1.11.0-beta.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0-alpha.2
    • Action Required
    • Other notable changes
• v1.11.0-alpha.2
  • Downloads for v1.11.0-alpha.2
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.11.0-alpha.1
    • Other notable changes
• v1.11.0-alpha.1
  • Downloads for v1.11.0-alpha.1
    • Client Binaries
    • Server Binaries
    • Node Binaries
  • Changelog since v1.10.0
    • Action Required
    • Other notable changes

v1.11.10

Documentation

Downloads for v1.11.10

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.9

Other notable changes

• [stackdriver addon] Bump prometheus-to-sd to v0.5.0 to pick up security fixes. (#75362, @serathius)
  • [fluentd-gcp addon] Bump fluentd-gcp-scaler to v0.5.1 to pick up security fixes.
  • [fluentd-gcp addon] Bump event-exporter to v0.2.4 to pick up security fixes.
  • [fluentd-gcp addon] Bump prometheus-to-sd to v0.5.0 to pick up security fixes.
  • [metatada-proxy addon] Bump prometheus-to-sd v0.5.0 to pick up security fixes.
• Ensure the backend pools are set correctly for Azure SLB with multiple backend pools (e.g. outbound rules) (#76691, @feiskyer)
• Increase Azure default maximumLoadBalancerRuleCount to 250. (#72621, @feiskyer)
• Fixes a NPD bug on GCI, so that it disables glog writing to files for log-counter (#76211, @wangzhen127)
• Node-Problem-Detector configuration is now decoupled from the Kubernetes release on GKE/GCE. (#73288, @wangzhen127)
• Restores --username and --password flags to kubectl (#75451, @liggitt)
• Update Cluster Autoscaler version to 1.3.8. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.8 (#75174, @losipiuk)
• Kubelet won't evict a static pod with priority system-node-critical upon resource pressure. (#74222, @Huang-Wei)

v1.11.9

Documentation

Downloads for v1.11.9

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.8

Other notable changes

• Prevent AWS Network Load Balancer security groups ingress rules to be deleted by ensuring target groups are tagged. (#73594, @masterzen)
• Allow disable outbound SNAT when Azure standard load balancer is used together with outbound rules. (#75282, @feiskyer)
• Ensure Azure load balancer cleaned up on 404 or 403 when deleting LoadBalancer services. (#75256, @feiskyer)
• Fix kubelet start failure issue on Azure Stack due to InstanceMetadata setting (#74936, @rjaini)
• Fix panic in kubectl cp command (#75037, @soltysh)
• Fixes an issue with missing apiVersion/kind in object data sent to admission webhooks (#74448, @liggitt)
• fix get azure accounts timeout issue when there is no out-bound IP (#74191, @andyzhangx)
• fix issue: fail to detach azure disk when there is server side error (#74398, @andyzhangx)
• fix parse devicePath issue on Azure Disk (#74499, @andyzhangx)
• fix Azure Container Registry anonymous repo image pull error (#74715, @andyzhangx)
• Add metrics-port to kube-proxy cmd flags. (#72682, @whypro)
• Reduce memory utilization of admission webhook metrics by removing resource related labels. (#69895, @jpbetz)

v1.11.8

Documentation

Downloads for v1.11.8

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.7

Other notable changes

• fix smb remount issue on Windows (#73661, @andyzhangx)
• kube-apiserver: a request body of a CREATE/UPDATE/PATCH/DELETE resource operation larger than 100 MB will return a 413 "request entity too large" error. (#73805, @caesarxuchao)
  • Custom apiservers built with the latest apiserver library will have the 100MB limit on the body of resource requests as well. The limit can be altered via ServerRunOptions.MaxRequestBodyBytes.
  • The body size limit does not apply to subresources like pods/proxy that proxy request content to another server.
• The apiserver, including both the kube-apiserver and apiservers built with the generic apiserver library, will now return 413 RequestEntityTooLarge error if a json patch contains more than 10,000 operations. (#74000, @caesarxuchao)
• Update Cluster Autoscaler to version 1.3.7. (#74136, @jkaniuk)
• Fix watch to not send the same set of events multiple times causing watcher to go back in time (#73845, @wojtek-t)
• fixes an error processing watch events when running skewed apiservers (#73482, @liggitt)
• MAC Address filter has been fixed in vSphere Cloud Provider, it no longer ignores 00:1c:14 and 00:05:69 prefixes (#73721, @frapposelli)
• add goroutine to move unschedulable pods to activeq if they are not retried for more than 1 minute (#72558, @denkensk)
• Scale max-inflight limits together with master VM sizes. (#73268, @wojtek-t)
• Update to go1.10.8 (#73379, @cblecker)

v1.11.7

Documentation

Downloads for v1.11.7

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.6

Other notable changes

• Allow for watching objects larger than 1MB given etcd accepts objects of size up to 1.5MB (#72053, @wojtek-t)
• Fix AWS NLB security group updates where valid security group ports were incorrectly removed (#68422, @kellycampbell)
  • when updating a service or when node changes occur.
• kubectl: fixed an issue with "too old resource version" errors continuously appearing when calling kubectl delete (#72825, @liggitt)
• Fixes issue where subpath volume content was deleted during orphaned pod cleanup for Local volumes that are directories (and not mount points) on the root filesystem. (#72291, @msau42)
• Fixes spurious 0-length API responses. (#72856, @liggitt)
• Do not detach volume if mount in progress (#71145, @gnufied)
• Improve efficiency of preemption logic in clusters with many pending pods. (#72895, @bsalamat)
• Fix scheduling starvation of pods in cluster with large number of unschedulable pods. (#72619, @everpeace)
• change azure disk host cache to ReadOnly by default (#72229, @andyzhangx)
• Fixes issue with cleaning up stale NFS subpath mounts (#71804, @msau42)
• Update Cluster Autoscaler to version 1.3.5. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.5 (#72579, @MaciekPytel)
• Fix kube-proxy PodSecurityPolicy binding on GCE & GKE. This was only an issue when running kube-proxy as a DaemonSet, with PodSecurityPolicy enabled. (#72761, @tallclair)
• Scheduler only activates unschedulable pods if node's scheduling related properties change. (#71551, @mlmhl)
• Fix a race condition in the scheduler preemption logic that could cause nominatedNodeName of a pod not to be considered in one or more scheduling cycles. (#72504, @bsalamat)
• fix race condition when attach azure disk in vmss (#71992, @andyzhangx)
• Fixed kubelet reporting "resource name may not be empty" when mounting a volume very quickly after unmount. (#71074, @jsafrane)
• Fix race condition introduced by graceful termination which can lead to a deadlock in kube-proxy (#72361, @lbernail)
• Fixes an issue where Portworx volumes cannot be mounted if 9001 port is already in use on the host and users remap 9001 to another port. (#70392, @harsh-px)
• kube-proxy in IPVS mode will stop initiating connections to terminating pods for services with sessionAffinity set. (#71834, @lbernail)
• Update to use go1.10.7 with fix for CVE-2018-16875 (#72074, @ixdy)

v1.11.6

Documentation

Downloads for v1.11.6

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.5

Other notable changes

• Scheduler only activates unschedulable pods if node's scheduling related properties change. (#71551, @mlmhl)
• Disable proxy to loopback and linklocal (#71980, @micahhausler)
• Scheduler only activates unschedulable pods if node's scheduling related properties change. (#71551, @mlmhl)
• fix issue: vm sku restriction policy does not work in azure disk attach/detach (#71941, @andyzhangx)
• Include CRD for BGPConfigurations, needed for calico 2.x to 3.x upgrade. (#71868, @satyasm)
• On GCI, NPD starts to monitor kubelet, docker, containerd crashlooping, read-only filesystem and corrupt docker overlay2 issues. (#71522, @wangzhen127)
• UDP connections now support graceful termination in IPVS mode (#71515, @lbernail)
• Fix a potential bug that scheduler preempts unnecessary pods. (#70898, @Huang-Wei)
• Only use the first IP address got from instance metadata. This is because Azure CNI would set up a list of IP addresses in instance metadata, while only the first one is the Node's IP. (#71736, @feiskyer)
• Fix scheduling starvation of pods in cluster with large number of unschedulable pods. (#71488, @bsalamat)
• Fixes an issue with stuck connections handling error responses (#71419, @liggitt)
• Ensure orphan public IPs on Azure deleted when service recreated with the same name. (#70463, @feiskyer)
• Upgrade Stackdriver Logging Agent addon image to 0.6-1.6.0-1 to use Fluentd v1.2. This provides nanoseconds timestamp granularity for logs. (#70954, @qingling128)
• [GCE] Filter out spammy audit logs from cluster autoscaler. (#70696, @loburm)
• Fix a scheduler panic due to internal cache inconsistency (#71063, @Huang-Wei)
• apiserver: fixes handling and logging of panics in REST handlers to prevent crashes (#71076, @liggitt)
• Upgrade golang.org/x/net image to release-branch.go1.10 (#70663, @wenjiaswe)
• Fixes a bug in previous releases where a pod could be placed inside another pod's cgroup when specifying --cgroup-root (#70678, @dashpole)
• Fixes ability for admin/edit/view users to see controller revisions, needed for kubectl rollout commands (#70699, @liggitt)
• remove retry operation on attach/detach azure disk (#70568, @andyzhangx)
• Fix cloud-controller-manager crash when using OpenStack provider and PersistentVolume initializing controller (#70459, @mvladev)
• Update defaultbackend to 1.5: move /metrics to a private endpoint (:10254). (#69383, @bowei)
• Allows changing nodeName in endpoint update. (#68575, @prameshj)
• Improve Azure instance metadata handling by adding caches. (#70353, @feiskyer)

v1.11.5

Documentation

Downloads for v1.11.5

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.4

Other notable changes

• CVE-2018-1002105: Fix critical security issue in kube-apiserver upgrade request proxy handler (#71411, @liggitt)
• IPVS proxier mode now support connection based graceful termination. (#66012, @Lion-Wei)
• Update Cluster Autoscaler to 1.3.4 (#70285, @losipiuk)
• Fix cluster autoscaler addon permissions so it can access batch/job. (#69858, @losipiuk)
• Do not remove shutdown nodes from api-server for vSphere. (#70291, @gnufied)
  • Fixes volume detach from shutdown nodes.
• Enable insertId generation, and update Stackdriver Logging Agent image to 0.5-1.5.36-1-k8s. This help reduce log duplication and guarantee log order. (#68920, @qingling128)
• fix azure disk attachment error on Linux (#70002, @andyzhangx)
• GCE/GKE load balancer health check default interval changes from 2 seconds to 8 seconds, unhealthyThreshold to 3. (#70099, @grayluck)
  • Health check parameters are configurable to be bigger than default values.

v1.11.4

Documentation

Downloads for v1.11.4

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.3

Other notable changes

• support cross resource group for azure file (#68117, @andyzhangx)
• change default azure file mount permission to 0777 (#69854, @andyzhangx)
• Scheduling conformance tests related to daemonsets should set the annotation that relaxes node selection restrictions, if any are set. This ensures conformance tests can run on a wider array of clusters. (#68793, @aveshagarwal)
• Add tolerations for Stackdriver Logging and Metadata Agents. (#69737, @qingling128)
• Fix scheduler crashes when Prioritize Map function returns error. (#69618, @DylanBLE)
• Add fallbacks to ARM API when getting empty node IP from Azure IMDS (#69077, @feiskyer)
• Verify invalid secret/configmap/projected volumes before calling setup (#68691, @gnufied)
• Immediately close the other side of the connection when proxying. (#67288, @MHBauer)
• fix a bug that overwhelming number of prometheus metrics are generated because $NAMESPACE is not replaced by string "{namespace}" (#68530, @wenjiaswe)
• locking fix in vSphere cleanup function (#68388, @wgliang)
  • The vSphere cloud provider has a cleanup function with a loop,
  • inside of which a write lock was intended to prevent races against
  • VM creation, but the usage of the sync.RWMutex type is incorrect,
  • plus is placed in a defer call so the locking is effectively
  • nonexistent. This patch adds an anonymous function call inside
  • the loop to correctly handle acquiring the write lock with deferred
  • unlock to run at that function's completion, following the normal
  • golang lock/deferred-unlock pattern.
• fix UnmountDevice failure on Windows (#68608, @andyzhangx)
• Get public IP for Azure vmss nodes. (#68498, @feiskyer)
• Role, ClusterRole and their bindings for cloud-provider is put under system namespace. Their addonmanager mode switches to EnsureExists. (#67224, @grayluck)
• Bump up version number of debian-base, debian-hyperkube-base and debian-iptables. (#67026, @satyasm)
  • Also updates dependencies of users of debian-base.
  • debian-base version 0.3.1 is already available.
• Update debian-iptables and hyperkube-base images to include CVE fixes. (#67365, @ixdy)
• Support configuring the Azure load balancer idle connection timeout for services (#66045, @cpuguy83)
• Let service controller retry creating load balancer when persistUpdate failed due to conflict. (#68087, @grayluck)
• Return apiserver panics as 500 errors instead terminating the apiserver process. (#68001, @sttts)
• adjusted http/2 buffer sizes for apiservers to prevent starvation issues between concurrent streams (#67902, @liggitt)
• Reduced excessive logging from fluentd-gcp-scaler. (#68837, @x13n)
• Fix potential panic when getting azure load balancer status (#68609, @feiskyer)
• Fix an issue where filesystems are not unmounted when a backend is not reachable and returns EIO. (#67097, @chakri-nelluri)
• [GCE] Enable by default audit logging truncating backend. (#68342, @loburm)
• [fluentd-gcp-scaler addon] Bump fluentd-gcp-scaler to 0.4 to pick up security fixes. (#67691, @loburm)
  • [prometheus-to-sd addon] Bump prometheus-to-sd to 0.3.1 to pick up security fixes, bug fixes and new features.
  • [event-exporter addon] Bump event-exporter to 0.2.3 to pick up security fixes.
• [GCP] Added env variables to control CPU requests of kube-controller-manager and kube-scheduler. (#68823, @loburm)
• Update Cluster Autoscaler to version 1.3.3. (#68936, @losipiuk)
• Bump addon-manager to v8.7 (#68299, @MrHohn)
  • Support extra --prune-whitelist resources in kube-addon-manager.
  • Update kubectl to v1.10.7.
• Support extra --prune-whitelist resources in kube-addon-manager. (#67743, @Random-Liu)
  • Before, when users run admission webhook as an addon pod, after they remove the webhook, kube-addon-manager won't cleanup the webhook configuration MutatingWebhookConfiguration/ValidationWebhookConfiguration, because those resources are not prune-whitelisted. Cluster will go into a bad state that no pod can be admitted.
  • With this feature, users can config kube-addon-manager to whitelist those resources to fix this issue.
• Bump ip-masq-agent to v2.1.1 (#67916, @MrHohn)
  • Update debian-iptables image for CVEs.
  • Change chain name to IP-MASQ to be compatible with the pre-injected masquerade rules.
• If TaintNodesByCondition is enabled, add node.kubernetes.io/unschedulable and node.kubernetes.io/network-unavailable automatically to DaemonSet pods. (#64954, @k82cn)
• Fix VMWare VM freezing bug by reverting #51066 (#67825, @nikopen)

v1.11.3

Documentation & Examples

Downloads for v1.11.3

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.2

Action Required

• action required: the API server and client-go libraries have been fixed to support additional non-alpha-numeric characters in UserInfo "extra" data keys. Both should be updated in order to properly support extra data containing "/" characters or other characters disallowed in HTTP headers. (#65799, @dekkagaijin)

Other notable changes

• Orphan DaemonSet when deleting with --cascade option set (#68007, @soltysh)
• Bump event-exporter to 0.2.2 to pick up security fixes. (#66157, @loburm)
• PVC may not be synced to controller local cache in time if PV is bound by external PV binder (e.g. kube-scheduler), double check if PVC is not found to prevent reclaiming PV wrongly. (#67062, @cofyc)
• kube-apiserver: fixes error creating system priority classes when starting multiple apiservers simultaneously (#67372, @tanshanshan)
• Add NoSchedule/NoExecute tolerations to ip-masq-agent, ensuring it to be scheduled in all nodes except master. (#66260, @tanshanshan)
• Prevent resourceVersion updates for custom resources on no-op writes. (#67562, @nikhita)
• fix cluster-info dump error (#66652, @charrywanganthony)
• attachdetach controller attaches volumes immediately when Pod's PVCs are bound (#66863, @cofyc)
• kube-apiserver now includes all registered API groups in discovery, including registered extension API group/versions for unavailable extension API servers. (#66932, @nilebox)
• fixes an issue with multi-line annotations injected via downward API files getting scrambled (#65992, @liggitt)
• Revert #63905: Setup dns servers and search domains for Windows Pods. DNS for Windows containers will be set by CNI plugins. (#66587, @feiskyer)
• Fix an issue that pods using hostNetwork keep increasing. (#67456, @Huang-Wei)
• Bump Heapster to v1.6.0-beta.1 (#67074, @kawych)
• fix azure disk create failure due to sdk upgrade (#67236, @andyzhangx)
• Fix creation of custom resources when the CRD contains non-conventional pluralization and subresources (#66249, @deads2k)
• Allows extension API server to dynamically discover the requestheader CA certificate when the core API server doesn't use certificate based authentication for it's clients (#66394, @rtripat)

v1.11.2

Documentation

Downloads for v1.11.2

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.1

Action Required

• Cluster Autoscaler version updated to 1.3.1. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.1 (#66123, @aleksandra-malinowska)
  • Default value for expendable pod priority cutoff in GCP deployment of Cluster Autoscaler changed from 0 to -10.
  • action required: users deploying workloads with priority lower than 0 may want to use priority lower than -10 to avoid triggering scale-up.

Other notable changes

• Metadata Agent Improvements (#66485, @bmoyles0117)
  • Bump metadata agent version to 0.2-0.0.21-1.
  • Expand the metadata agent's access to all API groups.
  • Remove metadata agent config maps in favor of command line flags.
  • Update the metadata agent's liveness probe to a new /healthz handler.
  • Logging Agent Improvements
  • Bump logging agent version to 0.2-1.5.33-1-k8s-1.
  • Appropriately set log severity for k8s_container.
  • Fix detect exceptions plugin to analyze message field instead of log field.
  • Fix detect exceptions plugin to analyze streams based on local resource id.
  • Disable the metadata agent for monitored resource construction in logging.
  • Disable timestamp adjustment in logs to optimize performance.
  • Reduce logging agent buffer chunk limit to 512k to optimize performance.
• Fix a bug on GCE that /etc/crictl.yaml is not generated when crictl is preloaded. (#66877, @Random-Liu)
• Fix validation for HealthzBindAddress in kube-proxy when --healthz-port is set to 0 (#66138, @wsong)
• fix acr could not be listed in sp issue (#66429, @andyzhangx)
• Fixed an issue which prevented gcloud from working on GCE when metadata concealment was enabled. (#66630, @dekkagaijin)
• Fix for resourcepool-path configuration in the vsphere.conf file. (#66261, @divyenpatel)
• This fix prevents a GCE PD volume from being mounted if the udev device link is stale and tries to correct the link. (#66832, @msau42)
• Fix kubelet startup failure when using ExecPlugin in kubeconfig (#66395, @awly)
• GCE: Fixes loadbalancer creation and deletion issues appearing in 1.10.5. (#66400, @nicksardo)
• kubeadm: Pull sidecar and dnsmasq-nanny images when using kube-dns (#66499, @rosti)
• fix smb mount issue (#65751, @andyzhangx)
• Extend TLS timeouts to work around slow arm64 math/big (#66264, @joejulian)
• Allow ScaleIO volumes to be provisioned without having to first manually create /dev/disk/by-id path on each kubernetes node (if not already present) (#66174, @ddebroy)
• kubeadm: stop setting UID in the kubelet ConfigMap (#66341, @runiq)
• fixes a panic when using a mutating webhook admission plugin with a DELETE operation (#66425, @liggitt)
• Update crictl to v1.11.1. (#66152, @Random-Liu)
• kubectl: fixes a panic displaying pods with nominatedNodeName set (#66406, @liggitt)
• fixes a validation error that could prevent updates to StatefulSet objects containing non-normalized resource requests (#66165, @liggitt)
• Tolerate missing watch permission when deleting a resource (#65370, @deads2k)
• prevents infinite CLI wait on delete when item is recreated (#66136, @deads2k)
• Preserve vmUUID when renewing nodeinfo in vSphere cloud provider (#66007, @w-leads)

v1.11.1

Documentation

Downloads for v1.11.1

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0

Action Required

• ACTION REQUIRED: Removes defaulting of CSI file system type to ext4. All the production drivers listed under https://kubernetes-csi.github.io/docs/drivers.html were inspected and should not be impacted after this change. If you are using a driver not in that list, please test the drivers on an updated test cluster first. (#65499, @krunaljain)
• kube-apiserver: the Priority admission plugin is now enabled by default when using --enable-admission-plugins. If using --admission-control to fully specify the set of admission plugins, the Priority admission plugin should be added if using the PodPriority feature, which is enabled by default in 1.11. (#65739, @liggitt)
• The system-node-critical and system-cluster-critical priority classes are now limited to the kube-system namespace by the PodPriority admission plugin. (#65593, @bsalamat)

Other notable changes

• kubeadm: run kube-proxy on non-master tainted nodes (#65931, @neolit123)
• Fix an issue with dropped audit logs, when truncating and batch backends enabled at the same time. (#65823, @loburm)
• set EnableHTTPSTrafficOnly in azure storage account creation (#64957, @andyzhangx)
• Re-adds pkg/generated/bindata.go to the repository to allow some parts of k8s.io/kubernetes to be go-vendorable. (#65985, @ixdy)
• Fix a scalability issue where high rates of event writes degraded etcd performance. (#64539, @ccding)
• "kubectl delete" no longer waits for dependent objects to be deleted when removing parent resources (#65908, @juanvallejo)
• Update to use go1.10.3 (#65726, @ixdy)
• Fix the bug where image garbage collection is disabled by mistake. (#66051, @jiaxuanzhou)
• Fix RunAsGroup which doesn't work since 1.10. (#65926, @Random-Liu)
• Fixed cleanup of CSI metadata files. (#65323, @jsafrane)
• Reload systemd config files before starting kubelet. (#65702, @mborsz)
• Fix a bug that preempting a pod may block forever. (#65987, @Random-Liu)
• fixes a regression in kubectl printing behavior when using go-template or jsonpath output that resulted in a "unable to match a printer" error message (#65979, @juanvallejo)
• add external resource group support for azure disk (#64427, @andyzhangx)
• Properly manage security groups for loadbalancer services on OpenStack. (#65373, @multi-io)
• Allow access to ClusterIP from the host network namespace when kube-proxy is started in IPVS mode without either masqueradeAll or clusterCIDR flags (#65388, @lbernail)
• kubeadm: Fix pause image to not use architecture, as it is a manifest list (#65920, @dims)
• bazel deb package bugfix: The kubeadm deb package now reloads the kubelet after installation (#65554, @rdodev)
• The garbage collector now supports CustomResourceDefinitions and APIServices. (#65915, @nikhita)
• fix azure storage account creation failure (#65846, @andyzhangx)
• skip nodes that have a primary NIC in a 'Failed' provisioningState (#65412, @yastij)
• Fix 'kubectl cp' with no arguments causes a panic (#65482, @wgliang)
• On COS, NPD creates a node condition for frequent occurrences of unregister_netdevice (#65342, @dashpole)
• bugfix: Do not print feature gates in the generic apiserver code for glog level 0 (#65584, @neolit123)
• Add prometheus scrape port to CoreDNS service (#65589, @rajansandeep)
• kubectl: fixes a regression with --use-openapi-print-columns that would not print object contents (#65600, @liggitt)
• fixes an out of range panic in the NoExecuteTaintManager controller when running a non-64-bit build (#65596, @liggitt)
• Fixed issue 63608, which is that under rare circumstances the ResourceQuota admission controller could lose track of an request in progress and time out after waiting 10 seconds for a decision to be made. (#64598, @MikeSpreitzer)

v1.11.0

Documentation

Downloads for v1.11.0

Client Binaries

Server Binaries

Node Binaries

Kubernetes 1.11 Release Notes

Urgent Upgrade Notes

(No, really, you MUST do this before you upgrade)

Before upgrading to Kubernetes 1.11, you must keep the following in mind:

• JSON configuration files that contain fields with incorrect case will no longer be valid. You must correct these files before upgrading. When specifying keys in JSON resource definitions during direct API server communication, the keys are case-sensitive. A bug introduced in Kubernetes 1.8 caused the API server to accept a request with incorrect case and coerce it to correct case, but this behaviour has been fixed in 1.11 and the API server will once again be enforcing the correct case. It’s worth noting that during this time, the kubectl tool continued to enforce case-sensitive keys, so users that strictly manage resources with kubectl will be unaffected by this change. (#65034, @caesarxuchao)
• Pod priority and preemption is now enabled by default. Note that this means that pods from any namespace can now request priority classes that compete with and/or cause preemption of critical system pods that are already running. If that is not desired, disable the PodPriority feature by setting --feature-gates=PodPriority=false on the kube-apiserver, kube-scheduler, and kubelet components before upgrading to 1.11. Disabling the PodPriority feature limits critical pods to the kube-system namespace.

Major Themes

SIG API Machinery

This release SIG API Machinery focused mainly on CustomResources. For example, subresources for CustomResources are now beta and enabled by default. With this, updates to the /status subresource will disallow updates to all fields other than .status (not just .spec and .metadata as before). Also, required and description can be used at the root of the CRD OpenAPI validation schema when the /status subresource is enabled.

In addition, users can now create multiple versions of CustomResourceDefinitions, but without any kind of automatic conversion, and CustomResourceDefinitions now allow specification of additional columns for kubectl get output via the spec.additionalPrinterColumns field.

SIG Auth

Work this cycle focused on graduating existing functions, and on making security functions more understandable for users.

RBAC cluster role aggregation, introduced in 1.9, graduated to stable status with no changes in 1.11, and client-go credential plugins graduated to beta status, while also adding support for obtaining TLS credentials from an external plugin.

Kubernetes 1.11 also makes it easier to see what's happening, as audit events can now be annotated with information about how an API request was handled:

• Authorization sets authorization.k8s.io/decision and authorization.k8s.io/reason annotations with the authorization decision ("allow" or "forbid") and a human-readable description of why the decision was made (for example, RBAC includes the name of the role/binding/subject which allowed a request).
• PodSecurityPolicy admission sets podsecuritypolicy.admission.k8s.io/admit-policy and podsecuritypolicy.admission.k8s.io/validate-policy annotations containing the name of the policy that allowed a pod to be admitted. (PodSecurityPolicy also gained the ability to limit hostPath volume mounts to be read-only.)

In addition, the NodeRestriction admission plugin now prevents kubelets from modifying taints on their Node API objects, making it easier to keep track of which nodes should be in use.

SIG CLI

SIG CLI's main focus this release was on refactoring kubectl internals to improve composability, readability and testability of kubectl commands. Those refactors will allow the team to extract a mechanism for extensibility of kubectl -- that is, plugins -- in the next releases.

SIG Cluster Lifecycle

SIG Cluster Lifecycle focused on improving kubeadm’s user experience by including a set of new commands related to maintaining the kubeadm configuration file, the API version of which has now has been incremented to v1alpha2. These commands can handle the migration of the configuration to a newer version, printing the default configuration, and listing and pulling the required container images for bootstrapping a cluster.

Other notable changes include:

• CoreDNS replaces kube-dns as the default DNS provider
• Improved user experience for environments without a public internet connection and users using other CRI runtimes than Docker
• Support for structured configuration for the kubelet, which avoids the need to modify the systemd drop-in file
• Many improvements to the upgrade process and other bug fixes

SIG Instrumentation

As far as Sig Instrumentation, the major change in Kubernetes 1.11 is the deprecation of Heapster as part of ongoing efforts to move to the new Kubernetes monitoring model. Clusters still using Heapster for autoscaling should be migrated over to metrics-server and the custom metrics API. See the deprecation section for more information.

SIG Network

The main milestones for SIG Network this release are the graduation of IPVS-based load balancing and CoreDNS to general availability.

IPVS is an alternative approach to in-cluster load balancing that uses in-kernel hash tables rather than the previous iptables approach, while CoreDNS is a replacement for kube-dns for service discovery.

SIG Node

SIG-Node advanced several features and made incremental improvements in a few key topic areas this release.

The dynamic kubelet config feature graduated to beta, so it is enabled by default, simplifying management of the node object itself. Kubelets that are configured to work with the CRI may take advantage of the log rotation feature, which is graduating to beta this release.

The cri-tools project, which aims to provide consistent tooling for operators to debug and introspect their nodes in production independent of their chosen container runtime, graduated to GA.

As far as platforms, working with SIG-Windows, enhancements were made to the kubelet to improve platform support on Windows operating systems, and improvements to resource management were also made. In particular, support for sysctls on Linux graduated to beta.

SIG OpenStack

SIG-OpenStack continued to build out testing, with eleven acceptance tests covering a wide-range of scenarios and use-cases. During the 1.11 cycle our reporting back to test-grid has qualified the OpenStack cloud provider as a gating job for the Kubernetes release.

New features include improved integration between the Keystone service and Kubernetes RBAC, and a number of stability and compatibility improvements across the entire provider code-base.

SIG Scheduling

Pod Priority and Preemption has graduated to Beta, so it is enabled by default. Note that this involves significant and important changes for operators. The team also worked on improved performance and reliability of the scheduler.

SIG Storage

Sig Storage graduated two features that had been introduced in previous versions and introduced three new features in an alpha state.

The StorageProtection feature, which prevents deletion of PVCs while Pods are still using them and of PVs while still bound to a PVC, is now generally available, and volume resizing, which lets you increase size of a volume after a Pod restarts is now beta, which means it is on by default.

New alpha features include:

• Online volume resizing will increase the filesystem size of a resized volume without requiring a Pod restart.
• AWS EBS and GCE PD volumes support increased limits on the maximum number of attached volumes per node.
• Subpath volume directories can be created using DownwardAPI environment variables.

SIG Windows

This release supports more of Kubernetes API for pods and containers on Windows, including:

• Metrics for Pod, Container, Log filesystem
• The run_as_user security contexts
• Local persistent volumes and fstype for Azure disk

Improvements in Windows Server version 1803 also bring new storage functionality to Kubernetes v1.11, including:

• Volume mounts for ConfigMap and Secret
• Flexvolume plugins for SMB and iSCSI storage are also available out-of-tree at Microsoft/K8s-Storage-Plugins

Known Issues

• IPVS based kube-proxy doesn't support graceful close connections for terminating pod. This issue will be fixed in a future release. (#57841, @jsravn)
• kube-proxy needs to be configured to override hostname in some environments. (#857, @detiber)
• There's a known issue where the Vertical Pod Autoscaler will radically change implementation in 1.12, so users of VPA (alpha) in 1.11 are warned that they will not be able to automatically migrate their VPA configs from 1.11 to 1.12.

Before Upgrading

• When Response is a metav1.Status, it is no longer copied into the audit.Event status. Only the "status", "reason" and "code" fields are set. For example, when we run kubectl get pods abc, the API Server returns a status object: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"abc\" not found","reason":"NotFound","details":{"name":"abc","kind":"pods"},"code":404} In previous versions, the whole object was logged in audit events. Starting in 1.11, only status, reason, and code are logged. Code that relies on the older version must be updated to avoid errors. (#62695, @CaoShuFeng)
• HTTP transport now uses context.Context to cancel dial operations. k8s.io/client-go/transport/Config struct has been updated to accept a function with a context.Context parameter. This is a breaking change if you use this field in your code. (#60012, @ash2k)
• kubectl: This client version requires the apps/v1 APIs, so it will not work against a cluster version older than v1.9.0. Note that kubectl only guarantees compatibility with clusters that are +/-1 minor version away. (#61419, @enisoc)
• Pod priority and preemption is now enabled by default. Even if you don't plan to use this feature, you might need to take some action immediately after upgrading. In multi-tenant clusters where not all users are trusted, you are advised to create appropriate quotas for two default priority classes, system-cluster-critical and system-node-critical, which are added to clusters by default. ResourceQuota should be created to limit users from creating Pods at these priorities if not all users of your cluster are trusted. We do not advise disabling this feature because critical system Pods rely on the scheduler preemption to be scheduled when cluster is under resource pressure.
• Default mount propagation has changed from HostToContainer ("rslave" in Linux terminology), as it was in 1.10, to None ("private") to match the behavior in 1.9 and earlier releases; HostToContainer as a default caused regressions in some pods. If you are relying on this behavior you will need to set it explicitly. (#62462, @jsafrane)
• The kube-apiserver --storage-version flag has been removed; you must use --storage-versions instead. (#61453, @hzxuzhonghu)
• Authors of aggregated API servers must not rely on authorization being done by the kube-apiserver, and must do delegated authorization in addition. (#61349, @sttts)
• GC is now bound by QPS so if you need more QPS to avoid ratelimiting GC, you'll have to set it explicitly. (#63657, @shyamjvs)
• kubeadm join is now blocking on the kubelet performing the TLS Bootstrap properly. Earlier, kubeadm join only did the discovery part and exited successfully without checking that the kubelet actually started properly and performed the TLS bootstrap correctly. Now, as kubeadm runs some post-join steps (for example, annotating the Node API object with the CRISocket), kubeadm join is now waiting for the kubelet to perform the TLS Bootstrap, and then uses that credential to perform further actions. This also improves the UX, as kubeadm will exit with a non-zero code if the kubelet isn't in a functional state, instead of pretending everything's fine. (#64792, @luxas)
• The structure of the kubelet dropin in the kubeadm deb package has changed significantly. Instead of hard-coding the parameters for the kubelet in the dropin, a structured configuration file for the kubelet is used, and is expected to be present in /var/lib/kubelet/config.yaml. For runtime-detected, instance-specific configuration values, a environment file with dynamically-generated flags at kubeadm init or kubeadm join run time is used. Finally, if you want to override something specific for the kubelet that can't be done via the kubeadm Configuration file (which is preferred), you might add flags to the KUBELET_EXTRA_ARGS environment variable in either /etc/default/kubelet or /etc/sysconfig/kubelet, depending on the system you're running on. (#64780, @luxas)
• The --node-name flag for kubeadm now dictates the Node API object name the kubelet uses for registration, in all cases but where you might use an in-tree cloud provider. If you're not using an in-tree cloud provider, --node-name will set the Node API object name. If you're using an in-tree cloud provider, you MUST make --node-name match the name the in-tree cloud provider decides to use. (#64706, @liztio)
• The PersistentVolumeLabel admission controller is now disabled by default. If you depend on this feature (AWS/GCE) then ensure it is added to the --enable-admission-plugins flag on the kube-apiserver. (#64326, @andrewsykim)
• kubeadm: kubelets in kubeadm clusters now disable the readonly port (10255). If you're relying on unauthenticated access to the readonly port, please switch to using the secure port (10250). Instead, you can now use ServiceAccount tokens when talking to the secure port, which will make it easier to get access to, for example, the /metrics endpoint of the kubelet, securely. (#64187, @luxas)
• The formerly publicly-available cAdvisor web UI that the kubelet ran on port 4194 by default is now turned off by default. The flag configuring what port to run this UI on --cadvisor-port was deprecated in v1.10. Now the default is --cadvisor-port=0, in other words, to not run the web server. If you still need to run cAdvisor, the recommended way to run it is via a DaemonSet. Note that the --cadvisor-port will be removed in v1.12 (#63881, @luxas)

New Deprecations

• As a reminder, etcd2 as a backend is deprecated and support will be removed in Kubernetes 1.13. Please ensure that your clusters are upgraded to etcd3 as soon as possible.
• InfluxDB cluster monitoring has been deprecated as part of the deprecation of Heapster. Instead, you may use the metrics server. It's a simplified heapster that is able to gather and serve current metrics values. It provides the Metrics API that is used by kubectl top, and horizontal pod autoscaler. Note that it doesn't include some features of Heapster, such as short term metrics for graphs in kube-dashboard and dedicated push sinks, which proved hard to maintain and scale. Clusters using Heapster for transferring metrics into long-term storage should consider using their metric solution's native Kubernetes support, if present, or should consider alternative solutions. (#62328, @serathius)
• The kubelet --rotate-certificates flag is now deprecated, and will be removed in a future release. The kubelet certificate rotation feature can now be enabled via the .RotateCertificates field in the kubelet's config file. (#63912, @luxas)
• The kubeadm configuration file version has been upgraded from v1alpha1 to v1alpha2. v1alpha1 read support exists in v1.11, but will be removed in v1.12. (#63788, @luxas) The following PRs changed the API spec:
  • In the new v1alpha2 kubeadm Configuration API, the .CloudProvider and .PrivilegedPods fields don't exist anymore. Instead, you should use the out-of-tree cloud provider implementations, which are beta in v1.11.
  • If you have to use the legacy in-tree cloud providers, you can rearrange your config like the example below. If you need the cloud-config file (located in {cloud-config-path}), you can mount it into the API Server and controller-manager containers using ExtraVolumes, as in:

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha2
apiServerExtraArgs:
  cloud-provider: "{cloud}"
  cloud-config: "{cloud-config-path}"
apiServerExtraVolumes:
- name: cloud
  hostPath: "{cloud-config-path}"
  mountPath: "{cloud-config-path}"
controllerManagerExtraArgs:
  cloud-provider: "{cloud}"
  cloud-config: "{cloud-config-path}"
controllerManagerExtraVolumes:
- name: cloud
  hostPath: "{cloud-config-path}"
  mountPath: "{cloud-config-path}"

• If you need to use the .PrivilegedPods functionality, you can still edit the manifests in /etc/kubernetes/manifests/, and set .SecurityContext.Privileged=true for the apiserver and controller manager. (#63866, @luxas)
• kubeadm: The Token-related fields in the MasterConfiguration object have now been refactored. Instead of the top-level .Token, .TokenTTL, .TokenUsages, .TokenGroups fields, there is now a BootstrapTokens slice of BootstrapToken objects that support the same features under the .Token, .TTL, .Usages, .Groups fields. (#64408, @luxas)
• .NodeName and .CRISocket in the MasterConfiguration and NodeConfiguration v1alpha1 API objects are now .NodeRegistration.Name and .NodeRegistration.CRISocket respectively in the v1alpha2 API. The .NoTaintMaster field has been removed in the v1alpha2 API. (#64210, @luxas)
• kubeadm: Support for .AuthorizationModes in the kubeadm v1alpha2 API has been removed. Instead, you can use the .APIServerExtraArgs and .APIServerExtraVolumes fields to achieve the same effect. Files using the v1alpha1 API and setting this field will be automatically upgraded to this v1alpha2 API and the information will be preserved. (#64068, @luxas)
• The annotation service.alpha.kubernetes.io/tolerate-unready-endpoints is deprecated. Users should use Service.spec.publishNotReadyAddresses instead. (#63742, @thockin)
• --show-all, which only affected pods, and even then only for human readable/non-API printers, is inert in v1.11, and will be removed in a future release. (#60793, @charrywanganthony)
• The kubectl rolling-update is now deprecated. Use kubectl rollout instead. (#61285, @soltysh)
• kube-apiserver: the default --endpoint-reconciler-type is now lease. The master-count endpoint reconciler type is deprecated and will be removed in 1.13. (#63383, @liggitt)
• OpenStack built-in cloud provider is now deprecated. Please use the external cloud provider for OpenStack. (#63524, @dims)
• The Kubelet's deprecated --allow-privileged flag now defaults to true. This enables users to stop setting --allow-privileged in order to transition to PodSecurityPolicy. Previously, users had to continue setting --allow-privileged, because the default was false. (#63442, @mtaufen)
• The old dynamic client has been replaced by a new one. The previous dynamic client will exist for one release in client-go/deprecated-dynamic. Switch as soon as possible. (#63446, @deads2k)
• In-tree support for openstack credentials is now deprecated. please use the "client-keystone-auth" from the cloud-provider-openstack repository. details on how to use this new capability is documented here (#64346, @dims)
• The GitRepo volume type is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. (#63445, @ericchiang)
• Alpha annotation for PersistentVolume node affinity has been removed. Update your PersistentVolumes to use the beta PersistentVolume.nodeAffinity field before upgrading to this release. (#61816, @wackxu )

Removed Deprecations

• kubeadm has removed the .ImagePullPolicy field in the v1alpha2 API version. Instead it's set statically to IfNotPresent for all required images. If you want to always pull the latest images before cluster init (as Always would do), run kubeadm config images pull before each kubeadm init. If you don't want the kubelet to pull any images at kubeadm init time, for example if you don't have an internet connection, you can also run kubeadm config images pull before kubeadm init or side-load the images some other way (such as docker load -i image.tar). Having the images locally cached will result in no pull at runtime, which makes it possible to run without any internet connection. (#64096, @luxas)
• kubeadm has removed .Etcd.SelfHosting from its configuration API. It was never used in practice (#63871, @luxas)
• The deprecated and inactive option '--enable-custom-metrics' has been removed in 1.11. (#60699, @CaoShuFeng)
• --include-extended-apis, which was deprecated back in #32894, has been removed. (#62803, @deads2k)
• Kubelets will no longer set externalID in their node spec. This feature has been deprecated since v1.1. (#61877, @mikedanese)
• The initresource admission plugin has been removed. (#58784, @wackxu)
• ObjectMeta , ListOptions, and DeleteOptions have been removed from the core api group. Please reference them in meta/v1 instead. (#61809, @hzxuzhonghu)
• The deprecated --mode flag in check-network-mode has been removed. (#60102, @satyasm)
• Support for the alpha.kubernetes.io/nvidia-gpu resource, which was deprecated in 1.10, has been removed. Please use the resource exposed by DevicePlugins instead (nvidia.com/gpu). (#61498, @mindprince)
• The kube-cloud-controller-manager flag --service-account-private-key-file has been removed. Use --use-service-account-credentials instead. (#60875, @charrywanganthony)
• The rknetes code, which was deprecated in 1.10, has been removed. Use rktlet and CRI instead. (#61432, @filbranden)
• DaemonSet scheduling associated with the alpha ScheduleDaemonSetPods feature flag has been removed. See https://github.com/kubernetes/enhancements/issues/548 for feature status. (#61411, @liggitt)
• The METADATA_AGENT_VERSION configuration option has been removed to keep metadata agent version consistent across Kubernetes deployments. (#63000, @kawych)
• The deprecated --service-account-private-key-file flag has been removed from the cloud-controller-manager. The flag is still present and supported in the kube-controller-manager. (#65182, @liggitt)
• Removed alpha functionality that allowed the controller manager to approve kubelet server certificates. This functionality should be replaced by automating validation and approval of node server certificate signing requests. (#62471, @mikedanese)

Graduated to Stable/GA

• IPVS-based in-cluster load balancing is now GA (ref)
• Enable CoreDNS as a DNS plugin for Kubernetes (ref)
• Azure Go SDK is now GA (#63063, @feiskyer)
• ClusterRole aggregation is now GA (ref)
• CRI validation test suite is now GA (ref)
• StorageObjectInUseProtection is now GA (ref) and (ref)

Graduated to Beta

• Supporting out-of-tree/external cloud providers is now considered beta (ref)
• Resizing PersistentVolumes after pod restart is now considered beta. (ref)
• sysctl support is now considered beta (ref)
• Support for Azure Virtual Machine Scale Sets is now considered beta. (ref)
• Azure support for Cluster Autoscaler is now considered beta. (ref)
• The ability to limit a node's access to the API is now considered beta. (ref)
• CustomResource versioning is now considered beta. (ref)
• Windows container configuration in CRI is now considered beta (ref)
• CRI logging and stats are now considered beta (ref)
• The dynamic Kubelet config feature is now beta, and the DynamicKubeletConfig feature gate is on by default. In order to use dynamic Kubelet config, ensure that the Kubelet's --dynamic-config-dir option is set. (#64275, @mtaufen)
• The Sysctls experimental feature has been promoted to beta (enabled by default via the Sysctls feature flag). PodSecurityPolicy and Pod objects now have fields for specifying and controlling sysctls. Alpha sysctl annotations will be ignored by 1.11+ kubelets. All alpha sysctl annotations in existing deployments must be converted to API fields to be effective. (#6371, @ingvagabund)
• Volume expansion is now considered Beta. (#64288, @gnufied)
• CRI container log rotation is now considered beta, and is enabled by default. (#64046, @yujuhong)
• The PriorityClass API has been promoted to scheduling.k8s.io/v1beta1 (#63100, @ravisantoshgudimetla)
• The priorityClass feature is now considered beta. (#63724, @ravisantoshgudimetla)
• client-go: credential exec plugins is now considered beta. (#64482, @ericchiang)
• Subresources for custom resources is now considered beta and enabled by default. With this, updates to the /status subresource will disallow updates to all fields other than .status (not just .spec and .metadata as before). Also, required can be used at the root of the CRD OpenAPI validation schema when the /status subresource is enabled. (#63598, @nikhita)

New alpha features

• kube-scheduler can now schedule DaemonSet pods (ref)
• You can now resize PersistentVolumes without taking them offline (ref)
• You can now set a maximum volume count (ref)
• You can now do environment variable expansion in a subpath mount. (ref)
• You can now run containers in a pod as a particular group. (ref) You can now bind tokens to service requests. (ref)
• The --experimental-qos-reserve kubelet flags has been replaced by the alpha level --qos-reserved flag or the QOSReserved field in the kubeletconfig, and requires the QOSReserved feature gate to be enabled. (#62509, @sjenning)

Other Notable Changes

SIG API Machinery

• Orphan delete is now supported for custom resources. (#63386, @roycaihw)
• Metadata of CustomResources is now pruned and schema-checked during deserialization of requests and when read from etcd. In the former case, invalid meta data is rejected, in the later it is dropped from the CustomResource objects. (#64267, @sttts)
• The kube-apiserver openapi doc now includes extensions identifying APIService and CustomResourceDefinition kinds (#64174, @liggitt)
• CustomResourceDefinitions Status subresource now supports GET and PATCH (#63619, @roycaihw)
• When updating /status subresource of a custom resource, only the value at the .status subpath for the update is considered. (#63385, @CaoShuFeng)
• Added a way to pass extra arguments to etcd. The these extra arguments can be used to adjust runtime configuration like heartbeat interval etc. (#63961, @mborsz)
• Added Establishing Controller on CRDs to avoid race between Established condition and CRs actually served. In HA setups, the Established condition is delayed by 5 seconds. (#63068, @xmudrii)
• Added spec.additionalPrinterColumns to CRDs to define server side printing columns. (#60991, @sttts)
• Added CRD Versioning with NOP converter (#63830, @mbohlool)
• Allow "required" and "description" to be used at the CRD OpenAPI validation schema root when the /status subresource is enabled. (#63533, @sttts)
• Etcd health checks by the apiserver now ensure the apiserver can connect to and exercise the etcd API. (#65027, @liggitt) api- machinery
• The deprecated --service-account-private-key-file flag has been removed from the cloud-controller-manager. The flag is still present and supported in the kube-controller-manager. (#65182, @liggitt)
• Webhooks for the mutating admission controller now support the "remove" operation. (#64255, @rojkov) sig-API machinery
• The CRD OpenAPI v3 specification for validation now allows additionalProperties, which are mutually exclusive to properties. (#62333, @sttts)
• Added the apiserver configuration option to choose the audit output version. (#60056, @crassirostris)
• Created a new dryRun query parameter for mutating endpoints. If the parameter is set, then the query will be rejected, as the feature is not implemented yet. This will allow forward compatibility with future clients; otherwise, future clients talking with older apiservers might end up modifying a resource even if they include the dryRun query parameter. (#63557, @apelisse)
• list/watch API requests with a fieldSelector that specifies metadata.name can now be authorized as requests for an individual named resource (#63469, @wojtek-t)
• Exposed /debug/flags/v to allow dynamically set glog logging level. For example, to change glog level to 3, send a PUT request such as curl -X PUT http://127.0.0.1:8080/debug/flags/v -d "3". (#63777, @hzxuzhonghu)
• Exec authenticator plugin supports TLS client certificates. (#61803, @awly)
• The serverAddressByClientCIDRs property in metav1.APIGroup(discovery API) is now optional instead of required. (#61963, @roycaihw)
• apiservices/status and certificatesigningrequests/status now support GET and PATCH (#64063, @roycaihw)
• APIServices with kube-like versions (e.g. v1, v2beta1, etc.) will be sorted appropriately within each group. (#64004, @mbohlool)
• Event object references with apiversion will now that value. (#63913, @deads2k)
• Fixes the kubernetes.default.svc loopback service resolution to use a loopback configuration. (#62649, @liggitt)

SIG Apps

• Added generators for apps/v1 deployments. (#61288, @ayushpateria)

SIG Auth

• RBAC information is now included in audit logs via audit.Event annotations:
  • authorization.k8s.io/decision = {allow, forbid}
  • authorization.k8s.io/reason = human-readable reason for the decision (#58807, @CaoShuFeng)
• kubectl certificate approve|deny will not modify an already approved or denied CSR unless the --force flag is provided. (#61971, @smarterclayton)
• The --bootstrap-kubeconfig argument to Kubelet previously created the first bootstrap client credentials in the certificates directory as kubelet-client.key and kubelet-client.crt. Subsequent certificates created by cert rotation were created in a combined PEM file that was atomically rotated as kubelet-client-DATE.pem in that directory, which meant clients relying on the node.kubeconfig generated by bootstrapping would never use a rotated cert. The initial bootstrap certificate is now generated into the cert directory as a PEM file and symlinked to kubelet-client-current.pem so that the generated kubeconfig remains valid after rotation. (#62152, @smarterclayton)
• Owner references can now be set during creation, even if the user doesn't have deletion power (#63403, @deads2k)
• Laid the groundwork for OIDC distributed claims handling in the apiserver authentication token checker. A distributed claim allows the OIDC provider to delegate a claim to a separate URL. (ref). (#63213, @filmil)
• RBAC: all configured authorizers are now checked to determine if an RBAC role or clusterrole escalation (setting permissions the user does not currently have via RBAC) is allowed. (#56358, @liggitt)
• kube-apiserver: OIDC authentication now supports requiring specific claims with --oidc-required-claim=<claim>=<value> Previously, there was no mechanism for a user to specify claims in the OIDC authentication process that were requid to be present in the ID Token with an expected value. This version now makes it possible to require claims support for the OIDC authentication. It allows users to pass in a --oidc-required-claims flag, and key=value pairs in the API config, which will ensure that the specified required claims are checked against the ID Token claims. (#62136, @rithujohn191)
• Included the list of security groups when failing with the errors that more than one is tagged. (#58874, @sorenmat)
• Added proxy for container streaming in kubelet for streaming auth. (#64006, @Random-Liu)
• PodSecurityPolicy admission information has been added to audit logs. (#58143, @CaoShuFeng)
• TokenRequests now are required to have an expiration duration between 10 minutes and 2^32 seconds. (#63999, @mikedanese)
• The NodeRestriction admission plugin now prevents kubelets from modifying/removing taints applied to their Node API object. (#63167, @liggitt)
• authz: nodes should not be able to delete themselves (#62818, @mikedanese)

SIG Autoscaling

• A cluster-autoscaler ClusterRole is added to cover only the functionality required by Cluster Autoscaler and avoid abusing system:cluster-admin role. Cloud providers other than GCE might want to update their deployments or sample yaml files to reuse the role created via add-on. (#64503, @kgolab)

SIG Azure

• The Azure cloud provider now supports standard SKU load balancer and public IP. excludeMasterFromStandardLB defaults to true, which means master nodes are excluded from the standard load balancer. Also note that because all nodes (except master) are added as loadbalancer backends, the standard load balancer doesn't work with the service.beta.kubernetes.io/azure-load-balancer-mode annotation. (#61884, #62707, @feiskyer)
• The Azure cloud provider now supports specifying allowed service tags by the service.beta.kubernetes.io/azure-allowed-service-tags annotation. (#61467, @feiskyer)
• You can now change the size of an azuredisk PVC using kubectl edit pvc pvc-azuredisk. Note that this operation will fail if the volume is already attached to a running VM. (#64386, @andyzhangx)
• Block device support has been added for azure disk. (#63841, @andyzhangx)
• Azure VM names can now contain the underscore (_) character (#63526, @djsly)
• Azure disks now support external resource groups. (#64427, @andyzhangx)
• Added reason message logs for non-existent Azure resources. (#64248, @feiskyer)

SIG CLI

• You can now use the base64decode function in kubectl go templates to decode base64-encoded data, such as kubectl get secret SECRET -o go-template='{{ .data.KEY | base64decode }}'. (#60755, @glb)
• kubectl patch now supports --dry-run. (#60675, @timoreimann)
• The global flag --match-server-version is now global. kubectl version will respect it. (#63613, @deads2k)
• kubectl will list all allowed print formats when an invalid format is passed. (#64371, @CaoShuFeng)
• The global flag "context" now gets applied to kubectl config view --minify. In previous versions, this command was only available for current-context. Now it will be easier for users to view other non current contexts when minifying. (#64608, @dixudx)
• kubectl apply --prune supports CronJob resources. (#62991, @tomoe)
• The --dry-run flag has been enabled for kubectl auth reconcile (#64458, @mrogers950)
• kubectl wait is a new command that allows waiting for one or more resources to be deleted or to reach a specific condition. It adds a kubectl wait --for=[delete|condition=condition-name] resource/string command. (#64034, @deads2k)
• kubectl auth reconcile only works with rbac.v1; all the core helpers have been switched over to use the external types. (#63967, @deads2k)
• kubectl and client-go now detect duplicated names for user, cluster and context when loading kubeconfig and report this condition as an error. (#60464, @roycaihw)
• Added 'UpdateStrategyType' and 'RollingUpdateStrategy' to 'kubectl describe sts' command output. (#63844, @tossmilestone)
• Initial Korean translation for kubectl has been added. (#62040, @ianychoi)
• kubectl cp now supports completion. (#60371, @superbrothers)
• The shortcuts that were moved server-side in at least 1.9 have been removed from being hardcoded in kubectl. This means that the client-based restmappers have been moved to client-go, where everyone who needs them can have access. (#63507, @deads2k)
• When using kubectl delete with selection criteria, the defaults to is now to ignore "not found" errors. Note that this does not apply when deleting a speciic resource. (#63490, @deads2k)
• kubectl create [secret | configmap] --from-file now works on Windows with fully-qualified paths (#63439, @liggitt)
• Portability across systems has been increased by the use of /usr/bin/env in all script shebangs. (#62657, @matthyx)
• You can now use kubectl api-resources to discover resources. (#42873, @xilabao)
• You can now display requests/limits of extended resources in node allocated resources. (#46079, @xiangpengzhao)
• The --remove-extra-subjects and --remove-extra-permissions flags have been enabled for kubectl auth reconcile (#64541, @mrogers950)
• kubectl now has improved compatibility with older servers when creating/updating API objects (#61949, @liggitt)
• kubectl apply view/edit-last-applied now supports completion. (#60499, @superbrothers)

SIG Cluster Lifecycle

• kubeadm: The :Etcd struct has been refactored in the v1alpha2 API. All the options now reside under either .Etcd.Local or .Etcd.External. Automatic conversions from the v1alpha1 API are supported. (#64066, @luxas)
• kubeadm now uses an upgraded API version for the configuration file, kubeadm.k8s.io/v1alpha2. kubeadm in v1.11 will still be able to read v1alpha1 configuration, and will automatically convert the configuration to v1alpha2, both internally and when storing the configuration in the ConfigMap in the cluster. (#63788, @luxas)
• Phase kubeadm alpha phase kubelet has been added to support dynamic kubelet configuration in kubeadm. (#57224, @xiangpengzhao)
• The kubeadm config option API.ControlPlaneEndpoint has been extended to take an optional port, which may differ from the apiserver's bind port. (#62314, @rjosephwright)
• The --cluster-name parameter has been added to kubeadm init, enabling users to specify the cluster name in kubeconfig. (#60852, @karan)
• The logging feature for kubeadm commands now supports a verbosity setting. (#57661, @vbmade2000)
• kubeadm now has a join timeout that can be controlled via the discoveryTimeout config option. This option is set to 5 minutes by default. (#60983, @rosti)
• Added the writable boolean option to kubeadm config. This option works on a per-volume basis for ExtraVolumes config keys. (#60428, @rosti)
• Added a new kubeadm upgrade node config command. (#64624, @luxas)
• kubeadm now makes the CoreDNS container more secure by dropping (root) capabilities and improves the integrity of the container by running the whole container in read-only. (#64473, @nberlee)
• kubeadm now detects the Docker cgroup driver and starts the kubelet with the matching driver. This eliminates a common error experienced by new users in when the Docker cgroup driver is not the same as the one set for the kubelet due to different Linux distributions setting different cgroup drivers for Docker, making it hard to start the kubelet properly. (#64347, @neolit123)
• Added a 'kubeadm config migrate' command to convert old API types to their newer counterparts in the new, supported API types. This is just a client-side tool; it just executes locally without requiring a cluster to be running, operating in much the same way as a Unix pipe that upgrades config files. (#64232, @luxas)
• kubeadm will now pull required images during preflight checks if it cannot find them on the system. (#64105, @chuckha)
• "kubeadm init" now writes a structured and versioned kubelet ComponentConfiguration file to /var/lib/kubelet/config.yaml and an environment file with runtime flags that you can source in the systemd kubelet dropin to /var/lib/kubelet/kubeadm-flags.env. (#63887, @luxas)
• A kubeadm config print-default command has now been added. You can use this command to output a starting point when writing your own kubeadm configuration files. (#63969, @luxas)
• Updated kubeadm's minimum supported kubernetes in v1.11.x to 1.10 (#63920, @dixudx)
• Added the kubeadm upgrade diff command to show how static pod manifests will be changed by an upgrade. This command shows the changes that will be made to the static pod manifests before applying them. This is a narrower case than kubeadm upgrade apply --dry-run, which specifically focuses on the static pod manifests. (#63930, @liztio)
• The kubeadm config images pull command can now be used to pull container images used by kubeadm. (#63833, @chuckha)
• kubeadm will now deploy CoreDNS by default instead of KubeDNS (#63509, @detiber)
• Preflight checks for kubeadm no longer validate custom kube-apiserver, kube-controller-manager and kube-scheduler arguments. (#63673, @chuckha)
• Added a kubeadm config images list command that lists required container images for a kubeadm install. (#63450, @chuckha)
• You can now use kubeadm token specifying --kubeconfig. In this case, kubeadm searches the current user home path and the environment variable KUBECONFIG for existing files. If provided, the --kubeconfig flag will be honored instead. (#62850, @neolit123)
• kubeadm now sets peer URLs for the default etcd instance. Previously we left the defaults, which meant the peer URL was unsecured.
• Kubernetes now packages crictl in a cri-tools deb and rpm package. (#64836, @chuckha)
• kubeadm now prompts the user for confirmation when resetting a master node. (#59115, @alexbrand)
• kubead now creates kube-proxy with a toleration to run on all nodes, no matter the taint. (#62390, @discordianfish)
• kubeadm now sets the kubelet --resolv-conf flag conditionally on init. (#64665, @stealthybox)
• Added ipset and udevadm to the hyperkube base image. (#61357, @rphillips)

SIG GCP

• Kubernetes clusters on GCE now have crictl installed. Users can use it to help debug their nodes. See the crictl documentation for details. (#63357, @Random-Liu)
• cluster/kube-up.sh now provisions a Kubelet config file for GCE via the metadata server. This file is installed by the corresponding GCE init scripts. (#62183, @mtaufen)
• GCE: Update cloud provider to use TPU v1 API (#64727, @yguo0905)
• GCE: Bump GLBC version to 1.1.1 - fixing an issue of handling multiple certs with identical certificates. (#62751, @nicksardo)

SIG Instrumentation

• Added prometheus cluster monitoring addon to kube-up. (#62195, @serathius)
• Kubelet now exposes a new endpoint, /metrics/probes, which exposes a Prometheus metric containing the liveness and/or readiness probe results for a container. (#61369, @rramkumar1)

SIG Network

• The internal IP address of the node is now added as additional information for kubectl. (#57623, @dixudx)
• NetworkPolicies can now target specific pods in other namespaces by including both a namespaceSelector and a podSelector in the same peer element. (#60452, @danwinship)
• CoreDNS deployment configuration now uses the k8s.gcr.io imageRepository. (#64775, @rajansandeep)
• kubelet's --cni-bin-dir option now accepts multiple comma-separated CNI binary directory paths, which are searched for CNI plugins in the given order. (#58714, @dcbw)
• You can now use --ipvs-exclude-cidrs to specify a list of CIDR's which the IPVS proxier should not touch when cleaning up IPVS rules. (#62083, @rramkumar1)
• You can now receive node DNS info with the --node-ip flag, which adds ExternalDNS, InternalDNS, and ExternalIP to kubelet's output. (#63170, @micahhausler)
• You can now have services that listen on the same host ports on different interfaces by specifying --nodeport-addresses. (#62003, @m1093782566)
• Added port-forward examples for service

SIG Node

• CRI: The container log path has been changed from containername_attempt#.log to containername/attempt#.log (#62015, @feiskyer)
• Introduced the ContainersReady condition in Pod status. (#64646, @freehan)
• Kubelet will now set extended resource capacity to zero after it restarts. If the extended resource is exported by a device plugin, its capacity will change to a valid value after the device plugin re-connects with the Kubelet. If the extended resource is exported by an external component through direct node status capacity patching, the component should repatch the field after kubelet becomes ready again. During the time gap, pods previously assigned with such resources may fail kubelet admission but their controller should create new pods in response to such failures. (#64784, @jiayingz) node
• You can now use a security context with Windows containers (#64009, @feiskyer)
• Added e2e regression tests for kubelet security. (#64140, @dixudx)
• The maximum number of images the Kubelet will report in the Node status can now be controlled via the Kubelet's --node-status-max-images flag. The default (50) remains the same. (#64170, @mtaufen)
• The Kubelet now exports metrics that report the assigned (node_config_assigned), last-known-good (node_config_last_known_good), and active (node_config_active) config sources, and a metric indicating whether the node is experiencing a config-related error (node_config_error). The config source metrics always report the value 1, and carry the node_config_name, node_config_uid, node_config_resource_version, and node_config_kubelet_key labels, which identify the config version. The error metric reports 1 if there is an error, 0 otherwise. (#57527, @mtaufen)
• You now have the ability to quota resources by priority. (#57963, @vikaschoudhary16)
• The gRPC max message size in the remote container runtime has been increased to 16MB. (#64672, @mcluseau)
• Added a feature gate for the plugin watcher. (#64605, @vikaschoudhary16)
• The status of dynamic Kubelet config is now reported via Node.Status.Config, rather than the KubeletConfigOk node condition. (#63314, @mtaufen)
• You must now specify Node.Spec.ConfigSource.ConfigMap.KubeletConfigKey when using dynamic Kubelet config to tell the Kubelet which key of the ConfigMap identifies its config file. (#59847, @mtaufen)
• The dynamic Kubelet config feature will now update the config in the event of a ConfigMap mutation, which reduces the chance for silent config skew. Only name, namespace, and kubeletConfigKey may now be set in Node.Spec.ConfigSource.ConfigMap. The least disruptive pattern for config management is still to create a new ConfigMap and incrementally roll out a new Node.Spec.ConfigSource. (#63221, @mtaufen)
• Change seccomp annotation from "docker/default" to "runtime/default" (#62662, @wangzhen127)
• The node authorizer now automatically sets up rules for Node.Spec.ConfigSource when the DynamicKubeletConfig feature gate is enabled. (#60100, @mtaufen)
• CRI now defines mounting behavior. If the host path doesn't exist, the runtime should return an error. If the host path is a symlink, the runtime should follow the symlink and mount the real destination to the container. (#61460, @feiskyer)

SIG OpenStack

• Provide a meaningful error message in the openstack cloud provider when no valid IP address can be found for a node, rather than just the first address of the node, which leads to a load balancer error if that address is a hostname or DNS name instead of an IP address. (#64318, @gonzolino)
• Restored the pre-1.10 behavior of the openstack cloud provider, which uses the instance name as the Kubernetes Node name. This requires instances be named with RFC-1123 compatible names. (#63903, @liggitt)
• Kubernetes will try to read the openstack auth config from the client config and fall back to read from the environment variables if the auth config is not available. (#60200, @dixudx)

SIG Scheduling

• Schedule DaemonSet Pods in scheduler, rather than the Daemonset controller. (#63223, @k82cn)
• Added MatchFields to NodeSelectorTerm; in 1.11, it only supports metadata.name. (#62002, @k82cn)
• kube-scheduler now has the --write-config-to flag so that Scheduler canwritets default configuration to a file. (#62515, @resouer)
• Performance of the affinity/anti-affinity predicate for the default scheduler has been significantly improved. (#62211, @bsalamat)
• The 'scheduling_latency_seconds' metric into has been split into finer steps (predicate, priority, preemption). (#65306, @shyamjvs)
• Scheduler now has a summary-type metric, 'scheduling_latency_seconds'. (#64838, @krzysied)
• nodeSelector.matchFields (node's metadata.node) is now supported in scheduler. (#62453, @k82cn)
• Added a parametrizable priority function mapping requested/capacity ratio to priority. This function is disabled by default and can be enabled via the scheduler policy config file. (#63929, @losipiuk)
• System critical priority classes are now automatically added at cluster boostrapping. (#60519, @bsalamat)

SIG Storage

• AWS EBS, Azure Disk, GCE PD and Ceph RBD volume plugins now support dynamic provisioning of raw block volumes. (#64447, @jsafrane)
• gitRepo volumes in pods no longer require git 1.8.5 or newer; older git versions are now supported. (#62394, @jsafrane)
• Added support for resizing Portworx volumes. (#62308, @harsh-px)
• Added block volume support to Cinder volume plugin. (#64879, @bertinatto)
• Provided API support for external CSI storage drivers to support block volumes. (#64723, @vladimirvivien)
• Volume topology aware dynamic provisioning for external provisioners is now supported. (#63193, @lichuqiang)
• Added a volume projection that is able to project service account tokens. (#62005, @mikedanese)
• PodSecurityPolicy now supports restricting hostPath volume mounts to be readOnly and under specific path prefixes (#58647, @jhorwit2)
• Added StorageClass API to restrict topologies of dynamically provisioned volumes. (#63233, @lichuqiang)
• Added Alpha support for dynamic volume limits based on node type (#64154, @gnufied)
• AWS EBS volumes can be now used as ReadOnly in pods. (#64403, @jsafrane)
• Basic plumbing for volume topology aware dynamic provisionin has been implemented. (#63232, @lichuqiang)
• Changed ext3/ext4 volume creation to not reserve any portion of the volume for the root user. When creating ext3/ext4 volume, mkfs defaults to reserving 5% of the volume for the super-user (root). This patch changes the mkfs to pass -m0 to disable this setting. (#64102, @atombender)
• Added support for NFS relations on kubernetes-worker charm. (#63817, @hyperbolic2346)
• Implemented kubelet side online file system resizing (#62460, @mlmhl)
• Generated subpath name from Downward API env (#49388, @kevtaylor)

SIG vSphere

• Added a mechanism in vSphere Cloud Provider to get credentials from Kubernetes secrets, rather than the plain text vsphere.conf file.(#63902, @abrarshivani)
• vSphere Cloud Provider: added SAML token authentication support (#63824, @dougm)

SIG Windows

• Added log and fs stats for Windows containers. (#62266, @feiskyer)
• Added security contexts for Windows containers. #64009, (@feiskyer)
• Added local persistent volumes for Windows containers. (#62012, @andyzhangx) and fstype for Azure disk (#61267, @andyzhangx)
• Improvements in Windows Server version 1803 also bring new storage functionality to Kubernetes v1.11, including:
  • Volume mounts for ConfigMap and Secret
  • Flexvolume plugins for SMB and iSCSI storage are also available out-of-tree at Microsoft/K8s-Storage-Plugins
• Setup dns servers and search domains for Windows Pods in dockershim. Docker EE version >= 17.10.0 is required for propagating DNS to containers. (#63905, @feiskyer)

Additional changes

• Extended the Stackdriver Metadata Agent by adding a new Deployment for ingesting unscheduled pods and services. (#62043, @supriyagarg)
• Added all kinds of resource objects' statuses in HPA description. (#59609, @zhangxiaoyu-zidif)
• Implemented preemption for extender with a verb and new interface (#58717, @resouer)
• Updated nvidia-gpu-device-plugin DaemonSet config to use RollingUpdate updateStrategy instead of OnDelete. (#64296, @mindprince)
• increased grpc client default response size. (#63977, @runcom)
• Applied pod name and namespace labels to pod cgroup in cAdvisor metrics (#63406, @derekwaynecarr)
• [fluentd-gcp addon] Use the logging agent's node name as the metadata agent URL. (#63353, @bmoyles0117)
• The new default value for the --allow-privileged parameter of the Kubernetes-worker charm has been set to true based on changes which went into the Kubernetes 1.10 release. Before this change the default value was set to false. If you're installing Canonical Kubernetes you should expect this value to now be true by default and you should now look to use PSP (pod security policies). (#64104, @CalvinHartwell)

External Dependencies

• Default etcd server version is v3.2.18 compared with v3.1.12 in v1.10 (#61198)
• Rescheduler is v0.4.0, compared with v0.3.1 in v1.10 (#65454)
• The validated docker versions are the same as for v1.10: 1.11.2 to 1.13.1 and 17.03.x (ref)
• The Go version is go1.10.2, as compared to go1.9.3 in v1.10. (#63412)
• The minimum supported go is the same as for v1.10: go1.9.1. (#55301)
• CNI is the same as v1.10: v0.6.0 (#51250)
• CSI is updated to 0.3.0 as compared to 0.2.0 in v1.10. (#64719)
• The dashboard add-on is the same as v1.10: v1.8.3. (#517326)
• Bump Heapster to v1.5.2 as compared to v1.5.0 in v1.10 (#61396)
• Updates Cluster Autoscaler version to v1.3.0 from v1.2.0 in v1.10. See release notes for details. (#65219)
• Kube-dns has been updated to v1.14.10, as compared to v1.14.8 in v1.10 (#62676)
• Influxdb is unchanged from v1.10: v1.3.3 (#53319)
• Grafana is unchanged from v1.10: v4.4.3 (#53319)
• CAdvisor is v0.30.1, as opposed to v0.29.1 in v1.10 (#64987)
• fluentd-gcp-scaler is unchanged from v1.10: v0.3.0 (#61269)
• fluentd in fluentd-es-image is unchanged from 1.10: v1.1.0 (#58525)
• fluentd-elasticsearch is unchanged from 1.10: v2.0.4 (#58525)
• fluentd-gcp is unchanged from 1.10: v3.0.0. (#60722)
• Ingress glbc is unchanged from 1.10: v1.0.0 (#61302)
• OIDC authentication is unchanged from 1.10: coreos/go-oidc v2 (#58544)
• Calico is unchanged from 1.10: v2.6.7 (#59130)
• hcsshim has been updated to v0.6.11 (#64272)
• gitRepo volumes in pods no longer require git 1.8.5 or newer; older git versions are now supported. (#62394)
• Update crictl on GCE to v1.11.0. (#65254)
• CoreDNS is now v1.1.3 (#64258)
• Setup dns servers and search domains for Windows Pods in dockershim. Docker EE version >= 17.10.0 is required for propagating DNS to containers. (#63905)
• Update version of Istio addon from 0.5.1 to 0.8.0. See full Istio release notes.(#64537)
• Update cadvisor godeps to v0.30.0 (#64800)
• Update event-exporter to version v0.2.0 that supports old (gke_container/gce_instance) and new (k8s_container/k8s_node/k8s_pod) stackdriver resources. (#63918)
• Rev the Azure SDK for networking to 2017-06-01 (#61955)

Bug Fixes

• Fixed spurious "unable to find api field" errors patching custom resources (#63146, @liggitt)
• Nodes are not deleted from kubernetes anymore if node is shutdown in Openstack. (#59931, @zetaab)
• Re-enabled nodeipam controller for external clouds. Re-enables nodeipam controller for external clouds. Also does a small refactor so that we don't need to pass in allocateNodeCidr into the controller. (#63049, @andrewsykim)
• Fixed a configuration error when upgrading kubeadm from 1.9 to 1.10+; Kubernetes must have the same major and minor versions as the kubeadm library. (#62568, @liztio)
• kubectl no longer renders a List as suffix kind name for CRD resources (#62512, @dixudx)
• Restored old behavior to the --template flag in get.go. In old releases, providing a --template flag value and no --output value implicitly assigned a default value ("go-template") to --output, printing using the provided template argument. (#65377,@juanvallejo)
• Ensured cloudprovider.InstanceNotFound is reported when the VM is not found on Azure (#61531, @feiskyer)
• Kubernetes version command line parameter in kubeadm has been updated to drop an unnecessary redirection from ci/latest.txt to ci-cross/latest.txt. Users should know exactly where the builds are stored on Google Cloud storage buckets from now on. For example for 1.9 and 1.10, users can specify ci/latest-1.9 and ci/latest-1.10 as the CI build jobs what build images correctly updates those. The CI jobs for master update the ci-cross/latest location, so if you are looking for latest master builds, then the correct parameter to use would be ci-cross/latest. (#63504, @dims)
• Fixes incompatibility with custom scheduler extender configurations specifying bindVerb (#65424, @liggitt)
• kubectl built for darwin from darwin now enables cgo to use the system-native C libraries for DNS resolution. Cross-compiled kubectl (e.g. from an official kubernetes release) still uses the go-native netgo DNS implementation. (#64219, @ixdy)
• API server properly parses propagationPolicy as a query parameter sent with a delete request (#63414, @roycaihw)
• Corrected a race condition in bootstrapping aggregated cluster roles in new HA clusters (#63761, @liggitt)
• kubelet: fix hangs in updating Node status after network interruptions/changes between the kubelet and API server (#63492, @liggitt)
• Added log and fs stats for Windows containers (#62266, @feiskyer)
• Fail fast if cgroups-per-qos is set on Windows (#62984, @feiskyer)
• Minor fix for VolumeZoneChecker predicate, storageclass can be in annotation and spec. (#63749, @wenlxie)
• Fixes issue for readOnly subpath mounts for SELinux systems and when the volume mountPath already existed in the container image. (#64351, @msau42)
• Fixed CSI gRPC connection leak during volume operations. (#64519, @vladimirvivien)
• Fixed error reporting of CSI volumes attachment. (#63303, @jsafrane)
• Fixed SELinux relabeling of CSI volumes. (#64026, @jsafrane)
• Fixed detach of already detached CSI volumes. (#63295, @jsafrane)
• fix rbd device works at block mode not get mapped to container (#64555, @wenlxie)
• Fixed an issue where Portworx PVCs remain in pending state when created using a StorageClass with empty parameters (#64895, @harsh-px) storage
• FIX: The OpenStack cloud providers DeleteRoute method fails to delete routes when it can’t find the corresponding instance in OpenStack. (#62729, @databus23)
• [fluentd-gcp addon] Increase CPU limit for fluentd to 1 core to achieve 100kb/s throughput. (#62430, @bmoyles0117)
• GCE: Fixed operation polling to adhere to the specified interval. Furthermore, operation errors are now returned instead of ignored. (#64630, @nicksardo)
• Included kms-plugin-container.manifest to master nifests tarball. (#65035, @immutableT)
• Fixed missing nodes lines when kubectl top nodes (#64389, @yue9944882) sig-cli
• Fixed kubectl drain --timeout option when eviction is used. (#64378, @wrdls) sig-cli
• Fixed kubectl auth can-i exit code. It will return 1 if the user is not allowed and 0 if it's allowed. (#59579, @fbac)
• Fixed data loss issue if using existing azure disk with partitions in disk mount (#63270, @andyzhangx)
• Fixed azure file size grow issue (#64383, @andyzhangx)
• Fixed SessionAffinity not updated issue for Azure load balancer (#64180, @feiskyer)
• Fixed kube-controller-manager panic while provisioning Azure security group rules (#64739, @feiskyer)
• Fixed API server panic during concurrent GET or LIST requests with non-empty resourceVersion. (#65092, @sttts)
• Fixed incorrect OpenAPI schema for CustomResourceDefinition objects (#65256, @liggitt)
• Fixed issue where PersistentVolume.NodeAffinity.NodeSelectorTerms were ANDed instead of ORed. (#62556, @msau42)
• Fixed potential infinite loop that can occur when NFS PVs are recycled. (#62572, @joelsmith)
• Fixed column alignment when kubectl get is used with custom columns from OpenAPI schema (#56629, @luksa)
• kubectl: restore the ability to show resource kinds when displaying multiple objects (#61985, @liggitt)
• Fixed a panic in kubectl run --attach ... when the api server failed to create the runtime object (due to name conflict, PSP restriction, etc.) (#61713, @mountkin)
• kube-scheduler has been fixed to use --leader-elect option back to true (as it was in previous versions) (#59732, @dims)
• kubectl: fixes issue with -o yaml and -o json omitting kind and apiVersion when used with --dry-run (#61808, @liggitt)
• Ensure reasons end up as comments in kubectl edit. (#60990, @bmcstdio)
• Fixes issue where subpath readOnly mounts failed (#63045, @msau42)
• Fix stackdriver metrics for node memory using wrong metric type (#63535, @serathius)
• fix mount unmount failure for a Windows pod (#63272, @andyzhangx)

General Fixes and Reliability

• Fixed a regression in kube-scheduler to properly load client connection information from a --config file that references a kubeconfig file. (#65507, @liggitt)
• Fix regression in v1.JobSpec.backoffLimit that caused failed Jobs to be restarted indefinitely. (#63650, @soltysh)
• fixes a potential deadlock in the garbage collection controller (#64235, @liggitt)
• fix formatAndMount func issue on Windows (#63248, @andyzhangx)
• Fix issue of colliding nodePorts when the cluster has services with externalTrafficPolicy=Local (#64349, @nicksardo)
• fixes a panic applying json patches containing out of bounds operations (#64355, @liggitt)
• Fix incorrectly propagated ResourceVersion in ListRequests returning 0 items. (#64150, @wojtek-t)
• GCE: Fix to make the built-in kubernetes service properly point to the master's load balancer address in clusters that use multiple master VMs. (#63696, @grosskur)
• Fixes fake client generation for non-namespaced subresources (#60445, @jhorwit2)
• Schedule even if extender is not available when using extender (#61445, @resouer)
• Fix panic create/update CRD when mutating/validating webhook configured. (#61404, @hzxuzhonghu)
• Pods requesting resources prefixed with *kubernetes.io will remain unscheduled if there are no nodes exposing that resource. (#61860, @mindprince)
• fix scheduling policy on ConfigMap breaks without the --policy-configmap-namespace flag set (#61388, @zjj2wry)
• Bugfix for erroneous upgrade needed messaging in kubernetes worker charm. (#60873, @wwwtyro)
• Fix inter-pod anti-affinity check to consider a pod a match when all the anti-affinity terms match. (#62715, @bsalamat)
• Pod affinity nodeSelectorTerm.matchExpressions may now be empty, and works as previously documented: nil or empty matchExpressions matches no objects in scheduler. (#62448, @k82cn)
• Fix an issue in inter-pod affinity predicate that cause affinity to self being processed correctly (#62591, @bsalamat)
• fix WaitForAttach failure issue for azure disk (#62612, @andyzhangx)
• Fix user visible files creation for windows (#62375, @feiskyer)
• Fix machineID getting for vmss nodes when using instance metadata (#62611, @feiskyer)
• Fix Forward chain default reject policy for IPVS proxier (#62007, @m1093782566)
• fix nsenter GetFileType issue in containerized kubelet (#62467, @andyzhangx)
• Ensure expected load balancer is selected for Azure (#62450, @feiskyer)
• Resolves forbidden error when the daemon-set-controller cluster role access controllerrevisions resources. (#62146, @frodenas)
• fix incompatible file type checking on Windows (#62154, @dixudx)
• fix local volume absolute path issue on Windows (#620s18, @andyzhangx)
• fix the issue that default azure disk fsypte(ext4) does not work on Windows (#62250, @andyzhangx)
• Fixed bug in rbd-nbd utility when nbd is used. (#62168, @piontec)
• fix local volume issue on Windows (#62012, @andyzhangx)
• Fix a bug that fluentd doesn't inject container logs for CRI container runtimes (containerd, cri-o etc.) into elasticsearch on GCE. (#61818, @Random-Liu)
• flexvolume: trigger plugin init only for the relevant plugin while probe (#58519, @linyouchong)
• Fixed ingress issue with CDK and pre-1.9 versions of kubernetes. (#61859, @hyperbolic2346)
• Fixed racy panics when using fake watches with ObjectTracker (#61195, @grantr)
• Fixed mounting of UNIX sockets(and other special files) in subpaths (#61480, @gnufscied)
• Fixed #61123 by triggering syncer.Update on all cases including when a syncer is created (#61124, @satyasm)
• Fixed data race in node lifecycle controller (#60831, @resouer)
• Fixed resultRun by resetting it to 0 on pod restart (#62853, @tony612)
• Fixed the liveness probe to use /bin/bash -c instead of /bin/bash c. (#63033, @bmoyles0117)
• Fixed scheduler informers to receive events for all the pods in the cluster. (#63003, @bsalamat)
• Fixed in vSphere Cloud Provider to handle upgrades from kubernetes version less than v1.9.4 to v1.9.4 and above. (#62919, @abrarshivani)
• Fixed error where config map for Metadata Agent was not created by addon manager. (#62909, @kawych)
• Fixed permissions to allow statefulset scaling for admins, editors, and viewers (#62336, @deads2k)
• GCE: Fixed for internal load balancer management resulting in backend services with outdated instance group links. (#62885, @nicksardo)
• Deployment will stop adding pod-template-hash labels/selector to ReplicaSets and Pods it adopts. Resources created by Deployments are not affected (will still have pod-template-hash labels/selector). (#61615, @janetkuo)
• Used inline func to ensure unlock is executed (#61644, @resouer)
• kubernetes-master charm now properly clears the client-ca-file setting on the apiserver snap (#61479, @hyperbolic2346)
• Bound cloud allocator to 10 retries with 100 ms delay between retries. (#61375, @satyasm)
• Respect fstype in Windows for azure disk (#61267, @andyzhangx)
• Unready pods will no longer impact the number of desired replicas when using horizontal auto-scaling with external metrics or object metrics. (#60886, @mattjmcnaughton)
• Removed unsafe double RLock in cpumanager (#62464, @choury)

Non-user-facing changes

• Remove UID mutation from request.context. (#63957, @hzxuzhonghu)
• Use Patch instead of Put to sync pod status. (#62306, @freehan)
• Allow env from resource with keys & updated tests (#60636, @PhilipGough)
• set EnableHTTPSTrafficOnly in azure storage account creation (#64957, @andyzhangx)
• New conformance test added for Watch. (#61424, @jennybuckley)
• Use DeleteOptions.PropagationPolicy instead of OrphanDependents in kubectl (#59851, @nilebox)
• Add probe based mechanism for kubelet plugin discovery (#63328, @vikaschoudhary16)
• keep pod state consistent when scheduler cache UpdatePod (#64692, @adohe)
• kubectl delete does not use reapers for removing objects anymore, but relies on server-side GC entirely (#63979, @soltysh)
• Updated default image for nginx ingress in CDK to match current Kubernetes docs. (#64285, @hyperbolic2346)
• Increase scheduler cache generation number monotonically in order to avoid collision and use of stale information in scheduler. (#63264, @bsalamat)
• Adding CSI driver registration code. (#64560, @sbezverk)
• Do not check vmSetName when getting Azure node's IP (#63541, @feiskyer)
• [fluentd-gcp addon] Update event-exporter image to have the latest base image. (#61727, @crassirostris)
• Make volume usage metrics available for Cinder (#62668, @zetaab)
• cinder volume plugin : When the cinder volume status is error, controller will not do attach and detach operation (#61082, @wenlxie)
• Allow user to scale l7 default backend deployment (#62685, @freehan)
• Add support to ingest log entries to Stackdriver against new "k8s_container" and "k8s_node" resources. (#62076, @qingling128)
• Disabled CheckNodeMemoryPressure and CheckNodeDiskPressure predicates if TaintNodesByCondition enabled (#60398, @k82cn)
• Support custom test configuration for IPAM performance integration tests (#61959, @satyasm)
• OIDC authentication now allows tokens without an "email_verified" claim when using the "email" claim. If an "email_verified" claim is present when using the "email" claim, it must be true. (#61508, @rithujohn191)
• Add e2e test for CRD Watch (#61025, @ayushpateria)
• Return error if get NodeStageSecret and NodePublishSecret failed in CSI volume plugin (#61096, @mlmhl)
• kubernetes-master charm now supports metrics server for horizontal pod autoscaler. (#60174, @hyperbolic2346)
• In a GCE cluster, the default HIRPIN_MODE is now "hairpin-veth". (#60166, @rramkumar1)
• Balanced resource allocation priority in scheduler to include volume count on node (#60525, @ravisantoshgudimetla)
• new dhcp-domain parameter to be used for figuring out the hostname of a node (#61890, @dims)
• Disable ipamperf integration tests as part of every PR verification. (#61863, @satyasm)
• Enable server-side print in kubectl by default, with the ability to turn it off with --server-print=false (#61477, @soltysh)
• Updated admission controller settings for Juju deployed Kubernetes clusters (#61427, @hyperbolic2346)
• Performance test framework and basic tests for the IPAM controller, to simulate behavior of the four supported modes under lightly loaded and loaded conditions, where load is defined as the number of operations to perform as against the configured kubernetes. (#61143, @satyasm)
• Removed always pull policy from the template for ingress on CDK. (#61598, @hyperbolic2346)
• make test-cmd now works on OSX. (#61393, @totherme)
• Conformance: ReplicaSet must be supported in the apps/v1 version. (#61367, @enisoc)
• Remove 'system' prefix from Metadata Agent rbac configuration (#61394, @kawych)
• Support new NODE_OS_DISTRIBUTION 'custom' on GCE on a new add event. (#61235, @yguo0905)
• include file name in the error when visiting files (#60919, @dixudx)
• Split PodPriority and PodPreemption feature gate (#62243, @resouer)
• Code generated for CRDs now passes go vet. (#62412, @bhcleek)
• "beginPort+offset" format support for port range which affects kube-proxy only (#58731, @yue9944882)
• Added e2e test for watch (#60331, @jennybuckley)
• add warnings on using pod-infra-container-image for remote container runtime (#62982, @dixudx)
• Mount additional paths required for a working CA root, for setups where /etc/ssl/certs doesn't contains certificates but just symlink. (#59122, @klausenbusk)
• Introduce truncating audit bacnd that can be enabled for existing backend to limit the size of individual audit events and batches of events. (#61711, @crassirostris)
• stop kubelet to cloud provider integration potentially wedging kubelet sync loop (#62543, @ingvagabund)
• Set pod status to "Running" if there is at least one container still reporting as "Running" status and others are "Completed". (#62642, @ceshihao)
• Fix memory cgroup notifications, and reduce associated log spam. (#63220, @dashpole)
• Remove never used NewCronJobControllerFromClient method (#59471, @dmathieu)

• v1.11.0-rc.3
• v1.11.0-rc.2
• v1.11.0-rc.1
• v1.11.0-beta.2
• v1.11.0-beta.1
• v1.11.0-alpha.2
• v1.11.0-alpha.1

v1.11.0-rc.3

Documentation

Downloads for v1.11.0-rc.3

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0-rc.2

Other notable changes

• Pass cluster_location argument to Heapster (#65176, @kawych)
• Fix concurrent map access panic (#65331, @dashpole)
  • Don't watch .mount cgroups to reduce number of inotify watches
  • Fix NVML initialization race condition
  • Fix brtfs disk metrics when using a subdirectory of a subvolume
• User can now use sudo crictl on GCE cluster. (#65389, @Random-Liu)

v1.11.0-rc.2

Documentation

Downloads for v1.11.0-rc.2

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0-rc.1

Other notable changes

• Prevents a kubectl delete hang when deleting controller managed lists (#65367, @deads2k)
• fixes a memory leak in the kube-controller-manager observed when large numbers of pods with tolerations are created/deleted (#65339, @liggitt)
• The "kubectl cp" command now supports path shortcuts (../) in remote paths. (#65189, @juanvallejo)
• Split 'scheduling_latency_seconds' metric into finer steps (predicate, priority, premption) (#65306, @shyamjvs)
• fixed incorrect OpenAPI schema for CustomResourceDefinition objects (#65256, @liggitt)

v1.11.0-rc.1

Documentation

Downloads for v1.11.0-rc.1

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0-beta.2

Action Required

• A cluster-autoscaler ClusterRole is added to cover only the functionality required by Cluster Autoscaler and avoid abusing system:cluster-admin role. (#64503, @kgolab)
  • action required: Cloud providers other than GCE might want to update their deployments or sample yaml files to reuse the role created via add-on.

Other notable changes

• The "kubectl cp" command now supports path shortcuts (../) in remote paths. (#65189, @juanvallejo)
• Update crictl on GCE to v1.11.0. (#65254, @Random-Liu)
• kubeadm: Use the release-1.11 branch by default (#65229, @luxas)
• Updates Cluster Autoscaler version to 1.3.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.0 (#65219, @aleksandra-malinowska)
• The deprecated --service-account-private-key-file flag has been removed from the cloud-controller-manager. The flag is still present and supported in the kube-controller-manager. (#65182, @liggitt)
• Update Cluster Autoscaler to v1.3.0-beta.2. Release notes for this version: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.0-beta.2 (#65148, @aleksandra-malinowska)
• Fixed API server panic during concurrent GET or LIST requests with non-empty resourceVersion. (#65092, @sttts)
• Kubernetes json deserializer is now case-sensitive to restore compatibility with pre-1.8 servers. (#65034, @caesarxuchao)
  • If your config files contains fields with wrong case, the config files will be now invalid.
• GCE: Fixes operation polling to adhere to the specified interval. Furthermore, operation errors are now returned instead of ignored. (#64630, @nicksardo)
• Updated hcsshim dependency to v0.6.11 (#64272, @jessfraz)
• Include kms-plugin-container.manifest to master manifests tarball. (#65035, @immutableT)
• kubeadm - Ensure the peer port is secured by explicitly setting the peer URLs for the default etcd instance. (#64988, @detiber)
  • kubeadm - Ensure that the etcd certificates are generated using a proper CN
  • kubeadm - Update generated etcd peer certificate to include localhost addresses for the default configuration.
  • kubeadm - Increase the manifest update timeout to make upgrades a bit more reliable.
• Kubernetes depends on v0.30.1 of cAdvisor (#64987, @dashpole)
• Update Cluster Autoscaler version to 1.3.0-beta.1. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.0-beta.1 (#64977, @aleksandra-malinowska)
• Webhooks for the mutating admission controller now support "remove" operation. (#64255, @rojkov)
• deprecated and inactive option '--enable-custom-metrics' is removed in 1.11 (#60699, @CaoShuFeng)
• kubernetes now packages cri-tools (crictl) in addition to all the other kubeadm tools in a deb and rpm. (#64836, @chuckha)
• Fix setup of configmap/secret/projected/downwardapi volumes (#64855, @gnufied)
• Setup dns servers and search domains for Windows Pods in dockershim. Docker EE version >= 17.10.0 is required for propagating DNS to containers. (#63905, @feiskyer)

v1.11.0-beta.2

Documentation

Downloads for v1.11.0-beta.2

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0-beta.1

Action Required

• [action required] kubeadm join is now blocking on the kubelet performing the TLS Bootstrap properly. (#64792, @luxas)
  • Earlier, kubeadm join only did the discovery part and exited successfully without checking that the
  • kubelet actually started properly and performed the TLS bootstrap correctly. Now, as kubeadm runs
  • some post-join steps (e.g. annotating the Node API object with the CRISocket as in this PR, as a
  • stop-gap until this is discoverable automatically), kubeadm join is now waiting for the kubelet to
  • perform the TLS Bootstrap, and then uses that credential to perform further actions. This also
  • improves the UX, as kubeadm will exit with a non-zero code if the kubelet isn't in a functional
  • state, instead of pretending like everything's fine.
• [action required] The structure of the kubelet dropin in the kubeadm deb package has changed significantly. (#64780, @luxas)
  • Instead of hard-coding the parameters for the kubelet in the dropin, a structured configuration file
  • for the kubelet is used, and is expected to be present in /var/lib/kubelet/config.yaml.
  • For runtime-detected, instance-specific configuration values, a environment file with
  • dynamically-generated flags at kubeadm init or kubeadm join run time is used.
  • Finally, if the user wants to override something specific for the kubelet that can't be done via
  • the kubeadm Configuration file (which is preferred), they might add flags to the
  • KUBELET_EXTRA_ARGS environment variable in either /etc/default/kubelet
  • or /etc/sysconfig/kubelet, depending on the system you're running on.
• [action required] The --node-name flag for kubeadm now dictates the Node API object name the (#64706, @liztio)
  • kubelet uses for registration, in all cases but where you might use an in-tree cloud provider.
  • If you're not using an in-tree cloud provider, --node-name will set the Node API object name.
  • If you're using an in-tree cloud provider, you MUST make --node-name match the name the
  • in-tree cloud provider decides to use.
• [action required] kubeadm: The Token-related fields in the MasterConfiguration object have now been refactored. Instead of the top-level .Token, .TokenTTL, .TokenUsages, .TokenGroups fields, there is now a BootstrapTokens slice of BootstrapToken objects that support the same features under the .Token, .TTL, .Usages, .Groups fields. (#64408, @luxas)

Other notable changes

• Add Vertical Pod Autoscaler to autoscaling/v2beta1 (#63797, @kgrygiel)
• kubeadm: only run kube-proxy on architecture consistent nodes (#64696, @dixudx)
• kubeadm: Add a new kubeadm upgrade node config command (#64624, @luxas)
• Orphan delete is now supported for custom resources (#63386, @roycaihw)
• Update version of Istio addon from 0.6.0 to 0.8.0. (#64537, @ostromart)
  • See https://istio.io/news/2018/announcing-0.8/ for full Isto release notes.
• Provides API support for external CSI storage drivers to support block volumes. (#64723, @vladimirvivien)
• kubectl will list all allowed print formats when an invalid format is passed. (#64371, @CaoShuFeng)
• Use IONice to reduce IO priority of du and find (#64800, @dashpole)
  • cAdvisor ContainerReference no longer contains Labels. Use ContainerSpec instead.
  • Fix a bug where cadvisor failed to discover a sub-cgroup that was created soon after the parent cgroup.
• Kubelet will set extended resource capacity to zero after it restarts. If the extended resource is exported by a device plugin, its capacity will change to a valid value after the device plugin re-connects with the Kubelet. If the extended resource is exported by an external component through direct node status capacity patching, the component should repatch the field after kubelet becomes ready again. During the time gap, pods previously assigned with such resources may fail kubelet admission but their controller should create new pods in response to such failures. (#64784, @jiayingz)
• Introduce ContainersReady condition in Pod Status (#64646, @freehan)
• The Sysctls experimental feature has been promoted to beta (enabled by default via the Sysctls feature flag). PodSecurityPolicy and Pod objects now have fields for specifying and controlling sysctls. Alpha sysctl annotations will be ignored by 1.11+ kubelets. All alpha sysctl annotations in existing deployments must be converted to API fields to be effective. (#63717, @ingvagabund)
• kubeadm now configures the etcd liveness probe correctly when etcd is listening on all interfaces (#64670, @stealthybox)
• Fix regression in v1.JobSpec.backoffLimit that caused failed Jobs to be restarted indefinitely. (#63650, @soltysh)
• GCE: Update cloud provider to use TPU v1 API (#64727, @yguo0905)
• Kubelet: Add security context for Windows containers (#64009, @feiskyer)
• Volume topology aware dynamic provisioning (#63193, @lichuqiang)
• CoreDNS deployment configuration now uses k8s.gcr.io imageRepository (#64775, @rajansandeep)
• Updated Container Storage Interface specification version to v0.3.0 (#64719, @davidz627)
• Kubeadm: Make CoreDNS run in read-only mode and drop all unneeded privileges (#64473, @nberlee)
• Add a volume projection that is able to project service account tokens. (#62005, @mikedanese)
• Fix kubectl auth can-i exit code. It will return 1 if the user is not allowed and 0 if it's allowed. (#59579, @fbac)
• apply global flag "context" for kubectl config view --minify (#64608, @dixudx)
• Fix kube-controller-manager panic while provisioning Azure security group rules (#64739, @feiskyer)
• API change for volume topology aware dynamic provisioning (#63233, @lichuqiang)
• Add azuredisk PV size grow feature (#64386, @andyzhangx)
• Modify e2e tests to use priorityClass beta version & switch priorityClass feature to beta (#63724, @ravisantoshgudimetla)
• Adding CSI driver registration code. (#64560, @sbezverk)
• fixes a potential deadlock in the garbage collection controller (#64235, @liggitt)
• Fixes issue for readOnly subpath mounts for SELinux systems and when the volume mountPath already existed in the container image. (#64351, @msau42)
• Add log and fs stats for Windows containers (#62266, @feiskyer)
• client-go: credential exec plugins have been promoted to beta (#64482, @ericchiang)
• Revert #64364 to resurrect rescheduler. For More info see[#64725] https://github.com/kubernetes/kubernetes/issues/64725 and (#64592, @ravisantoshgudimetla)
• Add RequestedToCapacityRatioPriority priority function. Function is parametrized with set of points mapping node utilization (0-100) to score (0-10). (#63929, @losipiuk)
  • Function is linear between points. Resource utilization is defined as one minus ratio of total amount of resource requested by pods on node and node's capacity (scaled to 100).
  • Final utilization used for computation is arithmetic mean of cpu utilization and memory utilization.
  • Function is disabled by default and can be enabled via scheduler policy config file.
  • If no parametrization is specified in config file it defaults to one which gives score 10 to utilization 0 and score 0 to utilization 100.
• kubeadm init detects if systemd-resolved is running and configures the kubelet to use a working resolv.conf. (#64665, @stealthybox)
• fix data loss issue if using existing azure disk with partitions in disk mount (#63270, @andyzhangx)
• Meta data of CustomResources is now pruned and schema checked during deserialization of requests and when read from etcd. In the former case, invalid meta data is rejected, in the later it is dropped from the CustomResource objects. (#64267, @sttts)
• Add Alpha support for dynamic volume limits based on node type (#64154, @gnufied)
• Fixed CSI gRPC connection leak during volume operations. (#64519, @vladimirvivien)
• in-tree support for openstack credentials is now deprecated. please use the "client-keystone-auth" from the cloud-provider-openstack repository. details on how to use this new capability is documented here - https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md (#64346, @dims)
• ScheduleDaemonSetPods is an alpha feature (since v1.11) that causes DaemonSet Pods (#63223, @k82cn)
  • to be scheduler by default scheduler, instead of Daemonset controller. When it is enabled,
  • the NodeAffinity term (instead of .spec.nodeName) is added to the DaemonSet Pods;
  • this enables the default scheduler to bind the Pod to the target host. If node affinity
  • of DaemonSet Pod already exists, it will be replaced.
  • DaemonSet controller will only perform these operations when creating DaemonSet Pods;
  • and those operations will only modify the Pods of DaemonSet, no changes are made to the
  • .spec.template of DaemonSet.
• fix formatAndMount func issue on Windows (#63248, @andyzhangx)
• AWS EBS, Azure Disk, GCE PD and Ceph RBD volume plugins support dynamic provisioning of raw block volumes. (#64447, @jsafrane)
• kubeadm upgrade apply can now ignore version errors with --force (#64570, @liztio)
• Adds feature gate for plugin watcher (#64605, @vikaschoudhary16)
• Kubelet now proxies container streaming between apiserver and container runtime. The connection between kubelet and apiserver is authenticated. Container runtime should change streaming server to serve on localhost, to make the connection between kubelet and container runtime local. (#64006, @Random-Liu)
  • In this way, the whole container streaming connection is secure. To switch back to the old behavior, set --redirect-container-streaming=true flag.
• TokenRequests now are required to have an expiration duration between 10 minutes and 2^32 seconds. (#63999, @mikedanese)
• Expose /debug/flags/v to allow dynamically set glog logging level, if want to change glog level to 3, you only have to send a PUT request with like curl -X PUT http://127.0.0.1:8080/debug/flags/v -d "3". (#63777, @hzxuzhonghu)
• New conformance test added for Watch. (#61424, @jennybuckley)
• The GitRepo volume type is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. (#63445, @ericchiang)
• kubeadm now preserves previous manifests after upgrades (#64337, @liztio)
• Implement kubelet side online file system resizing (#62460, @mlmhl)

v1.11.0-beta.1

Documentation

Downloads for v1.11.0-beta.1

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0-alpha.2

Action Required

• [action required] .NodeName and .CRISocket in the MasterConfiguration and NodeConfiguration v1alpha1 API objects are now .NodeRegistration.Name and .NodeRegistration.CRISocket respectively in the v1alpha2 API. The .NoTaintMaster field has been removed in the v1alpha2 API. (#64210, @luxas)
• (ACTION REQUIRED) PersisntVolumeLabel admission controller is now disabled by default. If you depend on this feature (AWS/GCE) then ensure it is added to the --enable-admission-plugins flag on the kube-apiserver. (#64326, @andrewsykim)
• [action required] kubeadm: The :Etcd struct has been refactored in the v1alpha2 API. All the options now reside under either .Etcd.Local or .Etcd.External. Automatic conversions from the v1alpha1 API are supported. (#64066, @luxas)
• [action required] kubeadm: kubelets in kubeadm clusters now disable the readonly port (10255). If you're relying on unauthenticated access to the readonly port, please switch to using the secure port (10250). Instead, you can now use ServiceAccount tokens when talking to the secure port, which will make it easier to get access to e.g. the /metrics endpoint of the kubelet securely. (#64187, @luxas)
• [action required] kubeadm: Support for .AuthorizationModes in the kubeadm v1alpha2 API has been removed. Instead, you can use the .APIServerExtraArgs and .APIServerExtraVolumes fields to achieve the same effect. Files using the v1alpha1 API and setting this field will be automatically upgraded to this v1alpha2 API and the information will be preserved. (#64068, @luxas)
• [action required] The formerly publicly-available cAdvisor web UI that the kubelet ran on port 4194 by default is now turned off by default. The flag configuring what port to run this UI on --cadvisor-port was deprecated in v1.10. Now the default is --cadvisor-port=0, in other words, to not run the web server. The recommended way to run cAdvisor if you still need it, is via a DaemonSet. The --cadvisor-port will be removed in v1.12 (#63881, @luxas)
• [action required] kubeadm: The .ImagePullPolicy field has been removed in the v1alpha2 API version. Instead it's set statically to IfNotPresent for all required images. If you want to always pull the latest images before cluster init (like what Always would do), run kubeadm config images pull before each kubeadm init. If you don't want the kubelet to pull any images at kubeadm init time, as you for instance don't have an internet connection, you can also run kubeadm config images pull before kubeadm init or side-load the images some other way (e.g. docker load -i image.tar). Having the images locally cached will result in no pull at runtime, which makes it possible to run without any internet connection. (#64096, @luxas)
• [action required] In the new v1alpha2 kubeadm Configuration API, the .CloudProvider and .PrivilegedPods fields don't exist anymore. (#63866, @luxas)
  • Instead, you should use the out-of-tree cloud provider implementations which are beta in v1.11.
  • If you have to use the legacy in-tree cloud providers, you can rearrange your config like the example below. In case you need the cloud-config file (located in {cloud-config-path}), you can mount it into the API Server and controller-manager containers using ExtraVolumes like the example below.
  • If you need to use the .PrivilegedPods functionality, you can still edit the manifests in /etc/kubernetes/manifests/, and set .SecurityContext.Privileged=true for the apiserver and controller manager.

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha2
apiServerExtraArgs:
  cloud-provider: "{cloud}"
  cloud-config: "{cloud-config-path}"
apiServerExtraVolumes:
- name: cloud
  hostPath: "{cloud-config-path}"
  mountPath: "{cloud-config-path}"
controllerManagerExtraArgs:
  cloud-provider: "{cloud}"
  cloud-config: "{cloud-config-path}"
controllerManagerExtraVolumes:
- name: cloud
  hostPath: "{cloud-config-path}"
  mountPath: "{cloud-config-path}"

• [action required] kubeadm now uses an upgraded API version for the configuration file, kubeadm.k8s.io/v1alpha2. kubeadm in v1.11 will still be able to read v1alpha1 configuration, and will automatically convert the configuration to v1alpha2 internally and when storing the configuration in the ConfigMap in the cluster. (#63788, @luxas)
• The annotation service.alpha.kubernetes.io/tolerate-unready-endpoints is deprecated. Users should use Service.spec.publishNotReadyAddresses instead. (#63742, @thockin)
• avoid duplicate status in audit events (#62695, @CaoShuFeng)

Other notable changes

• Remove rescheduler from master. (#64364, @ravisantoshgudimetla)
• Declare IPVS-based kube-proxy GA (#58442, @m1093782566)
• kubeadm: conditionally set the kubelet cgroup driver for Docker (#64347, @neolit123)
• kubectl built for darwin from darwin now enables cgo to use the system-native C libraries for DNS resolution. Cross-compiled kubectl (e.g. from an official kubernetes release) still uses the go-native netgo DNS implementation. (#64219, @ixdy)
• AWS EBS volumes can be now used as ReadOnly in pods. (#64403, @jsafrane)
• Exec authenticator plugin supports TLS client certificates. (#61803, @awly)
• Use Patch instead of Put to sync pod status (#62306, @freehan)
• kubectl apply --prune supports CronJob resource. (#62991, @tomoe)
• Label ExternalEtcdClientCertificates can be used for ignoring all preflight check issues related to client certificate files for external etcd. (#64269, @kad)
• Provide a meaningful error message in openstack cloud provider when no valid IP address can be found for a node (#64318, @gonzolino)
• kubeadm: Add a 'kubeadm config migrate' command to convert old API types to their newer counterparts in the new, supported API types. This is just a client-side tool, it just executes locally without requiring a cluster to be running. You can think about this as an Unix pipe that upgrades config files. (#64232, @luxas)
• The --dry-run flag has been enabled for kubectl auth reconcile (#64458, @mrogers950)
• Add probe based mechanism for kubelet plugin discovery (#63328, @vikaschoudhary16)
• Add Establishing Controller on CRDs to avoid race between Established condition and CRs actually served. In HA setups, the Established condition is delayed by 5 seconds. (#63068, @xmudrii)
• CoreDNS is now v1.1.3 (#64258, @rajansandeep)
• kubeadm will pull required images during preflight checks if it cannot find them on the system (#64105, @chuckha)
• kubeadm: rename the addon parameter kube-dns to coredns for kubeadm alpha phases addons as CoreDNS is now the default DNS server in 1.11. (#64274, @neolit123)
• kubeadm: when starting the API server use the arguments --enable-admission-plugins and --disable-admission-plugins instead of the deprecated --admission-control. (#64165, @neolit123)
• Add spec.additionalPrinterColumns to CRDs to define server side printing columns. (#60991, @sttts)
• fix azure file size grow issue (#64383, @andyzhangx)
• Fix issue of colliding nodePorts when the cluster has services with externalTrafficPolicy=Local (#64349, @nicksardo)
• fixes a panic applying json patches containing out of bounds operations (#64355, @liggitt)
• Fail fast if cgroups-per-qos is set on Windows (#62984, @feiskyer)
• Move Volume expansion to Beta (#64288, @gnufied)
• kubectl delete does not use reapers for removing objects anymore, but relies on server-side GC entirely (#63979, @soltysh)
• Basic plumbing for volume topology aware dynamic provisioning (#63232, @lichuqiang)
• API server properly parses propagationPolicy as a query parameter sent with a delete request (#63414, @roycaihw)
• Property serverAddressByClientCIDRs in metav1.APIGroup (discovery API) now become optional instead of required (#61963, @roycaihw)
• The dynamic Kubelet config feature is now beta, and the DynamicKubeletConfig feature gate is on by default. In order to use dynamic Kubelet config, ensure that the Kubelet's --dynamic-config-dir option is set. (#64275, @mtaufen)
• Add reason message logs for non-exist Azure resources (#64248, @feiskyer)
• Fix SessionAffinity not updated issue for Azure load balancer (#64180, @feiskyer)
• The kube-apiserver openapi doc now includes extensions identifying APIService and CustomResourceDefinition kinds (#64174, @liggitt)
• apiservices/status and certificatesigningrequests/status now support GET and PATCH (#64063, @roycaihw)
• kubectl: This client version requires the apps/v1 APIs, so it will not work against a cluster version older than v1.9.0. Note that kubectl only guarantees compatibility with clusters that are +/-1 minor version away. (#61419, @enisoc)
• Correct the way we reset containers and pods in kubeadm via crictl (#63862, @runcom)
• Allow env from resource with keys & updated tests (#60636, @PhilipGough)
• The kubelet certificate rotation feature can now be enabled via the .RotateCertificates field in the kubelet's config file. The --rotate-certificates flag is now deprecated, and will be removed in a future release. (#63912, @luxas)
• Use DeleteOptions.PropagationPolicy instead of OrphanDependents in kubectl (#59851, @nilebox)
• add block device support for azure disk (#63841, @andyzhangx)
• Fix incorrectly propagated ResourceVersion in ListRequests returning 0 items. (#64150, @wojtek-t)
• Changes ext3/ext4 volume creation to not reserve any portion of the volume for the root user. (#64102, @atombender)
• Add CRD Versioning with NOP converter (#63830, @mbohlool)
• adds a kubectl wait command (#64034, @deads2k)
• "kubeadm init" now writes a structured and versioned kubelet ComponentConfiguration file to /var/lib/kubelet/config.yaml and an environment file with runtime flags (you can source this file in the systemd kubelet dropin) to /var/lib/kubelet/kubeadm-flags.env. (#63887, @luxas)
• kubectl auth reconcile only works with rbac.v1 (#63967, @deads2k)
• The dynamic Kubelet config feature will now update config in the event of a ConfigMap mutation, which reduces the chance for silent config skew. Only name, namespace, and kubeletConfigKey may now be set in Node.Spec.ConfigSource.ConfigMap. The least disruptive pattern for config management is still to create a new ConfigMap and incrementally roll out a new Node.Spec.ConfigSource. (#63221, @mtaufen)
• Graduate CRI container log rotation to beta, and enable it by default. (#64046, @yujuhong)
• APIServices with kube-like versions (e.g. v1, v2beta1, etc.) will be sorted appropriately within each group. (#64004, @mbohlool)
• kubectl and client-go now detects duplicated name for user, cluster and context when loading kubeconfig and reports error (#60464, @roycaihw)
• event object references with apiversion will now report an apiversion. (#63913, @deads2k)
• Subresources for custom resources is now beta and enabled by default. With this, updates to the /status subresource will disallow updates to all fields other than .status (not just .spec and .metadata as before). Also, required can be used at the root of the CRD OpenAPI validation schema when the /status subresource is enabled. (#63598, @nikhita)
• increase grpc client default response size (#63977, @runcom)
• HTTP transport now uses context.Context to cancel dial operations. k8s.io/client-go/transport/Config struct has been updated to accept a function with a context.Context parameter. This is a breaking change if you use this field in your code. (#60012, @ash2k)
• Adds a mechanism in vSphere Cloud Provider to get credentials from Kubernetes secrets (#63902, @abrarshivani)
• kubeadm: A kubeadm config print-default command has now been added that you can use as a starting point when writing your own kubeadm configuration files (#63969, @luxas)
• Update event-exporter to version v0.2.0 that supports old (gke_container/gce_instance) and new (k8s_container/k8s_node/k8s_pod) stackdriver resources. (#63918, @cezarygerard)
• Cluster Autoscaler 1.2.2 (release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.2.2) (#63974, @aleksandra-malinowska)
• Update kubeadm's minimum supported kubernetes in v1.11.x to 1.10 (#63920, @dixudx)
• Add 'UpdateStrategyType' and 'RollingUpdateStrategy' to 'kubectl describe sts' command output. (#63844, @tossmilestone)
• Remove UID mutation from request.context. (#63957, @hzxuzhonghu)
• kubeadm has removed .Etcd.SelfHosting from its configuration API. It was never used in practice. (#63871, @luxas)
• list/watch API requests with a fieldSelector that specifies metadata.name can now be authorized as requests for an individual named resource (#63469, @wojtek-t)
• Add a way to pass extra arguments to etcd. (#63961, @mborsz)
• minor fix for VolumeZoneChecker predicate, storageclass can be in annotation and spec. (#63749, @wenlxie)
• vSphere Cloud Provider: add SAML token authentication support (#63824, @dougm)
• adds the kubeadm upgrade diff command to show how static pod manifests will be changed by an upgrade. (#63930, @liztio)
• Fix memory cgroup notifications, and reduce associated log spam. (#63220, @dashpole)
• Adds a kubeadm config images pull command to pull container images used by kubeadm. (#63833, @chuckha)
• Restores the pre-1.10 behavior of the openstack cloud provider which uses the instance name as the Kubernetes Node name. This requires instances be named with RFC-1123 compatible names. (#63903, @liggitt)
• Added support for NFS relations on kubernetes-worker charm. (#63817, @hyperbolic2346)
• Stop using InfluxDB as default cluster monitoring (#62328, @serathius)
  • InfluxDB cluster monitoring is deprecated and will be removed in v1.12
• GCE: Fix to make the built-in kubernetes service properly point to the master's load balancer address in clusters that use multiple master VMs. (#63696, @grosskur)
• Kubernetes cluster on GCE have crictl installed now. Users can use it to help debug their node. The documentation of crictl can be found https://github.com/kubernetes-incubator/cri-tools/blob/master/docs/crictl.md. (#63357, @Random-Liu)
• The NodeRestriction admission plugin now prevents kubelets from modifying/removing taints applied to their Node API object. (#63167, @liggitt)
• The status of dynamic Kubelet config is now reported via Node.Status.Config, rather than the KubeletConfigOk node condition. (#63314, @mtaufen)
• kubeadm now checks that IPv4/IPv6 forwarding is enabled (#63872, @kad)
• kubeadm will now deploy CoreDNS by default instead of KubeDNS (#63509, @detiber)
• This PR will leverage subtests on the existing table tests for the scheduler units. (#63658, @xchapter7x)
  • Some refactoring of error/status messages and functions to align with new approach.
• kubeadm upgrade now supports external etcd setups again (#63495, @detiber)
• fix mount unmount failure for a Windows pod (#63272, @andyzhangx)
• CRI: update documents for container logpath. The container log path has been changed from containername_attempt#.log to containername/attempt#.log (#62015, @feiskyer)
• Create a new dryRun query parameter for mutating endpoints. If the parameter is set, then the query will be rejected, as the feature is not implemented yet. This will allow forward compatibility with future clients; otherwise, future clients talking with older apiservers might end up modifying a resource even if they include the dryRun query parameter. (#63557, @apelisse)
• kubelet: fix hangs in updating Node status after network interruptions/changes between the kubelet and API server (#63492, @liggitt)
• The PriorityClass API is promoted to scheduling.k8s.io/v1beta1 (#63100, @ravisantoshgudimetla)
• Services can listen on same host ports on different interfaces with --nodeport-addresses specified (#62003, @m1093782566)
• kubeadm will no longer generate an unused etcd CA and certificates when configured to use an external etcd cluster. (#63806, @detiber)
• corrects a race condition in bootstrapping aggregated cluster roles in new HA clusters (#63761, @liggitt)
• Adding initial Korean translation for kubectl (#62040, @ianychoi)
• Report node DNS info with --node-ip flag (#63170, @micahhausler)
• The old dynamic client has been replaced by a new one. The previous dynamic client will exist for one release in client-go/deprecated-dynamic. Switch as soon as possible. (#63446, @deads2k)
• CustomResourceDefinitions Status subresource now supports GET and PATCH (#63619, @roycaihw)
• Re-enable nodeipam controller for external clouds. (#63049, @andrewsykim)
• Removes a preflight check for kubeadm that validated custom kube-apiserver, kube-controller-manager and kube-scheduler arguments. (#63673, @chuckha)
• Adds a list-images subcommand to kubeadm that lists required images for a kubeadm install. (#63450, @chuckha)
• Apply pod name and namespace labels to pod cgroup in cAdvisor metrics (#63406, @derekwaynecarr)
• try to read openstack auth config from client config and fall back to read from the environment variables if not available (#60200, @dixudx)
• GC is now bound by QPS (it wasn't before) and so if you need more QPS to avoid ratelimiting GC, you'll have to set it. (#63657, @shyamjvs)
• The Kubelet's deprecated --allow-privileged flag now defaults to true. This enables users to stop setting --allow-privileged in order to transition to PodSecurityPolicy. Previously, users had to continue setting --allow-privileged, because the default was false. (#63442, @mtaufen)
• You must now specify Node.Spec.ConfigSource.ConfigMap.KubeletConfigKey when using dynamic Kubelet config to tell the Kubelet which key of the ConfigMap identifies its config file. (#59847, @mtaufen)
• Kubernetes version command line parameter in kubeadm has been updated to drop an unnecessary redirection from ci/latest.txt to ci-cross/latest.txt. Users should know exactly where the builds are stored on Google Cloud storage buckets from now on. For example for 1.9 and 1.10, users can specify ci/latest-1.9 and ci/latest-1.10 as the CI build jobs what build images correctly updates those. The CI jobs for master update the ci-cross/latest location, so if you are looking for latest master builds, then the correct parameter to use would be ci-cross/latest. (#63504, @dims)
• Search standard KubeConfig file locations when using kubeadm token without --kubeconfig. (#62850, @neolit123)
• Include the list of security groups when failing with the errors that more then one is tagged (#58874, @sorenmat)
• Allow "required" to be used at the CRD OpenAPI validation schema when the /status subresource is enabled. (#63533, @sttts)
• When updating /status subresource of a custom resource, only the value at the .status subpath for the update is considered. (#63385, @CaoShuFeng)
• Supported nodeSelector.matchFields (node's metadata.node) in scheduler. (#62453, @k82cn)
• Do not check vmSetName when getting Azure node's IP (#63541, @feiskyer)
• Fix stackdriver metrics for node memory using wrong metric type (#63535, @serathius)
• [fluentd-gcp addon] Use the logging agent's node name as the metadata agent URL. (#63353, @bmoyles0117)
• kubectl cp supports completion. (#60371, @superbrothers)
• Azure VMSS: support VM names to contain the _ character (#63526, @djsly)
• OpenStack built-in cloud provider is now deprecated. Please use the external cloud provider for OpenStack. (#63524, @dims)
• the shortcuts which were moved server-side in at least 1.9 have been removed from being hardcoded in kubectl (#63507, @deads2k)
• Fixes fake client generation for non-namespaced subresources (#60445, @jhorwit2)
• kubectl delete with selection criteria defaults to ignoring not found errors (#63490, @deads2k)
• Increase scheduler cache generation number monotonically in order to avoid collision and use of stale information in scheduler. (#63264, @bsalamat)
• Fixes issue where subpath readOnly mounts failed (#63045, @msau42)
• Update to use go1.10.2 (#63412, @praseodym)
• kubectl create [secret | configmap] --from-file now works on Windows with fully-qualified paths (#63439, @liggitt)
• kube-apiserver: the default --endpoint-reconciler-type is now lease. The master-count endpoint reconciler type is deprecated and will be removed in 1.13. (#63383, @liggitt)
• owner references can be set during creation without deletion power (#63403, @deads2k)
• Lays groundwork for OIDC distributed claims handling in the apiserver authentication token checker. (#63213, @filmil)
  • A distributed claim allows the OIDC provider to delegate a claim to a
  • separate URL. Distributed claims are of the form as seen below, and are
  • defined in the OIDC Connect Core 1.0, section 5.6.2.
  • For details, see:
  • http://openid.net/specs/openid-connect-core-1_0.html#AggregatedDistributedClaims
• Use /usr/bin/env in all script shebangs to increase portability. (#62657, @matthyx)

v1.11.0-alpha.2

Documentation

Downloads for v1.11.0-alpha.2

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.11.0-alpha.1

Other notable changes

• kubeadm upgrade plan now accepts a version which improves the UX nicer in air-gapped environments. (#63201, @chuckha)
• kubectl now supports --field-selector for delete, label, and annotate (#60717, @liggitt)
• kube-apiserver: --endpoint-reconciler-type now defaults to lease. The master-count reconciler is deprecated and will be removed in 1.13. (#58474, @rphillips)
• OpenStack cloudprovider: Fix deletion of orphaned routes (#62729, @databus23)
• Fix a bug that headless service without ports fails to have endpoint created. (#62497, @MrHohn)
• Fix panic for attaching AzureDisk to vmss nodes (#63275, @feiskyer)
• kubectl api-resources now supports filtering to resources supporting specific verbs, and can output fully qualified resource names suitable for combining with commands like kubectl get (#63254, @liggitt)
• fix cephfs fuse mount bug when user is not admin (#61804, @zhangxiaoyu-zidif)
• StorageObjectInUseProtection feature is GA. (#62870, @pospispa)
• fixed spurious "unable to find api field" errors patching custom resources (#63146, @liggitt)
• KUBE_API_VERSIONS is no longer respected. It was used for testing, but runtime-config is the proper flag to set. (#63165, @deads2k)
• Added CheckNodePIDPressurePredicate to checks if a pod can be scheduled on (#60007, @k82cn)
  • a node reporting pid pressure condition.
• Upgrade Azure Go SDK to stable version (v14.6.0) (#63063, @feiskyer)
• kubeadm: prompt the user for confirmation when resetting a master node (#59115, @alexbrand)
• add warnings on using pod-infra-container-image for remote container runtime (#62982, @dixudx)
• Deprecate kubectl rolling-update (#61285, @soltysh)
• client-go developers: the new dynamic client is easier to use and the old is deprecated, you must switch. (#62913, @deads2k)
• Fix issue where on re-registration of device plugin, allocatable was not getting updated. This issue makes devices invisible to the Kubelet if device plugin restarts. Only work-around, if this fix is not there, is to restart the kubelet and then start device plugin. (#63118, @vikaschoudhary16)
• Remove METADATA_AGENT_VERSION configuration option. (#63000, @kawych)
• kubelets are no longer allowed to delete their own Node API object. Prior to 1.11, in rare circumstances related to cloudprovider node ID changes, kubelets would attempt to delete/recreate their Node object at startup. If a legacy kubelet encounters this situation, a cluster admin can remove the Node object: kubectl delete node/<nodeName> (#62818, @mikedanese)
  • or grant self-deletion permission explicitly:
    • kubectl create clusterrole self-deleting-nodes --verb=delete --resource=nodes
    • kubectl create clusterrolebinding self-deleting-nodes --clusterrole=self-deleting-nodes --group=system:nodes
• kubeadm creates kube-proxy with a toleration to run on all nodes, no matter the taint. (#62390, @discordianfish)
• fix resultRun by resetting it to 0 on pod restart (#62853, @tony612)
• Mount additional paths required for a working CA root, for setups where /etc/ssl/certs doesn't contains certificates but just symlink. (#59122, @klausenbusk)
• Introduce truncating audit backend that can be enabled for existing backend to limit the size of individual audit events and batches of events. (#61711, @crassirostris)
• kubeadm upgrade no longer races leading to unexpected upgrade behavior on pod restarts (#62655, @stealthybox)
  • kubeadm upgrade now successfully upgrades etcd and the controlplane to use TLS
  • kubeadm upgrade now supports external etcd setups
  • kubeadm upgrade can now rollback and restore etcd after an upgrade failure
• Add --ipvs-exclude-cidrs flag to kube-proxy. (#62083, @rramkumar1)
• Fix the liveness probe to use /bin/bash -c instead of /bin/bash c. (#63033, @bmoyles0117)
• Added MatchFields to NodeSelectorTerm; in 1.11, it only support metadata.name. (#62002, @k82cn)
• Fix scheduler informers to receive events for all the pods in the cluster. (#63003, @bsalamat)
• removed unsafe double RLock in cpumanager (#62464, @choury)
• Fix in vSphere Cloud Provider to handle upgrades from kubernetes version less than v1.9.4 to v1.9.4 and above. (#62919, @abrarshivani)
• The --bootstrap-kubeconfig argument to Kubelet previously created the first bootstrap client credentials in the certificates directory as kubelet-client.key and kubelet-client.crt. Subsequent certificates created by cert rotation were created in a combined PEM file that was atomically rotated as kubelet-client-DATE.pem in that directory, which meant clients relying on the node.kubeconfig generated by bootstrapping would never use a rotated cert. The initial bootstrap certificate is now generated into the cert directory as a PEM file and symlinked to kubelet-client-current.pem so that the generated kubeconfig remains valid after rotation. (#62152, @smarterclayton)
• stop kubelet to cloud provider integration potentially wedging kubelet sync loop (#62543, @ingvagabund)
• Fix error where config map for Metadata Agent was not created by addon manager. (#62909, @kawych)
• Fixes the kubernetes.default.svc loopback service resolution to use a loopback configuration. (#62649, @liggitt)
• Code generated for CRDs now passes go vet. (#62412, @bhcleek)
• fix permissions to allow statefulset scaling for admins, editors, and viewers (#62336, @deads2k)
• Add support of standard LB to Azure vmss (#62707, @feiskyer)
• GCE: Fix for internal load balancer management resulting in backend services with outdated instance group links. (#62885, @nicksardo)
• The --experimental-qos-reserve kubelet flags is replaced by the alpha level --qos-reserved flag or QOSReserved field in the kubeletconfig and requires the QOSReserved feature gate to be enabled. (#62509, @sjenning)
• Set pod status to "Running" if there is at least one container still reporting as "Running" status and others are "Completed". (#62642, @ceshihao)
• Split PodPriority and PodPreemption feature gate (#62243, @resouer)
• Add support to resize Portworx volumes. (#62308, @harsh-px)

v1.11.0-alpha.1

Documentation

Downloads for v1.11.0-alpha.1

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.10.0

Action Required

• ACTION REQUIRED: Alpha annotation for PersistentVolume node affinity has been removed. Update your PersistentVolumes to use the beta PersistentVolume.nodeAffinity field before upgrading to this release (#61816, @wackxu)
• ACTION REQUIRED: In-place node upgrades to this release from versions 1.7.14+, 1.8.9+, and 1.9.4+ are not supported if using subpath volumes with PVCs. Such pods should be drained from the node first. (#61373, @msau42)

Other notable changes

• Make volume usage metrics available for Cinder (#62668, @zetaab)
• kubectl stops rendering List as suffix kind name for CRD resources (#62512, @dixudx)
• Removes --include-extended-apis which was deprecated back in https://github.com/kubernetes/kubernetes/pull/32894 (#62803, @deads2k)
• Add write-config-to to scheduler (#62515, @resouer)
• Kubelets will no longer set externalID in their node spec. (#61877, @mikedanese)
• kubeadm preflight: check CRI socket path if defined, otherwise check for Docker (#62481, @taharah)
• fix network setup in hack/local-up-cluster.sh (https://github.com/kubernetes/kubernetes/pull/60431) (#60633, @pohly)
  • better error diagnostics in hack/local-up-cluster.sh output
• Add prometheus cluster monitoring addon to kube-up (#62195, @serathius)
• Fix inter-pod anti-affinity check to consider a pod a match when all the anti-affinity terms match. (#62715, @bsalamat)
• GCE: Bump GLBC version to 1.1.1 - fixing an issue of handling multiple certs with identical certificates (#62751, @nicksardo)
• fixes configuration error when upgrading kubeadm from 1.9 to 1.10+ (#62568, @liztio)
  • enforces kubeadm upgrading kubernetes from the same major and minor versions as the kubeadm binary.
• Allow user to scale l7 default backend deployment (#62685, @freehan)
• Pod affinity nodeSelectorTerm.matchExpressions may now be empty, and works as previously documented: nil or empty matchExpressions matches no objects in scheduler. (#62448, @k82cn)
• Add @andrewsykim as an approver for CCM related code. (#62749, @andrewsykim)
• Fix an issue in inter-pod affinity predicate that cause affinity to self being processed incorrectly (#62591, @bsalamat)
• fix WaitForAttach failure issue for azure disk (#62612, @andyzhangx)
• Update kube-dns to Version 1.14.10. Major changes: (#62676, @MrHohn)
  • Fix a bug in DNS resolution for externalName services
  • and PTR records that need to query from upstream nameserver.
• Update version of Istio addon from 0.5.1 to 0.6.0. (#61911, @ostromart)
  • See https://istio.io/news/releases/0.x/announcing-0.6/ for full Isto release notes.
• Phase kubeadm alpha phase kubelet is added to support dynamic kubelet configuration in kubeadm. (#57224, @xiangpengzhao)
• kubeadm alpha phase kubeconfig user supports groups (organizations) to be specified in client cert. (#62627, @xiangpengzhao)
• Fix user visible files creation for windows (#62375, @feiskyer)
• remove deprecated initresource admission plugin (#58784, @wackxu)
• Fix machineID getting for vmss nodes when using instance metadata (#62611, @feiskyer)
• Fixes issue where PersistentVolume.NodeAffinity.NodeSelectorTerms were ANDed instead of ORed. (#62556, @msau42)
• Fix potential infinite loop that can occur when NFS PVs are recycled. (#62572, @joelsmith)
• Fix Forward chain default reject policy for IPVS proxier (#62007, @m1093782566)
• The kubeadm config option API.ControlPlaneEndpoint has been extended to take an optional port which may differ from the apiserver's bind port. (#62314, @rjosephwright)
• cluster/kube-up.sh now provisions a Kubelet config file for GCE via the metadata server. This file is installed by the corresponding GCE init scripts. (#62183, @mtaufen)
• Remove alpha functionality that allowed the controller manager to approve kubelet server certificates. (#62471, @mikedanese)
• gitRepo volumes in pods no longer require git 1.8.5 or newer, older git versions are supported too now. (#62394, @jsafrane)
• Default mount propagation has changed from "HostToContainer" ("rslave" in Linux terminology) to "None" ("private") to match the behavior in 1.9 and earlier releases. "HostToContainer" as a default caused regressions in some pods. (#62462, @jsafrane)
• improve performance of affinity/anti-affinity predicate of default scheduler significantly. (#62211, @bsalamat)
• fix nsenter GetFileType issue in containerized kubelet (#62467, @andyzhangx)
• Ensure expected load balancer is selected for Azure (#62450, @feiskyer)
• Resolves forbidden error when the daemon-set-controller cluster role access controllerrevisions resources. (#62146, @frodenas)
• Adds --cluster-name to kubeadm init for specifying the cluster name in kubeconfig. (#60852, @karan)
• Upgrade the default etcd server version to 3.2.18 (#61198, @jpbetz)
• [fluentd-gcp addon] Increase CPU limit for fluentd to 1 core to achieve 100kb/s throughput. (#62430, @bmoyles0117)
• GCE: Bump GLBC version to 1.1.0 - supporting multiple certificates and HTTP2 (#62427, @nicksardo)
• Fixed #731 kubeadm upgrade ignores HighAvailability feature gate (#62455, @fabriziopandini)
• Cluster Autoscaler 1.2.1 (release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.2.1) (#62457, @mwielgus)
• Add generators for apps/v1 deployments. (#61288, @ayushpateria)
• kubeadm: surface external etcd preflight validation errors (#60585, @alexbrand)
• kube-apiserver: oidc authentication now supports requiring specific claims with --oidc-required-claim=<claim>=<value> (#62136, @rithujohn191)
• Implements verbosity logging feature for kubeadm commands (#57661, @vbmade2000)
• Allow additionalProperties in CRD OpenAPI v3 specification for validation, mutually exclusive to properties. (#62333, @sttts)
• cinder volume plugin : (#61082, @wenlxie)
  • When the cinder volume status is error, controller will not do attach and detach operation
• fix incompatible file type checking on Windows (#62154, @dixudx)
• fix local volume absolute path issue on Windows (#62018, @andyzhangx)
• Remove ObjectMeta ListOptions DeleteOptions from core api group. Please use that in meta/v1 (#61809, @hzxuzhonghu)
• fix the issue that default azure disk fsypte(ext4) does not work on Windows (#62250, @andyzhangx)
• RBAC information is included in audit logs via audit.Event annotations: (#58807, @CaoShuFeng)
  • authorization.k8s.io/decision = {allow, forbid}
  • authorization.k8s.io/reason = human-readable reason for the decision
• Update kube-dns to Version 1.14.9 in kubeadm. (#61918, @MrHohn)
• Add support to ingest log entries to Stackdriver against new "k8s_container" and "k8s_node" resources. (#62076, @qingling128)
• remove deprecated --mode flag in check-network-mode (#60102, @satyasm)
• Schedule even if extender is not available when using extender (#61445, @resouer)
• Fixed column alignment when kubectl get is used with custom columns from OpenAPI schema (#56629, @luksa)
• Fixed bug in rbd-nbd utility when nbd is used. (#62168, @piontec)
• Extend the Stackdriver Metadata Agent by adding a new Deployment for ingesting unscheduled pods, and services. (#62043, @supriyagarg)
• Disabled CheckNodeMemoryPressure and CheckNodeDiskPressure predicates if TaintNodesByCondition enabled (#60398, @k82cn)
• kubeadm config can now override the Node CIDR Mask Size passed to kube-controller-manager. (#61705, @jstangroome)
• Add warnings that authors of aggregated API servers must not rely on authorization being done by the kube-apiserver. (#61349, @sttts)
• Support custom test configuration for IPAM performance integration tests (#61959, @satyasm)
• GCE: Updates GLBC version to 1.0.1 which includes a fix which prevents multi-cluster ingress objects from creating full load balancers. (#62075, @nicksardo)
• OIDC authentication now allows tokens without an "email_verified" claim when using the "email" claim. If an "email_verified" claim is present when using the "email" claim, it must be true. (#61508, @rithujohn191)
• fix local volume issue on Windows (#62012, @andyzhangx)
• kubeadm: Introduce join timeout that can be controlled via the discoveryTimeout config option (set to 5 minutes by default). (#60983, @rosti)
• Add e2e test for CRD Watch (#61025, @ayushpateria)
• Fix panic create/update CRD when mutating/validating webhook configured. (#61404, @hzxuzhonghu)
• Fix a bug that fluentd doesn't inject container logs for CRI container runtimes (containerd, cri-o etc.) into elasticsearch on GCE. (#61818, @Random-Liu)
• Support for "alpha.kubernetes.io/nvidia-gpu" resource which was deprecated in 1.10 is removed. Please use the resource exposed by DevicePlugins instead ("nvidia.com/gpu"). (#61498, @mindprince)
• Pods requesting resources prefixed with *kubernetes.io will remain unscheduled if there are no nodes exposing that resource. (#61860, @mindprince)
• flexvolume: trigger plugin init only for the relevant plugin while probe (#58519, @linyouchong)
• Update to use go1.10.1 (#60597, @cblecker)
• Rev the Azure SDK for networking to 2017-06-01 (#61955, @brendandburns)
• Return error if get NodeStageSecret and NodePublishSecret failed in CSI volume plugin (#61096, @mlmhl)
• kubectl: improves compatibility with older servers when creating/updating API objects (#61949, @liggitt)
• kubernetes-master charm now supports metrics server for horizontal pod autoscaler. (#60174, @hyperbolic2346)
• fix scheduling policy on ConfigMap breaks without the --policy-configmap-namespace flag set (#61388, @zjj2wry)
• kubectl: restore the ability to show resource kinds when displaying multiple objects (#61985, @liggitt)
• kubectl certificate approve|deny will not modify an already approved or denied CSR unless the --force flag is provided. (#61971, @smarterclayton)
• Kubelet now exposes a new endpoint /metrics/probes which exposes a Prometheus metric containing the liveness and/or readiness probe results for a container. (#61369, @rramkumar1)
• Balanced resource allocation priority in scheduler to include volume count on node (#60525, @ravisantoshgudimetla)
• new dhcp-domain parameter to be used for figuring out the hostname of a node (#61890, @dims)
• Fixed a panic in kubectl run --attach ... when the api server failed to create the runtime object (due to name conflict, PSP restriction, etc.) (#61713, @mountkin)
• Ensure reasons end up as comments in kubectl edit. (#60990, @bmcstdio)
• kube-scheduler has been fixed to use --leader-elect option back to true (as it was in previous versions) (#59732, @dims)
• Azure cloud provider now supports standard SKU load balancer and public IP. To use it, set cloud provider config with (#61884, @feiskyer)
  • {
  • "loadBalancerSku": "standard",
  • "excludeMasterFromStandardLB": true,
  • }
  • If excludeMasterFromStandardLB is not set, it will be default to true, which means master nodes are excluded to the backend of standard LB.
  • Also note standard load balancer doesn't work with annotation service.beta.kubernetes.io/azure-load-balancer-mode. This is because all nodes (except master) are added as the LB backends.
• The node authorizer now automatically sets up rules for Node.Spec.ConfigSource when the DynamicKubeletConfig feature gate is enabled. (#60100, @mtaufen)
• Update kube-dns to Version 1.14.9. Major changes: (#61908, @MrHohn)
  • Fix for kube-dns returns NXDOMAIN when not yet synced with apiserver.
  • Don't generate empty record for externalName service.
  • Add validation for upstreamNameserver port.
  • Update go version to 1.9.3.
• CRI: define the mount behavior when host path does not exist: runtime should report error if the host path doesn't exist (#61460, @feiskyer)
• Fixed ingress issue with CDK and pre-1.9 versions of kubernetes. (#61859, @hyperbolic2346)
• Removed rknetes code, which was deprecated in 1.10. (#61432, @filbranden)
• Disable ipamperf integration tests as part of every PR verification. (#61863, @satyasm)
• Enable server-side print in kubectl by default, with the ability to turn it off with --server-print=false (#61477, @soltysh)
• Add ipset and udevadm to the hyperkube base image. (#61357, @rphillips)
• In a GCE cluster, the default HAIRPIN_MODE is now "hairpin-veth". (#60166, @rramkumar1)
• Deployment will stop adding pod-template-hash labels/selector to ReplicaSets and Pods it adopts. Resources created by Deployments are not affected (will still have pod-template-hash labels/selector). (#61615, @janetkuo)
• kubectl: fixes issue with -o yaml and -o json omitting kind and apiVersion when used with --dry-run (#61808, @liggitt)
• Updated admission controller settings for Juju deployed Kubernetes clusters (#61427, @hyperbolic2346)
• Performance test framework and basic tests for the IPAM controller, to simulate behavior of the four supported modes under lightly loaded and loaded conditions, where load is defined as the number of operations to perform as against the configured kubernetes API server QPS. (#61143, @satyasm)
• kubernetes-master charm now properly clears the client-ca-file setting on the apiserver snap (#61479, @hyperbolic2346)
• Fix racy panics when using fake watches with ObjectTracker (#61195, @grantr)
• [fluentd-gcp addon] Update event-exporter image to have the latest base image. (#61727, @crassirostris)
• Use inline func to ensure unlock is executed (#61644, @resouer)
• kubectl apply view/edit-last-applied support completion. (#60499, @superbrothers)
• Automatically add system critical priority classes at cluster boostrapping. (#60519, @bsalamat)
• Ensure cloudprovider.InstanceNotFound is reported when the VM is not found on Azure (#61531, @feiskyer)
• Azure cloud provider now supports specifying allowed service tags by annotation service.beta.kubernetes.io/azure-allowed-service-tags (#61467, @feiskyer)
• Add all kinds of resource objects' statuses in HPA description. (#59609, @zhangxiaoyu-zidif)
• Bound cloud allocator to 10 retries with 100 ms delay between retries. (#61375, @satyasm)
• Removed always pull policy from the template for ingress on CDK. (#61598, @hyperbolic2346)
• escape literal percent sign when formatting (#61523, @dixudx)
• Cluster Autoscaler 1.2.0 - release notes available here: https://github.com/kubernetes/autoscaler/releases (#61561, @mwielgus)
• Fix mounting of UNIX sockets(and other special files) in subpaths (#61480, @gnufied)
• kubectl patch now supports --dry-run. (#60675, @timoreimann)
• fix sorting taints in case the sorting keys are equal (#61255, @dixudx)
• NetworkPolicies can now target specific pods in other namespaces by including both a namespaceSelector and a podSelector in the same peer element. (#60452, @danwinship)
• include node internal ip as additional information for kubectl (#57623, @dixudx)
• Add apiserver configuration option to choose audit output version. (#60056, @crassirostris)
• make test-cmd now works on OSX. (#61393, @totherme)
• Remove kube-apiserver --storage-version flag, use --storage-versions instead. (#61453, @hzxuzhonghu)
• Bump Heapster to v1.5.2 (#61396, @kawych)
• Conformance: ReplicaSet must be supported in the apps/v1 version. (#61367, @enisoc)
• You can now use the base64decode function in kubectl go templates to decode base64-encoded data, for example kubectl get secret SECRET -o go-template='{{ .data.KEY | base64decode }}'. (#60755, @glb)
• Remove 'system' prefix from Metadata Agent rbac configuration (#61394, @kawych)
• Remove --tls-ca-file flag. (#61386, @hzxuzhonghu)
• fix sorting tolerations in case the keys are equal (#61252, @dixudx)
• respect fstype in Windows for azure disk (#61267, @andyzhangx)
• --show-all (which only affected pods and only for human readable/non-API printers) is inert in v1.11, and will be removed in a future release. (#60793, @charrywanganthony)
• Remove never used NewCronJobControllerFromClient method (#59471, @dmathieu)
• Support new NODE_OS_DISTRIBUTION 'custom' on GCE (#61235, @yguo0905)
• Fixed #61123 by triggering syncer.Update on all cases including when a syncer is created (#61124, @satyasm)
  • on a new add event.
• Unready pods will no longer impact the number of desired replicas when using horizontal auto-scaling with external metrics or object metrics. (#60886, @mattjmcnaughton)
• include file name in the error when visiting files (#60919, @dixudx)
• Implement preemption for extender with a verb and new interface (#58717, @resouer)
• kube-cloud-controller-manager flag --service-account-private-key-file is removed in v1.11 (#60875, @charrywanganthony)
• kubeadm: Add the writable boolean option to kubeadm config. The option works on a per-volume basis for *ExtraVolumes config keys. (#60428, @rosti)
• DaemonSet scheduling associated with the alpha ScheduleDaemonSetPods feature flag has been removed from the 1.10 release. See https://github.com/kubernetes/enhancements/issues/548 for feature status. (#61411, @liggitt)
• Bugfix for erroneous upgrade needed messaging in kubernetes worker charm. (#60873, @wwwtyro)
• Fix data race in node lifecycle controller (#60831, @resouer)
• Nodes are not deleted from kubernetes anymore if node is shutdown in Openstack. (#59931, @zetaab)
• "beginPort+offset" format support for port range which affects kube-proxy only (#58731, @yue9944882)
• Added e2e test for watch (#60331, @jennybuckley)
• kubelet's --cni-bin-dir option now accepts multiple comma-separated CNI binary directory paths, which are search for CNI plugins in the given order. (#58714, @dcbw)