Release notes for kops 1.12 series

Significant changes

kops 1.12 enables etcd-manager by default. For kubernetes 1.12 (and later) we default to etcd3. We also enable TLS for etcd communications when using etcd-manager. The upgrade is therefore disruptive to the masters. More information is in the etcd migration documentation. This documentation is useful even if you are already using etcd3 with TLS.
Components are no longer allowed to interact with etcd directly. Calico will be switched to use CRDs instead of directly with etcd. This is a disruptive upgrade, please read the calico notes in the etcd migration documentation

Required Actions

Please back-up important data before upgrading, as the etcd2 to etcd3 migration is higher risk than most upgrades. The upgrade is disruptive to the masters, see notes above.
Note that the upgrade for Calico users is disruptive, because it requires switching from direct-etcd-storage to CRD backed storage.

Full change list since 1.11.0 release

1.11.0 to 1.12.0-alpha.1

machine-type generator: Warn if instance type not in ENI map @justinsb #6118
Include name of unhealthy component in validation error @justinsb #6122
Bump alpha channel kubernetes versions @justinsb #6123
Add missing locking to awsmock LaunchConfigurations @justinsb #6124
Add a1 and c5n instance types @justinsb #6117
Simplify makefile for update-machine-types @justinsb #6121
Update docs, removing brew --devel @mikesplain #6125
machine-types: remove duplicate dedup @justinsb #6127
Update amazon cni to 1.3.0 @mikesplain #6128
Enable HPA tolerance configuration @rlees85 #6130
Update addons dashboard version @jeefy #6136
Spotinst: Bump controller image @liranp #6129
Add cni to usage network option for kops create cluster @nak3 #6139
Workspace updates for bazel / fix tests @mikesplain #6144
Promote alpha channels to stable @mikesplain #6146
Add GCE europe-north1-{a,b,c} @eetujalonen #6152
Add self to security contacts @mikesplain #6147
Fix missed stable channel upgrade path @mikesplain #6158
Fix Calico upgrade job to use the correct version @tmjd #6156
Fix for when node and master use the same SG. @rdrgmnzs #6175
Add experimental and metrics flags for docker @rbtcollins #6171
Add y flag for upgrade command for consistency @mikesplain #6177
Add-ons spec example is missing "manifest". @qlikcoe #6170
ExperimentalAllowedUnsafeSysctls has moved to AllowedUnsafeSysctls in k8s 1.11 @rdrgmnzs #6179
Let a user specify the validation timeout when rotating a cluster. @rdrgmnzs #6185
fix(docs): fix the compatibility matrics on hpa.md @Cryptophobia #6193
bump prometheus-operator version and deploy file @zouyee #6196
update heapster version and mark it retired @zouyee #6195
Add Docker 18.06.1 for CentOS and RHEL 7 @bcorijn #6202
Print --name with kops update cluster @joshbranham #6208
Add --post-drain-delay to rolling-update cluster command @rifelpet #6211
Adding kubernetes-dashboard v1.10.1 deployment to kops addons @schweizerbolzonello #6224
Consider pending pods to be a validation failure @justinsb #6231
Adding support for the new Stockholm region @liranp #6212
Document how to update an existing vendored dependency @justinsb #6238
Update to k8s 1.12 libraries @rdrgmnzs #5932
Bump channels and bump alpha to latest @mikesplain #6239
Automagically use curl instead of wget if that's what's available @eherot #6090
cloudmock: replace unimplemented methods with interface embedding @justinsb #6243
bazel: cleanup gobindata generation @justinsb #6235
Update apimachinery for k8s 1.12 @justinsb #6245
Bulk spelling fixes @justinsb #6242
Don't panic when an etcd cluster is added @justinsb #6180
Update aws-sdk-go to 1.16.9 @justinsb #6237
Add p3dn.24xlarge @mikesplain #6253
Rationalize deserialiation code @justinsb #6259
Always log when a retry loop fails @justinsb #6260
Update compatibility for v1.11.0 @mikesplain #6258
AWS SDK v1.16.11 @gambol99 #6276
nodeup: include underlying error in error message @andrestc #6279
release process: add the relnotes command @justinsb #6269
Fix missed error check in hasPlaceHolderIP @justinsb #6272
Create dev-upload tasks, for a faster upload during dev builds @justinsb #6233
Update recommended kubernetes version @justinsb #6271
Release notes for 1.11 @justinsb #6270
fixed the sentence mistake @abhijitio #6281
update calico version to version 3.4.0 @chrisz100 #6263
Remove duplicate Deployment for prometheus-operator @Smirl #6265
Update aws-china.md @qqshfox #6262
Recognize 2019 as a year @justinsb #6288
Change jessie to stretch @abhijitio #6293
Included type in SSL certificate documentation @walkafwalka #6289
Update distroless @justinsb #6287
Promote alpha kubernetes versions to stable @justinsb #6298
Create prow-postsubmit target for release candidates @justinsb #6299
Include windows build in distribution @justinsb #6300
Fix kubelet api admin @gambol99 #6312
GCE terraform: map source tags in firewallrule @justinsb #6295
GCE terraform: support labels @justinsb #6296
Add extra privilege to prometheus-k8s ClusterRole #6305
Kubelet API RBAC Manifest @gambol99 #6317
Upgrading coredns version to 1.3.0 @harshal-shah #6326
Release 1.12.0-alpha.1 @justinsb #6257
Retry Logging @gambol99 #6327
Fix prow-postsubmit by copying prebuilt archive in bazel @justinsb #6328
Remove Initializers from default admission plugins for 1.12+ @liggitt #6350
include docker 18.06.1 missed dependency @nareshku #6338
Fix alternative AWS partitions in custom instance profiles @rifelpet #6226
Add doc regarding upgrading to CoreDNS @joshbranham #6344
AWS: Enable ICMP Type 3 Code 4 for API server ELBs @davidarcher #6297
Additional Storage & Volume Mounting @gambol99 #6066
kOps for Openstack @jrperritt,@drekle,@wozniakjan,@marsavela #6351
Update go version to 1.10.8 @justinsb #6401
Suffix openstack subnet name with cluster name @wozniakjan #6380
Update upgrade.md @ms4720 #6396
minor grammar improvements to kops terraform docs @discdiver #6301
Docs: Drop last DrainAndValidateRollingUpdate note @meeee #6374
Allow users to set kubelet cpu-cfs-quota and cpu-cfs-quota-period flags @wndhydrnt #6375
implement etcd status for openstack @zetaab #6381
remove using deviceowner when filtering existing routerinterfaces @zetaab #6382
ignore openstack managed volume tags @zetaab #6383
kops version: Add --short flag, use it to get version in scripts @justinsb #6232
find sshkey resource when updating cluster @zetaab #6384
implement GetCloudGroups for openstack @zetaab #6386
minor fixes to openstack @zetaab #6387
fix openstack lb pool member logic @zetaab #6388
Support "egress: External" to avoid configuring networking @justinsb,@cassandracomar,@moustafab #6218
Bump alpha channels @mikesplain #6405
Update bazel rules @mikesplain #6406
implement delete cluster for openstack @zetaab #6385
Openstack Floating IP Deletion @drekle #6425
update openstack documentation @zetaab #6423
Updated OWNERS file to include link to docs @rlenferink #6450
[jjo] add docker-ce 18.06.2 for CVE-2019-5736 @jjo #6460
Add permission for CreateTag on ENI to amazon-vpc-cni-k8s @nak3 #6389
Document etcd3 migration process @justinsb #6408
Normalize etcd cluster provider names @justinsb #6410
Support etcd-manager v3, suitable for backporting @justinsb #6411
Openstack loadbalancers erronous modification requests @drekle #6413
fix typos for addon doc @fatsheep9146 #6416
upgrade calico to 2.6.12 to fix TTA-2018-001 @mechpen #6422
Use the forward plugin instead of proxy plugin in CoreDNS @rajansandeep #6424
Update bazel workspace @mikesplain #6426
Fix machine types and cleanup makefile @mikesplain #6427
Add jessie patch @jjo,@mikesplain #6461
Allow NodeAuthorizer to speak via HTTP Proxy if configured @KashifSaadat #6468
Updated Canal manifest to v3.5.0 for k8s v1.12+ @KashifSaadat #6469
Update document for GPU support @yujunz #6246
Fixing kops-4049 @mmerrill3 #6210
kube-apiserver: Add oidc-required-claim flag @jeyglk #6453
add OWNERS file to openstack spesific folders @zetaab #6367
Update Loadbalancer Pools @drekle #6433
fix hostnames in kops openstack @zetaab #6442
implement ig deletegroup for openstack @zetaab #6418
Removing openstack credential file support @drekle #6480
fix error when updating/creating lb in openstack @zetaab #6431
recheck floatingip after server is active @zetaab #6432
Ability to scale down instancegroup in openstack @zetaab #6421
expose DryRunTarget changes and deletions @zetaab #6415
support both octavia and old lbaasv2 api in openstack @zetaab #6438
Guess SSH usernames for RHEL & Centos in toolbox dump @justinsb #6487
Choose docker version 18.06.2 for k8s >= 1.12 @justinsb #6488
Install kubelet config for default centos user @justinsb #6489
Update the CoreDNS manifest @rajansandeep #6485
docs: improve the queries for finding RHEL/CentOS images @justinsb #6486
Workaround for overlay2 vs rhel-family docker bug @justinsb #6491
retry l3floatingip list in fresh cluster @zetaab #6497
Update 1.12 addon manifests to use apps/v1, rbac v1 @liggitt #6397
Fix package name & version for container-selinux @justinsb #6492
AWS Mixed Instances Policy / Fleet @gambol99 #6277
Adding Comment @gambol99 #6508
Kube Proxy Metrics Option @gambol99 #6513
Sprig (Toolbox Templating) @gambol99 #6515
Etcd memory and cpu requests @integrii #6313
Map docker 18.06.3 @justinsb #6523
Make docker 18.06.3 the default for k8s >= 1.12 @justinsb #6524
Document strategy for cve_2019_5736 @justinsb #6522
Try using chattr to mark docker-runc as immutable @justinsb #6506
Simple mirror support @justinsb #6503
Bump etcd-manager version to 3.0.20190224 @justinsb #6526
update gophercloud vendor dependencies @zetaab #6478
specify dns servers to openstack subnet @zetaab #6530
possibility to specify floatingip subnet for resources in openstack @zetaab #6477
Add Experimental Cluster Signing Duration flag @pgdagenais #6525
set net.ipv4.ip_local_reserved_ports to the KubeAPIServer ServiceNodePortRange parameter on nodeup @sp-joseluis-ledesma #6343
spread instances equally to all AZs @zetaab #6534
update-machine-types: more metal instance types @justinsb #6551
Add changelist for 1.11.1 @justinsb #6565
Fix panic when using etcd-manager and resource requests are nil @KashifSaadat #6563
Promote Kubernetes 1.11.7 to stable @olemarkus #6566
Upgrade alpha to latest @mikesplain #6568
implement delete instance, this is needed in rolling-update @zetaab #6576
Stop setting deprecated --allow-privileged Kubelet flag in 1.14 @mtaufen #6340
Openstack Security Group hardening @drekle #6521
Update embargo doc link in SECURITY_CONTACTS and change PST to PSC @joelsmith #6601
Instance LaunchConfig/Template Bug Fix @gambol99 #6590
add docker.insecureRegistries flag @kimxogus #6586
Add line breaks in example release cycle @MMeent #6591
[jjo] Update Weave Net to version 2.5.1 @jjo #6370
Adding installation guidelines for Windows @EchoDelta #6594
Remove confusing comma in README @mattjmcnaughton #6607
Add ServiceAccountKeyFile to KubeAPIServerConfig @Smirl #6578
moving chrisz100 to approver level @chrisz100 #6434
Fix dashboard yaml that returned 404 @mausch #6479
Replace Y / N Markings of Compatibility Matrix in readme with ✔ / ❌ @compilenix #6539
Rename addon.yml to addon.yaml @jsharpe #6323
addons/cluster-autoscaler: Add jq installation for OSX environment @iBluemind #6567
Update docs on authentication @flands #6575
Omit IP-in-IP protocols in Openstack CNI Rules @marsavela #6614
External out-of-tree CloudControllerManager support for openstack @zetaab #6444
Use EnsureTask for create static pod directory @Smirl #6616
Fix documentation about targetGroupArn key @phyrog #6611
Update rolling_update.md @rj03hou,@Pharb #6247
fix typo @zqm19941101 #6017
Correcly handle CRLF in the manifest @gtrafimenkov #6570
Add support for Docker 18.09.3. @tsuna #6347
Fix confusing k8s upgrade docs for Terraform users @tspacek,@justinsb #6275
Added Audit Webhook config @mbelangerupgrade,@jpbelangerupgrade #6361
Spotinst: Avoid spurious changes @liranp #6028
Fix amazon-vpc-routed-eni yaml template @tvi #6502
Replace gcr.io URL with k8s.gcr.io vanity URL @justinsb #6623
support gossip for AliCloud @LilyFaFa #6319
add natGateways tasks for ALICloud @LilyFaFa #6402
Fix some of the docker package names & versions @justinsb #6620
Apply scope fix in #6502 to all manifest versions @tvi,@justinsb #6622
Add --kubeconfig flag to kops export kubecfg @adamyy #5955
add support to set cluster spec.kubelet @phedoreanu #6619
Upgrade bazel gazelle @mikesplain #6609
Fix typo @justinsb #6621
Support g3s for gpu driver installation @reverson #6538
Fix docker-healthcheck to work around Docker bug. @tsuna #6448
docs: create checklist for new kubernetes version @justinsb #5818
Fix metrics server addon @itskingori #6201
Always create /var/lib/kubelet, even in bootstrap mode @justinsb #5982
Launch Template Feature Flag @gambol99 #6512
Remove docker-prestart hook @stevenjm #6564
kops 1.12 configuration for calico: use CRDs @justinsb #6358
Quick Clean @gambol99 #6634
Sync data-types for webhook config with upstream @justinsb #6626
Add manage security groups for loadbalancers @zetaab #6632
Enable etcd-manager / etcd3 / etcd-tls in kops 1.12 @justinsb #6359
Use EnsureTask for internal api route53 record @Smirl #6629
Added reminder to publish conformance results in release process @chrisz100 #6640
Update aws-china.md @qqshfox #6643
Openstack server name collisions @drekle #6650
tiny backslash arrangement @sevenfourk #6652
Openstack environment escaping @drekle #6657
Update upgrade.md @gamename #6654
add ALI flags @LilyFaFa #6628
Override volume zone name @zetaab #6655
Updated Flannel manifest to 0.11.0 @gordonbondon #6660
Update flannel version in bootstrapchannelbuilder @gordonbondon,@justinsb #6663
Add flags for TLS Cipher suites customization for API Server, Kubelet and Controller-Manager @rochacon #6470
If using etcd-backup and TLS is enabled, pass relevant options @KashifSaadat #6562
Bump etcd-manager / etcd-backup to 3.0.20190325 @justinsb #6664
2048 - Add cloudLabels as tags to API ELB resource @ryan-dyer #6646
Bump K8s 1.11 to 1.11.9 in the alpha channel @olemarkus #6665
Upgrade rules go @mikesplain #6667
Fix a missing dep lock @mikesplain #6668

1.12.0-alpha.1 to 1.12.0-alpha.2

Support download protokube from mirror @justinsb #6673
Promote alpha to stable and update alpha @mikesplain #6669
Upload protokube to github as part of release @justinsb #6674
Use CNI 0.7.5 @justinsb #6671
Put 1.12 into stable channel, for users of kops 1.12-alphas @justinsb #6672
Support mirrors with restricted characters @justinsb #6675

1.12.0-alpha.2 to 1.12.0-alpha.3

Fix Key error change Overrides to Override @granular-ryanbonham #6691
Add selector back to calico 1.12 deployment @justinsb #6682
Update etcd-manager to 1.0.20190328 @justinsb #6695

1.12.0-alpha.3 to 1.12.0-beta.1

Fix tagging and remove tagging elbs @mikesplain #6705
Add DNS Resource Settings @granular-ryanbonham #6731
Update instances types @mikesplain #6733
Update kube-dns 1.3.0 to 1.3.3 @mikesplain #6734
kube-dns-autoscaler: Add node watch to permissions @justinsb #6740
Increase apiserver timeout to 45 seconds @justinsb #6743
Fix issue #6700: User Data for launch templates & other terraform issues @rdrgmnzs #6732

1.12.0-beta.1 to 1.12.0-beta.2

kube-dns: Update to 1.14.13 @justinsb #6741
Launch Template use version number as well as name.@granular-ryanbonham #6755
use dynamic s3 prefix in addAmazonVPCCNIPermissions func @bksteiny #6765

1.12.0-beta.2 to 1.12.0

IAM Permission to Support Scaling from 0 with Lauch Templates @granular-ryanbonham #6861
Avoid concurrent write corruption to /etc/hosts @justinsb,@granular-ryanbonham #6893
Add i3en instance types @mikesplain #6898
Add t3a family @mikesplain #6905
Use existing SSHKeyName if no public key is created. @rralcala #6886
bazel: fix distroless imports for latest bazel @justinsb #6910
pkg/model: Fix dropped error @alrs #6911
Add ability to specify cpuRequest for API Server @granular-ryanbonham #6706
KubeAPIServer HTTP2 Stream Parameter @gambol99 #6913
Add support for AWS ap-east-1 region @wxdao #6835
Add min-resync-period for Controller Manager @maruina #6737
Allow the AWS IAM Authenticator image name to be overridden @rifelpet #6730
Add cpu management policy config @lynchc #5961
Carry Provisioned IOPS to Terraform and CloudFormation templates @MathieuMailhos #6776
update tolerations to openstack external cloud provider @zetaab #6821
Fix typo in aws-iam-authenticator image field name @rifelpet #6840
add the registry-qps kubelet flag @sp-joseluis-ledesma #6357
Deep-copy proto state to prevent concurrent modification @justinsb #6707
Publish utils.tar.gz to github releases also @justinsb #6680
Allow uneven etcd zones @adammw #6641
Add terraform support for additional CIDR blocks. @rdrgmnzs #6693
Canal manifest updates for k8s v1.12+ @KashifSaadat #6823
Update to etcd-manager 1.0.20190509 @justinsb #6917
S3 VFS: Default to current region from metadata service @justinsb,@granular-ryanbonham #6943
etcd-manager: Update to 3.0.20190513 @justinsb #6959
Fix Docker not being installed on Ubuntu 16.04 @meeee #6965
Issue #6945 @pkutishch,@mikesplain #6951

1.12.0 to 1.12.1

Don't panic when deleting instancegroups @justinsb #7000
etcd-manager: update to 3.0.20190516 @justinsb #7007
Terraform: fix options field, should be spot_options @kimxogus #6988

1.12.1 to 1.12.2

Mark ENI 0 as delete_on_termination for LaunchTemplates @granular-ryanbonham #7094

1.12.2 to 1.12.3

Cherry pick of #7211: Use NodeAuthorizer config options instead of soely @jacksontj #7232
Cherry pick of #7219: Make an actual deep-copy of the state @jacksontj #7235
Upgrade Calico to 3.7.2 @asincu #7051
Update canal to 3.6.4, for TTA-2019-002 @justinsb #7275
Bumping calico to 3.7.4. @michalschott #7249
Cherry pick of #7185: Replace behavior for aws hostnameOverride @jacksontj #7308
Calico -> 3.7.4 for older versions @justinsb #7282
Bump etcd-manager to 3.0.20190801 @justinsb #7349
Warn/prevent if the version of etcd is unsupported with etcd-manager @justinsb #7340