docs/rfcs/20210223_external_connectivity_in_k8s.md
To give users out of the box experience Redpanda Kubernetes operator should pave the way for the networking stack to expose each Redpanda node in k8s environment.
The most performant and cost-efficient solution is to expose each process to the outside, by specifying host port in the container ports specification for each Redpanda node.
The Redpanda is focused on performance and its low tail latency. To not add more network hops this proposal tries to merit dynamic Kubernetes environment with the direct access to Redpanda process.
Service of type NodePort will have assigned port that is free on each node and will be used in the advertised kafka API configuration. The node port will be set in the host port configuration of Redpanda container along with internal, RPC and Admin ports. The external Kafka API port will be not configurable and arbitrarily chosen to internal Kafka API + 1. Init container will discover public IP of the host on which it was scheduled. The port will be set in the advertised Kafka API address.
The Kubernetes worker nodes must open port per Redpanda cluster which might be an additional attack vector. The additional firewalls should be managed to allow node access.
Redpanda can be easily managed in multiple environments, but vast range of users plan to manage their workload in Kubernetes environment. Saying that, teams would like to not introduce clients application into their Kubernetes clusters. The Kafka API requires that each broker will be reachable at any given time. Canonical way of exposing webservers in Kubernetes environment is to create Ingress resources that can be reconciled into cloud provider load balancer. Another way of exposing different kind of workloads is to expose their TCP port using Service type LoadBalancer and again cloud provider load balancer will be provisioned. On the other hand the TCP port can be exposed on each Kubernetes worker node using Service type Node Port. All the mentioned methods doesn't guarantee that each Redpanda node will be accessible individually. One could create Service type Node Port for each broker, but that would be a waste of ports in the whole cluster. The optimal solution would be to pick one port and take public IP of a node on which Redpanda is scheduled and use it in advertised Kafka API.
During testing in AWS environment the service type Node Port doesn't have assigned external IP. That's why init container needs to retrieve node information from the Kubernetes API and use it in advertised Kafka API address.
The operator should accommodate dynamic nature of any cloud where nodes can disappear or be rescheduled.
Exposing each Redpanda node to networks outside the Kubernetes cluster.
The Redpanda operator will make sure that from the network standpoint
brokers will be reachable. Users need to manage security of the
exposed Kubernetes worker nodes e.g. aws security groups.
The Kafka API requires that each broker will be reachable from the client side. To expose Redpanda nodes, but not add additional latency to the communication each Redpanda process will have assigned unique host port. In case of cloud providers the host VM need to be publicly reachable and have opened some range ports. The operator would be responsible to orchestrate Kubernetes to achieve that.
( This is the technical portion of the RFC )
None.
The host port need to be added in kafka port in the Statefulset definition. Service type node port needs to be added. The init container needs to know how to get the public IP of the node. New ServiceAccount need to be added with the RBAC rule for getting node information.
Each Kubernetes node needs to be exposed to the internet in order
to expose Redpanda node. Operator's responsibility is creation of
Redpanda configuration that has correctly registered advertised
Kafka API and assign correct unique and unused node port. With the
help of Kubernetes Service type node port the unused port can be
injected into container host port. This service doesn't have selector,
so that no rule should be created in iptable or any other CNI
implementation. Init container will retrieve from the Kubernetes API
server information about node and its public IP. That container will
then change the Redpanda configuration, so that internal and external
ports and addresses will be in Kafka API and Advertised Kafka API
section.
The nodes must be exposed to some other networks. This scope goes beyond the Kubernetes. There is no easy way for the operator to manage firewalls that can be blocking the network traffic.
The more natural way of integrating with k8s resource is to use service
type LoadBalancer per pod. This would add latency and cost will scale
as the bytes comes through.
Each instance could be registered in DNS provider of one's choice. The benefits with this approach is nice looking brokers names. On the other hand the latency of propagating change in DNS systems could not be acceptable.
Following external-dns
tutorial for headless services
there should be an easy way of exposing each pod. During the tests
in AWS EKS the DNS A records shows internal IP address of a Pod in
the AWS route 53 service.
Exposing all brokers via LoadBalancer and Envoy proxy. The Envoy can reroute connection to proper broker. The Envoy can be replaced with any other proxy Nginx etc. There will be bottleneck in hot path as every connection will go through LB + proxy. It doesn't scale well as one client can starve the network hardware.