vertical-pod-autoscaler/enhancements/3919-customized-recommender-vpa/README.md
Today, the current VPA recommends CPU/Mem requests based on one recommender, which recommends the future requests based on the historical usage observed in a rolling time window. As there is no universal recommendation policy that applies to all types of workloads, this KEP suggests supporting multiple customized recommenders in VPA. Thus, users can run different recommenders for different workloads, as they may exhibit very distinct resource usage behaviors.
A VPA is used to recommend the requested resources of containers in pods when the actual CPU/memory usage of a container is significantly different from the resources requested. Resource usage-based recommendation is the basic approach that resizes containers according to the actual usage observed and is implemented in the default VPA recommender. Users can configure the time window and a certain percentile of observed usage in the past as the prediction of future requests/limits for CPU/memory.
However, as containers running different types of workloads may have different resource usage patterns, there is no universal policy that applies to all. The existing VPA recommender may not accurately predict future resource usage when containers exhibit certain resource usage behaviors, such as trending, periodically changing, or occasional spikes, resulting in significant over-provisioning and OOM kills for microservices. Learning different types of resource usage behaviors for workloads and applying different algorithms to improve resource utilization (CPU and Memory) predictions can significantly reduce over-provisioning and OOM kills in VPA.
Containers used in monitoring may receive load periodically to process but need to be long-running to listen to incoming traffic. Thus, these containers usually exhibit cyclic patterns, alternating between usage spikes and idling. Resizing containers according to usage observed in the previous time window may always lead to under-provision for a short period when the load spikes just arrive. The problem will happen for memory if the cyclic pattern length is >8 days. For CPU, the problem may be visible for example with lower usage on the weekend. The problem will even lead to frequent pod evictions when the pod was resized according to the idling period and the host resource has been taken by other pods.
Containers running spark/deep learning training workloads are known to show recurring and repeating patterns in resource usage. Prior research has shown that different but recurrent behaviors occur for different containerized tasks, such as Spark or deep learning training. These common patterns can be represented by phases, which display similar resource usage of computational resources over time. There are common sequences of patterns for different executions of the workload and they can be used to proactively predict future resource usage more accurately. The default recommender in the current VPA adopts a reactive approach so a more proactive recommender is needed for these types of workload.
The following describes the details of implementing a first-citizen approach to
support the customized recommender. Namely, a dedicated field recommenders is
added to the VPA crd definition in deploy/vpa-v1.crd.yaml. We identify
recommender to use with a struct containing name of the recommender rather than
using a plain string to identify the recommender. This will allow us to pass
parameters to recommenders if we need to do that in the future.
validation:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: []
properties:
recommenders:
type: array
items:
properties:
name:
description: Name of the recommmender
type: string
type: object
targetRef:
type: object
updatePolicy:
type: object
Correspondingly, the VerticalPodAutoscalerSpec in pkg/apis/autoscaling.k8s.io/v1/types.go
should be updated to include the recommenders field.
// VerticalPodAutoscalerRecommenderSelector points to a specificic Vertical Pod Autoscaler recommender
// in the future it might pass parameters to the recommender.
type VerticalPodAutoscalerRecommenderSelector struct {
// Name of the recommender responsible for generating recommendation for this object.
Name string `json:"name" protobuf:"bytes,1,opt,name=name"`
}
// VerticalPodAutoscalerSpec is the specification of the behavior of the autoscaler.
type VerticalPodAutoscalerSpec struct {
// TargetRef points to the controller managing the set of pods for the
// autoscaler to control - e.g. Deployment, StatefulSet. VerticalPodAutoscaler
// can be targeted at controller implementing scale subresource (the pod set is
// retrieved from the controller's ScaleStatus) or some well known controllers
// (e.g. for DaemonSet the pod set is read from the controller's spec).
// If VerticalPodAutoscaler cannot use specified target it will report
// ConfigUnsupported condition.
// Note that VerticalPodAutoscaler does not require full implementation
// of scale subresource - it will not use it to modify the replica count.
// The only thing retrieved is a label selector matching pods grouped by
// the target resource.
TargetRef *autoscaling.CrossVersionObjectReference `json:"targetRef" protobuf:"bytes,1,name=targetRef"`
// Describes the rules on how changes are applied to the pods.
// If not specified, all fields in the `PodUpdatePolicy` are set to their
// default values.
// +optional
UpdatePolicy *PodUpdatePolicy `json:"updatePolicy,omitempty" protobuf:"bytes,2,opt,name=updatePolicy"`
// Controls how the autoscaler computes recommended resources.
// The resource policy may be used to set constraints on the recommendations
// for individual containers. If not specified, the autoscaler computes recommended
// resources for all containers in the pod, without additional constraints.
// +optional
ResourcePolicy *PodResourcePolicy `json:"resourcePolicy,omitempty" protobuf:"bytes,3,opt,name=resourcePolicy"`
// Recommender responsible for generating recommendation for this object.
// List should be empty (then the default recommender will generate the
// recommendation) or contain exactly one recommender.
// +optional
Recommenders []*VerticalPodAutoscalerRecommenderSelector `json:"recommenders,omitempty" protobuf:"bytes,4,opt,name=recommenders"`
}
When creating a recommender object for recommendations, the recommender main routine should initiate itself
with a predefined recommender name, which can be defined as a constant in the pkg/recomender/main.go routine,
const RecommenderName = "default"
recommender := routines.NewRecommender(config, *checkpointsGCInterval, useCheckpoints, RecommenderName, *vpaObjectNamespace)
where the routines.NewRecommender can pass the RecommenderName to the clusterState object.
// NewRecommender creates a new recommender instance.
// Dependencies are created automatically.
// Deprecated; use RecommenderFactory instead.
func NewRecommender(config *rest.Config, checkpointsGCInterval time.Duration, useCheckpoints bool, recommender_name string, namespace string) Recommender {
clusterState := model.NewClusterState(recommender_name)
return RecommenderFactory{
ClusterState: clusterState,
ClusterStateFeeder: input.NewClusterStateFeeder(config, clusterState, *memorySaver, namespace),
CheckpointWriter: checkpoint.NewCheckpointWriter(clusterState, vpa_clientset.NewForConfigOrDie(config).AutoscalingV1()),
VpaClient: vpa_clientset.NewForConfigOrDie(config).AutoscalingV1(),
PodResourceRecommender: logic.CreatePodResourceRecommender(),
CheckpointsGCInterval: checkpointsGCInterval,
UseCheckpoints: useCheckpoints,
}.Make()
}
// NewClusterState returns a new ClusterState with no pods.
func NewClusterState(recommender_name string) *ClusterState {
return &ClusterState{
RecommenderName: recommender_name,
Pods: make(map[PodID]*PodState),
Vpas: make(map[VpaID]*Vpa),
EmptyVPAs: make(map[VpaID]time.Time),
aggregateStateMap: make(aggregateContainerStatesMap),
labelSetMap: make(labelSetMap),
}
}
Therefore, when loading VPA objects to the clusterStateFeeder, it should
ignore VPA objects that don't have an item in the recommenders field with
name equal to the current clusterState’s RecommenderName.
func implicitDefaultRecommender(selectors[]*VerticalPodAutoscalerRecommenderSelector) bool {
return len(selectors) == 0
}
func selectsRecommender(selectors[]*VerticalPodAutoscalerRecommenderSelector, name *string) bool {
for _, s := range(selectors) {
if s.Name == *name {
return true
}
}
return false
}
// Fetch VPA objects and load them into the cluster state.
func (feeder *clusterStateFeeder) LoadVPAs() {
// List VPA API objects.
allVpaCRDs, err := feeder.vpaLister.List(labels.Everything())
if err != nil {
klog.Errorf("Cannot list VPAs. Reason: %+v", err)
return
}
var vpaCRDs []*vpa_types.VerticalPodAutoscaler
for _, vpaCRD := range allVpaCRDs {
currentRecommenderName := feeder.clusterState.RecommenderName
if (!implicitDefaultRecommender(vpaCRD.Spec.Recommenders) && !selectsRecommender(vpaCRD.Spec.Recommenders, currentRecommenderName)) {
klog.V(6).Infof("Ignoring the vpaCRD as current recommender's name %v doesn't appear among its recommenders", currentRecommenderName)
continue
}
vpaCRDs = append(vpaCRDs, vpaCRD)
klog.V(3).Infof("Fetched %d VPAs.", len(vpaCRDs))
// Add or update existing VPAs in the model.
vpaKeys := make(map[model.VpaID]bool)
…
feeder.clusterState.ObservedVpas = vpaCRDs
}
The customized recommender is supposed to be deployed as a separate deployment that is chosen by different sets of VPA objects. Each VPA object is supposed to choose only one recommender at a time. The way how the default recommender and the customized recommender are running and interacting with VPA objects are shown in the following drawing.
Though we do not support a VPA object to use multiple recommenders in this
proposal, we leave the possibility of necessary changes of using multiple
recommenders in the future. Namely, we define recommenders to be an array
instead of an object, but we support one element only in this proposal. We
modify the admission controller to validate that the array has <= 1 elements.
We will add the following check in the
func validateVPA(vpa *vpa_types.VerticalPodAutoscaler, isCreate bool)
function.
if len(vpa.Spec.Recommenders) > 1 {
return fmt.Errorf("VPA object shouldn't specify more than one recommender.")
}
For now we don't allow users to pass any parameters to recommenders but we leave that as a possibility for the future. We do this by using a struct (which can be extended if we need to) rather than a string.
For cluster upgrades, the VPA from the previous version will continue working as before. There is no change in behavior or flags which have to be enabled or disabled.
Add a webhook system for customized recommendations. The default VPA recommender would call any available recommendation webhooks, and if any of them make a recommendation, the recommender would use that recommendation instead of making its own. If none make a recommendation, it would make its recommendation as it currently does. The plugin alternative is rejected because it involves much more design changes and code changes. It might be considered in the future if there are more use cases where running multiple recommenders for the same VPA object is needed.
Add a label for the CRD object to denote the recommender’s name. When making
recommendations in the recommender, only the VpaCrds with the label
recommender=default will be loaded and updated by the existing recommender.
A label selector approach is rejected because it is too powerful and users can easily
ignore those labels and misconfigure the VPA objects.