k8s v1.18增加了hpa v2beta2的behavior字段，可以更精细化的控制伸缩的行为：
- 若不指定behavior字段，则按默认的behavior行为执行伸缩；
- 若指定behavior字段，则按自定义的behavior行为执行伸缩；
一. demo
若behavior的策略(包括冷却时间+伸缩策略)不满足需求，可以通过自定义behavior精细化控制伸缩的策略。
比如下面的behavior:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: sample-app
spec:
...
behavior:
scaleUp:
policies:
- type: Percent
value: 900
periodSeconds: 300
scaleDown:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 1
periodSeconds: 10扩容时：
- 立即扩容；
- 每次扩容最大(1+9)*currentReplicas，即10倍的replicas；
- 1次扩容后，冷却300s后才能继续扩容；
缩容时：
- 冷却60s才进行缩容，每次缩容1个副本；
- 1次缩容后，冷却10s后才能继续缩容；
以上面的Hpa v2beta2的定义为例，查看其扩缩容的过程:
扩容，将指标猛增(1-->13)：
- 首先，按照指标计算，应该将副本数从1扩容到13；但由于scaleUp.policies的限制，最多扩容10倍，即1-->10个副本；
- 然后，根据指标计算，冷却300s后，最终将副本数扩容至13；
# kubectl describe hpa
Name: sample-app
Namespace: default
Labels: <none>
Annotations: <none>
Reference: Deployment/sample-app
Metrics: ( current / target )
"metric_hpa" on pods: 1 / 1
Min replicas: 1
Max replicas: 15
Behavior:
Scale Up:
Stabilization Window: 0 seconds
Select Policy: Max
Policies:
- Type: Percent Value: 900 Period: 300 seconds
Scale Down:
Stabilization Window: 60 seconds
Select Policy: Max
Policies:
- Type: Pods Value: 1 Period: 10 seconds
Deployment pods: 13 current / 13 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric metric_hpa
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 6m29s horizontal-pod-autoscaler New size: 10; reason: pods metric metric_hpa above target
Normal SuccessfulRescale 79s horizontal-pod-autoscaler New size: 13; reason: pods metric metric_hpa above target缩容，将指标猛降(13-->1):
- 首先，冷却60s后进行缩容，每次缩容1个副本；
- 然后，待上次缩容后10s，再次缩容1个副本；
- 最终，缩容至1个副本；
# kubectl describe hpa
Name: sample-app
Namespace: default
Labels: <none>
Annotations: <none>
Reference: Deployment/sample-app
Metrics: ( current / target )
"metric_hpa" on pods: 1 / 1
Min replicas: 1
Max replicas: 15
Behavior:
Scale Up:
Stabilization Window: 0 seconds
Select Policy: Max
Policies:
- Type: Percent Value: 900 Period: 300 seconds
Scale Down:
Stabilization Window: 60 seconds
Select Policy: Max
Policies:
- Type: Pods Value: 1 Period: 10 seconds
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric metric_hpa
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 12m horizontal-pod-autoscaler New size: 10; reason: pods metric metric_hpa above target
Normal SuccessfulRescale 6m59s horizontal-pod-autoscaler New size: 13; reason: pods metric metric_hpa above target
Normal SuccessfulRescale 3m25s horizontal-pod-autoscaler New size: 12; reason: All metrics below target
Normal SuccessfulRescale 3m9s horizontal-pod-autoscaler New size: 11; reason: All metrics below target
Normal SuccessfulRescale 2m54s horizontal-pod-autoscaler New size: 10; reason: All metrics below target
Normal SuccessfulRescale 2m38s horizontal-pod-autoscaler New size: 9; reason: All metrics below target
Normal SuccessfulRescale 2m23s horizontal-pod-autoscaler New size: 8; reason: All metrics below target
Normal SuccessfulRescale 2m7s horizontal-pod-autoscaler New size: 7; reason: All metrics below target
Normal SuccessfulRescale 112s horizontal-pod-autoscaler New size: 6; reason: All metrics below target
Normal SuccessfulRescale 34s (x5 over 96s) horizontal-pod-autoscaler (combined from similar events): New size: 1; reason: All metrics below target二. 源码分析
整个过程分以下几步：
- 首先，若未设置scaleDown的冷却时间，则配置默认冷却时间=300s；
- 然后，根据scaleUp/scaleDown的stabilizationWindow，计算目标副本数；
- 最后，根据scaleUp/scaleDown的Policies，计算目标副本数；
// pkg/controller/podautoscaler/horizontal.go
func (a *HorizontalController) normalizeDesiredReplicasWithBehaviors(hpa *autoscalingv2.HorizontalPodAutoscaler, key string, currentReplicas, prenormalizedDesiredReplicas, minReplicas int32) int32 {
// 1. 设置scaleDown的默认冷却时间
a.maybeInitScaleDownStabilizationWindow(hpa)
normalizationArg := NormalizationArg{
Key: key,
ScaleUpBehavior: hpa.Spec.Behavior.ScaleUp,
ScaleDownBehavior: hpa.Spec.Behavior.ScaleDown,
MinReplicas: minReplicas,
MaxReplicas: hpa.Spec.MaxReplicas,
CurrentReplicas: currentReplicas,
DesiredReplicas: prenormalizedDesiredReplicas}
// 2. 根据冷却时间计算副本数
stabilizedRecommendation, reason, message := a.stabilizeRecommendationWithBehaviors(normalizationArg)
normalizationArg.DesiredReplicas = stabilizedRecommendation
...
// 3. 根据策略计算副本数
desiredReplicas, reason, message := a.convertDesiredReplicasWithBehaviorRate(normalizationArg)
...
return desiredReplicas
}1. 设置scaleDown的默认冷却时间
若scaleDown.StabilizationWindowSeoncds未设置，则默认=300s；
// pkg/controller/podautoscaler/horizontal.go
func (a *HorizontalController) maybeInitScaleDownStabilizationWindow(hpa *autoscalingv2.HorizontalPodAutoscaler) {
behavior := hpa.Spec.Behavior
if behavior != nil && behavior.ScaleDown != nil && behavior.ScaleDown.StabilizationWindowSeconds == nil {
stabilizationWindowSeconds := (int32)(a.downscaleStabilisationWindow.Seconds()) // 默认=300s
hpa.Spec.Behavior.ScaleDown.StabilizationWindowSeconds = &stabilizationWindowSeconds
}
}2. 根据冷却时间计算副本数
根据stabilizationWindow计算目标副本数，最终返回recommendation:
- 首先，初始值=上一步计算的副本数(即指标计算的副本数)；
然后：
- 若扩容，则recommendation=min(最近stabilizationWindowSeconds的伸缩副本数)，这也意味着冷却stabilizationWindowSeconds；
- 若缩容，则recommendation=max(最近stabilizationWindowSeconds的伸缩副本数)，这也意味着冷却stabilizationWindowSeconds；
// pkg/controller/podautoscaler/horizontal.go
func (a *HorizontalController) stabilizeRecommendationWithBehaviors(args NormalizationArg) (int32, string, string) {
recommendation := args.DesiredReplicas
...
var betterRecommendation func(int32, int32) int32
// 扩容
if args.DesiredReplicas >= args.CurrentReplicas {
scaleDelaySeconds = *args.ScaleUpBehavior.StabilizationWindowSeconds
betterRecommendation = min // min函数
reason = "ScaleUpStabilized"
message = "recent recommendations were lower than current one, applying the lowest recent recommendation"
} else { // 缩容
scaleDelaySeconds = *args.ScaleDownBehavior.StabilizationWindowSeconds
betterRecommendation = max // max函数
reason = "ScaleDownStabilized"
message = "recent recommendations were higher than current one, applying the highest recent recommendation"
}
...
cutoff := time.Now().Add(-time.Second * time.Duration(scaleDelaySeconds))
for i, rec := range a.recommendations[args.Key] {
if rec.timestamp.After(cutoff) {
recommendation = betterRecommendation(rec.recommendation, recommendation)
}
...
}
...
return recommendation, reason, message
}3. 根据policies计算副本数
根据policies计算目标副本数
若是扩容：
- 根据scaleUp.Policies(percent/pods/period)计算扩容上限；
- 扩容上限必须 <= hpaMaxReplicas；
- 最终扩容副本数必须 <= 上一步计算的desiredReplicas；
若是缩容：
- 根据scaleDown.Policies(percent/pods/period)计算缩容上限；
- 缩容上限必须 >= hpaMinReplicas；
- 最终缩容副本数必须 >= 上一步计算的desiredReplicas；
// pkg/controller/podautoscaler/horizontal.go
func (a *HorizontalController) convertDesiredReplicasWithBehaviorRate(args NormalizationArg) (int32, string, string) {
var possibleLimitingReason, possibleLimitingMessage string
// 扩容
if args.DesiredReplicas > args.CurrentReplicas {
// 根据scaleUp.Policies计算扩容上限
scaleUpLimit := calculateScaleUpLimitWithScalingRules(args.CurrentReplicas, a.scaleUpEvents[args.Key], args.ScaleUpBehavior)
...
// 扩容上限必须 <= hpaMaxReplicas
maximumAllowedReplicas := args.MaxReplicas
if maximumAllowedReplicas > scaleUpLimit {
maximumAllowedReplicas = scaleUpLimit
possibleLimitingReason = "ScaleUpLimit"
possibleLimitingMessage = "the desired replica count is increasing faster than the maximum scale rate"
} else {
possibleLimitingReason = "TooManyReplicas"
possibleLimitingMessage = "the desired replica count is more than the maximum replica count"
}
// 扩容副本数必须 <= 上一步计算的desiredReplicas
if args.DesiredReplicas > maximumAllowedReplicas {
return maximumAllowedReplicas, possibleLimitingReason, possibleLimitingMessage
}
} else if args.DesiredReplicas < args.CurrentReplicas { // 缩容
// 根据scaleDown.Policies计算缩容上限
scaleDownLimit := calculateScaleDownLimitWithBehaviors(args.CurrentReplicas, a.scaleDownEvents[args.Key], args.ScaleDownBehavior)
...
// 缩容上限必须 >= hpaMinReplicas
minimumAllowedReplicas := args.MinReplicas
if minimumAllowedReplicas < scaleDownLimit {
minimumAllowedReplicas = scaleDownLimit
possibleLimitingReason = "ScaleDownLimit"
possibleLimitingMessage = "the desired replica count is decreasing faster than the maximum scale rate"
} else {
possibleLimitingMessage = "the desired replica count is less than the minimum replica count"
possibleLimitingReason = "TooFewReplicas"
}
// 缩容副本数必须 >= 上一步计算的desiredReplicas
if args.DesiredReplicas < minimumAllowedReplicas {
return minimumAllowedReplicas, possibleLimitingReason, possibleLimitingMessage
}
}
return args.DesiredReplicas, "DesiredWithinRange", "the desired count is within the acceptable range"
}扩容上限的计算，依据policy.percent/pods/period:
- 可以存在多个policy：由scaleUp.SelectPolicy决定是选择这些policy的max、min还是disable；
// pkg/controller/podautoscaler/horizontal.go
func calculateScaleUpLimitWithScalingRules(currentReplicas int32, scaleEvents []timestampedScaleEvent, scalingRules *autoscalingv2.HPAScalingRules) int32 {
var result int32
var proposed int32
var selectPolicyFn func(int32, int32) int32
if *scalingRules.SelectPolicy == autoscalingv2.DisabledPolicySelect {
return currentReplicas // Scaling is disabled
} else if *scalingRules.SelectPolicy == autoscalingv2.MinPolicySelect {
result = math.MaxInt32
selectPolicyFn = min // For scaling up, the lowest change ('min' policy) produces a minimum value
} else {
result = math.MinInt32
selectPolicyFn = max // Use the default policy otherwise to produce a highest possible change
}
for _, policy := range scalingRules.Policies {
replicasAddedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleEvents)
periodStartReplicas := currentReplicas - replicasAddedInCurrentPeriod
if policy.Type == autoscalingv2.PodsScalingPolicy { // Pods
proposed = periodStartReplicas + policy.Value
} else if policy.Type == autoscalingv2.PercentScalingPolicy { // Percent
// the proposal has to be rounded up because the proposed change might not increase the replica count causing the target to never scale up
proposed = int32(math.Ceil(float64(periodStartReplicas) * (1 + float64(policy.Value)/100)))
}
result = selectPolicyFn(result, proposed)
}
return result
}对于每个scaleUp的policy：
- 首先，计算过去policy.periodSeconds这段时间的伸缩总副本数；
- 然后，计算 当前副本数 - 过去policy.periodSconds时间内的伸缩副本数；
前面两步的目的：
- 是抹平policy.periodSeconds时间内的伸缩副本数；
- 即下一次伸缩距离上一次伸缩的冷却时间；
最后：
- 若policy.Type=Pods，则再 + policy.Pods.Value副本数；
- 若policy.Type=Percent，则再 * (1 + percent.Value)/100=最终副本数；
policy.periodSeconds时间内的伸缩总副本数的计算：
// pkg/controller/podautoscaler/horizontal.go
func getReplicasChangePerPeriod(periodSeconds int32, scaleEvents []timestampedScaleEvent) int32 {
period := time.Second * time.Duration(periodSeconds)
cutoff := time.Now().Add(-period)
var replicas int32
for _, rec := range scaleEvents {
if rec.timestamp.After(cutoff) {
replicas += rec.replicaChange // 汇总periodSeconds时间内的伸缩副本总数，扩容=+M，缩容=-N
}
}
return replicas
}缩容上限的计算，依据policy.percent/pods/period：
- 与扩容上限的计算方法类似；
区别在于：由于是缩容
- perioldStartReplicas = 当前副本数 + 过去periodSeoncds内伸缩副本数；
- 若policy.Type=Pods，则再 - policy.Pods.Value副本数；
- 若policy.Type=Percent，则再 * (1 - percent.Value)/100=最终副本数；
// pkg/controller/podautoscaler/horizontal.go
func calculateScaleDownLimitWithBehaviors(currentReplicas int32, scaleEvents []timestampedScaleEvent, scalingRules *autoscalingv2.HPAScalingRules) int32 {
var result int32
var proposed int32
var selectPolicyFn func(int32, int32) int32
if *scalingRules.SelectPolicy == autoscalingv2.DisabledPolicySelect {
return currentReplicas // Scaling is disabled
} else if *scalingRules.SelectPolicy == autoscalingv2.MinPolicySelect {
result = math.MinInt32
selectPolicyFn = max // For scaling down, the lowest change ('min' policy) produces a maximum value
} else {
result = math.MaxInt32
selectPolicyFn = min // Use the default policy otherwise to produce a highest possible change
}
for _, policy := range scalingRules.Policies {
replicasDeletedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleEvents)
periodStartReplicas := currentReplicas + replicasDeletedInCurrentPeriod
if policy.Type == autoscalingv2.PodsScalingPolicy { // Pod
proposed = periodStartReplicas - policy.Value
} else if policy.Type == autoscalingv2.PercentScalingPolicy { // Percent
proposed = int32(float64(periodStartReplicas) * (1 - float64(policy.Value)/100))
}
result = selectPolicyFn(result, proposed)
}
return result
}
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。