AWS EKS 集群自动扩容 Cluster Autoscaler

配置policy

vim cluster-autoscaler-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

aws iam create-policy --policy-name AmazonEKSClusterAutoscalerPolicy --policy-document file://cluster-autoscaler-policy.json

vim trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::x:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/x"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.me-central-1.amazonaws.com/id/x:sub": "system:serviceaccount:kube-system:cluster-autoscaler"
        }
      }
    }
  ]
}

aws iam create-role --role-name AmazonEKSClusterAutoscalerRole --assume-role-policy-document file://"trust-policy.json"

aws iam attach-role-policy --policy-arn arn:aws:iam:policy/AmazonEKSClusterAutoscalerPolicy --role-name AmazonEKSClusterAutoscalerRole

wget https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml
编辑cluster-autoscaler-autodiscover.yaml
变更1，2的内容，增加3

kubectl apply -f cluster-autoscaler-autodiscover.yaml

kubectl annotate serviceaccount cluster-autoscaler -n kube-system eks.amazonaws.com/role-arn=arn:aws:iam::x:role/AmazonEKSClusterAutoscalerRole
kubectl patch deployment cluster-autoscaler -n kube-system -p '{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}}}}}'

报错
E0726 06:44:32.760321       1 aws_manager.go:125] Failed to regenerate ASG cache: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
        status code: 403, request id: 92ebcf31-31d0-47d0-80ac-ab157b605a8c

修复
新集群每个对象最好单独创建，不要重用

I0823 10:33:10.516504       1 pre_filtering_processor.go:67] Skipping ip-10-0-33-254.me-central-1.compute.internal - node group min size reached (current: 7, min: 7)
I0823 10:33:10.516563       1 static_autoscaler.go:623] Scale down status: lastScaleUpTime=2023-08-23 08:49:37.249534786 +0000 UTC m=-3578.906214192 lastScaleDownDeleteTime=2023-08-23 08:49:37.249534786 +0000 UTC m=-3578.906214192 lastScaleDownFailTime=2023-08-23 08:49:37.249534786 +0000 UTC m=-3578.906214192 scaleDownForbidden=false scaleDownInCooldown=false
I0823 10:33:10.516601       1 static_autoscaler.go:632] Starting scale down
I0823 10:33:10.516629       1 legacy.go:296] No candidates for scale down
I0823 10:33:20.533858       1 static_autoscaler.go:289] Starting main loop
I0823 10:33:20.534790       1 aws_manager.go:185] Found multiple availability zones for ASG "eks-prod-0815-50c4fbcc-cb1c-02c1-75d0-c54f2d81d038"; using me-central-1c for failure-domain.beta.kubernetes.io/zone label
I0823 10:33:20.535012       1 filter_out_schedulable.go:63] Filtering out schedulables
I0823 10:33:20.535027       1 filter_out_schedulable.go:120] 0 pods marked as unschedulable can be scheduled.
I0823 10:33:20.535036       1 filter_out_schedulable.go:83] No schedulable pods
I0823 10:33:20.535042       1 filter_out_daemon_sets.go:40] Filtering out daemon set pods
I0823 10:33:20.535048       1 filter_out_daemon_sets.go:49] Filtered out 0 daemon set pods, 0 unschedulable pods left
I0823 10:33:20.535071       1 static_autoscaler.go:552] No unschedulable pods
I0823 10:33:20.535094       1 static_autoscaler.go:575] Calculating unneeded nodes

horizontalpodautoscaler
需先安装Metrics Server，默认情况下它不会部署在 Amazon EKS 集群中
https://docs.aws.amazon.com/eks/latest/userguide/metrics-serv...

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: x
  namespace: dev
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: x
  minReplicas: 2
  maxReplicas: 6
  targetCPUUtilizationPercentage: 80