Horizontal Pod Autoscaler - Andreas Karis Blog (2024)

Understanding the Horizontal Pod Autoscaler (HPA)

The basics for Horiontal Pod Autoscaler (HPA) are well explained in:

For OpenShift 4.x, consider that there is a bug with the v1 API for autoscaling. However, the v2beta2 API works fine:

https://access.redhat.com/solutions/5428871

Getting the examples

Clone the following repository. Then, go to directory hpa-test:

12	`git clone https://github.com/andreaskaris/kubernetes.gitcd kubernetes/hpa-test`

You will find all required resources for the following steps. See REAME.md in case you want to build your own hpa-tester image.

Deploying an HPA tester deployment

First, deploy the HPA-Tester deployment. The deployment runs 2 applications which simulate a cumulative load. Meaning that COMBINED_CPU_MS and COMBINED_MEMORY_MB will be divided by the number of replicas for the deployment and each pod will then individually run that fraction of memory and CPU load. This check happens every SLEEP_TIME seconds.

Deploy the deployment in a dedicated namespace with:

1234	`oc new-project hpa-test \|\| oc project hpa-testoc apply -f role-list-deployments-pods.yamloc apply -f role-binding-list-deployments-pods.yamloc apply -f deployment-hpa-tester.yaml`

If you change the replica count, the entrypoint.sh will make sure to distribute the memory MB and CPU ms between the number of replicas.

Scaling basics based on the average CPU scaler metric

For the most basic example, .spec.template.spec.containers[0].resources is:

1234	`resources: requests: cpu: "1000m" memory: "1024Mi"`

The Horizontal Pod Autoscaler will by default compare a pod's actual CPU usage to the pod's requested values:https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. If a target raw value is set, the raw metric values are used directly. The controller then takes the mean of the utilization or the raw value (depending on the type of target specified) across all targeted Pods, and produces a ratio used to scale the number of desired replicas.Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. See the algorithm details section below for more information about how the autoscaling algorithm works.

Scaling based on absolute CPU values

It is also possible to scale for a specific CPU value. First, scale back the deployment to 1 replica or delete and recreate it. Then, apply the following HorizontalPodAutoscaler:

 1 2 3 4 5 6 7 8 91011121314151617181920

apiVersion: autoscaling/v2beta2kind: HorizontalPodAutoscalermetadata: annotations: name: hpa-tester namespace: testspec: maxReplicas: 20 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: hpa-tester metrics:  - type: Resource resource: name: cpu target: type: AverageValue averageValue: 200m

And wait for 5 minutes. The result will be 9 pods - ceil [(1600ms + overhead) / 800ms]:

 1 2 3 4 5 6 7 8 910111213141516171819202122232425

[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester 199m/200m 1 20 9 6m51s[root@openshift-jumpserver-0 hpa-test]# oc get podsNAME READY STATUS RESTARTS AGEhpa-tester-7bff6856bd-2dnjn 1/1 Running 0 2m22shpa-tester-7bff6856bd-5pf46 1/1 Running 0 7m16shpa-tester-7bff6856bd-bx49l 1/1 Running 0 2m7shpa-tester-7bff6856bd-d487n 1/1 Running 0 2m22shpa-tester-7bff6856bd-h2kd8 1/1 Running 0 112shpa-tester-7bff6856bd-mmk7c 1/1 Running 0 2m22shpa-tester-7bff6856bd-n48nj 1/1 Running 0 2m7shpa-tester-7bff6856bd-qhjtl 1/1 Running 0 2m7shpa-tester-7bff6856bd-wx8r8 1/1 Running 0 2m7s[root@openshift-jumpserver-0 hpa-test]# oc get podmetricsNAME CPU MEMORY WINDOWhpa-tester-7bff6856bd-2dnjn 192m 478724Ki 5m0shpa-tester-7bff6856bd-5pf46 199m 509580Ki 5m0shpa-tester-7bff6856bd-bx49l 194m 507920Ki 5m0shpa-tester-7bff6856bd-d487n 189m 507564Ki 5m0shpa-tester-7bff6856bd-h2kd8 190m 503232Ki 5m0shpa-tester-7bff6856bd-mmk7c 200m 503428Ki 5m0shpa-tester-7bff6856bd-n48nj 202m 506848Ki 5m0shpa-tester-7bff6856bd-qhjtl 197m 477332Ki 5m0shpa-tester-7bff6856bd-wx8r8 194m 479528Ki 5m0s

Scaling based on memory utiliation

It is also possible to scale for a specific memory utiliation. First, scale back the deployment to 1 replica or delete and recreate it. Then, apply the following HorizontalPodAutoscaler:

 1 2 3 4 5 6 7 8 91011121314151617181920

apiVersion: autoscaling/v2beta2kind: HorizontalPodAutoscalermetadata: annotations: name: hpa-tester namespace: testspec: maxReplicas: 20 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: hpa-tester metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 100

Apply the HPA and verify:

 1 2 3 4 5 6 7 8 9101112131415161718

[root@openshift-jumpserver-0 hpa-test]# oc apply -f hpa.yaml oc ghorizontalpodautoscaler.autoscaling/hpa-tester created[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester <unknown>/100% 1 20 0 2s[root@openshift-jumpserver-0 hpa-test]# oc describe hpa hpa-testerName: hpa-testerNamespace: testLabels: <none>Annotations: <none>CreationTimestamp: Mon, 01 Mar 2021 10:56:02 +0000Reference: Deployment/hpa-testerMetrics: ( current / target ) resource memory on pods (as a percentage of request): <unknown> / 100%Min replicas: 1Max replicas: 20Deployment pods: 0 current / 0 desiredEvents: <none>

And wait for 5 minutes. With 0 overhead inside the pods, the result be 4 pods - 4096 MB (cumulative Memory) / 1024 MB (request container spec) = 400%, then 400% / 100% = 4). Given that the cumulative memory utilization will be a bit higher than 4096 MB, HPA will scale out to 5 pods:

 1 2 3 4 5 6 7 8 91011121314151617

[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester 83%/100% 1 20 5 4m19s[root@openshift-jumpserver-0 hpa-test]# oc get podsNAME READY STATUS RESTARTS AGEhpa-tester-7bff6856bd-4dxgb 1/1 Running 0 4m6shpa-tester-7bff6856bd-7r4r8 1/1 Running 0 3m51shpa-tester-7bff6856bd-h45s7 1/1 Running 0 5m23shpa-tester-7bff6856bd-nxp29 1/1 Running 0 4m6shpa-tester-7bff6856bd-trqv9 1/1 Running 0 4m6s[root@openshift-jumpserver-0 hpa-test]# oc get podmetricsNAME CPU MEMORY WINDOWhpa-tester-7bff6856bd-4dxgb 338m 878064Ki 5m0shpa-tester-7bff6856bd-7r4r8 347m 886932Ki 5m0shpa-tester-7bff6856bd-h45s7 338m 854892Ki 5m0shpa-tester-7bff6856bd-nxp29 337m 880600Ki 5m0shpa-tester-7bff6856bd-trqv9 342m 878492Ki 5m0s

Scaling based on absolute memory values

It is also possible to scale for a specific CPU value. First, scale back the deployment to 1 replica or delete and recreate it. Then, apply the following HorizontalPodAutoscaler:

 1 2 3 4 5 6 7 8 91011121314151617181920

apiVersion: autoscaling/v2beta2kind: HorizontalPodAutoscalermetadata: annotations: name: hpa-tester namespace: testspec: maxReplicas: 20 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: hpa-tester metrics: - type: Resource resource: name: memory target: type: AverageValue averageValue: "2048Mi"

Verify after deploying this HPA:

 1 2 3 4 5 6 7 8 910111213141516

[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester <unknown>/2Gi 1 20 0 3s[root@openshift-jumpserver-0 hpa-test]# oc describe hpa hpa-testerName: hpa-testerNamespace: testLabels: <none>Annotations: <none>CreationTimestamp: Mon, 01 Mar 2021 11:01:49 +0000Reference: Deployment/hpa-testerMetrics: ( current / target ) resource memory on pods: <unknown> / 2GiMin replicas: 1Max replicas: 20Deployment pods: 0 current / 0 desiredEvents: <none>

And wait for 5 minutes. The result will be 3 pods ceil[(4096+overhead MB / 2048 MB)]:

 1 2 3 4 5 6 7 8 910111213

[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester 1476487850666m/2Gi 1 20 3 16m[root@openshift-jumpserver-0 hpa-test]# oc get podsNAME READY STATUS RESTARTS AGEhpa-tester-7bff6856bd-6mvbv 1/1 Running 0 16mhpa-tester-7bff6856bd-mpjvb 1/1 Running 0 17mhpa-tester-7bff6856bd-t5xc4 1/1 Running 0 16m[root@openshift-jumpserver-0 hpa-test]# oc get podmetricsNAME CPU MEMORY WINDOWhpa-tester-7bff6856bd-6mvbv 552m 1446040Ki 5m0shpa-tester-7bff6856bd-mpjvb 542m 1425156Ki 5m0shpa-tester-7bff6856bd-t5xc4 557m 1449152Ki 5m0s

Combining CPU and memory metrics

It is also possible to combine CPU and memory targets. For example:

 1 2 3 4 5 6 7 8 91011121314151617181920212223242526

apiVersion: autoscaling/v2beta2kind: HorizontalPodAutoscalermetadata: annotations: name: hpa-tester namespace: testspec: maxReplicas: 20 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: hpa-tester metrics:  - type: Resource resource: name: memory target: type: AverageValue averageValue: "2048Mi" - type: Resource resource: name: cpu target: type: AverageValue averageValue: "200m"

Verification:

123	`[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester <unknown>/2Gi, <unknown>/200m 1 20 0 7s`

Wait for a short while and the memory metrics will already lead to a scale-out to 3 pods:

 1 2 3 4 5 6 7 8 91011121314151617181920212223242526272829303132

[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester 4344119296/2Gi, <unknown>/200m 1 20 3 48s[root@openshift-jumpserver-0 hpa-test]# oc describe hpaName: hpa-testerNamespace: testLabels: <none>Annotations: <none>CreationTimestamp: Mon, 01 Mar 2021 11:41:03 +0000Reference: Deployment/hpa-testerMetrics: ( current / target ) resource memory on pods: 4343648256 / 2Gi resource cpu on pods: <unknown> / 200mMin replicas: 1Max replicas: 20Deployment pods: 1 current / 3 desiredConditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource ScalingLimited False DesiredWithinRange the desired count is within the acceptable rangeEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedGetResourceMetric 11s horizontal-pod-autoscaler did not receive metrics for any ready pods Normal SuccessfulRescale 10s horizontal-pod-autoscaler New size: 3; reason: memory resource above target[root@openshift-jumpserver-0 hpa-test]# oc get podsNAME READY STATUS RESTARTS AGEhpa-tester-7bff6856bd-4mlh6 1/1 Running 0 57shpa-tester-7bff6856bd-c7862 1/1 Running 0 2m8shpa-tester-7bff6856bd-vm27z 1/1 Running 0 57s

It will take a bit longer for the environment to calculate the CPU averages but eventually a second scale-out step will follow to 9 pods:

 1 2 3 4 5 6 7 8 910111213141516171819202122232425

[root@openshift-jumpserver-0 hpa-test]# oc get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-tester Deployment/hpa-tester 500336867555m/2Gi, 190m/200m 1 20 9 6m37s[root@openshift-jumpserver-0 hpa-test]# oc get podsNAME READY STATUS RESTARTS AGEhpa-tester-7bff6856bd-4mlh6 1/1 Running 0 6m28shpa-tester-7bff6856bd-6c968 1/1 Running 0 86shpa-tester-7bff6856bd-c7862 1/1 Running 0 7m39shpa-tester-7bff6856bd-cj8dz 1/1 Running 0 86shpa-tester-7bff6856bd-ngp4l 1/1 Running 0 102shpa-tester-7bff6856bd-nwx7r 1/1 Running 0 101shpa-tester-7bff6856bd-tg6vb 1/1 Running 0 86shpa-tester-7bff6856bd-vm27z 1/1 Running 0 6m28shpa-tester-7bff6856bd-xfptr 1/1 Running 0 101s[root@openshift-jumpserver-0 hpa-test]# oc get podmetricsNAME CPU MEMORY WINDOWhpa-tester-7bff6856bd-4mlh6 199m 508864Ki 5m0shpa-tester-7bff6856bd-6c968 185m 478548Ki 5m0shpa-tester-7bff6856bd-c7862 181m 484504Ki 5m0shpa-tester-7bff6856bd-cj8dz 193m 476976Ki 5m0shpa-tester-7bff6856bd-ngp4l 194m 477744Ki 5m0shpa-tester-7bff6856bd-nwx7r 195m 507352Ki 5m0shpa-tester-7bff6856bd-tg6vb 191m 478620Ki 5m0shpa-tester-7bff6856bd-vm27z 181m 484528Ki 5m0shpa-tester-7bff6856bd-xfptr 209m 505692Ki 5m0s

Custom application metrics for auto-scaling in OpenShift

https://docs.openshift.com/container-platform/4.6/monitoring/exposing-custom-application-metrics-for-autoscaling.html