Kubernetes pod scaling by custom metrics with Keda

In this article I will try to explain usage of Keda to achieve pod autoscaling with custom AWS SQS metrics. To not spend money I have created my local Kubernetes cluster and deployed Keda. I have created AWS Secret and Access keys to use inside of my application(kubernetes deployment) and Keda trigger authentication. Keda will scale up pods when in AWS SQS queue count will be more than 5. Keda itself creating HPA object with minimal pod 1 and maximal to 100. AWS CLI already installed and configured in the Linux Desktop.

Install Keda to On-Prem cluster

$ mkdir keda && cd keda
$ kubectl create ns keda
$ helm repo add kedacore https://kedacore.github.io/charts && helm repo update
$ helm install keda kedacore/keda --namespace keda # --set logLevel=debug

After deployment check Keda metrics server exposed metrics

$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": []
}

Create global variables to execute our script

export NS_NAME=scale-by-aws-sqs
export REGION=us-east-1
export QUEUE_NAME=keda-test
export QueueURL=$(aws sqs create-queue --queue-name=${QUEUE_NAME} --region=$REGION --output=text --query=QueueUrl)
export AWS_ACCESS_KEY_ID=$(cat ~/.aws/credentials | grep access_key_id  | awk '{ print $(NF)}')
export AWS_SECRET_ACCESS_KEY=$(cat ~/.aws/credentials | grep secret_access_key  | awk '{ print $(NF)}')
export SQS_IMAGE='jamalshahverdiev/keda-aws-sqs-consumer:latest'
export KEDA_SCALEDOBJECT_TEMPLATE_FILE=keda-scaledobject-template.yaml
export DEPLOYMENT_TEMPLATE_FILE=consumer-deployment-template.yaml
export KEDA_TRIGGER_TEMPLATE=keda-trigger-template.yaml
export KEDA_TRIGGER_NAME=keda-trigger-auth-aws-credentials
export KEDA_DEPLOYMENT_NAME=keda-scale-by-aws-sqs
export CONATNER_NAME=sqs-consumer
export SECRET_NAME=${NS_NAME}-secrets

Create KEDA Trigger,Deployment and Scaledobject which will be used to scale POD by collected metrics from AWS SQS

$ cat <<'EOF' > ${KEDA_TRIGGER_TEMPLATE}
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: ${KEDA_TRIGGER_NAME}
  namespace: ${NS_NAME}
spec:
  secretTargetRef:
  - parameter: awsAccessKeyID     
    name: ${SECRET_NAME}
    key: AWS_ACCESS_KEY_ID
  - parameter: awsSecretAccessKey
    name: ${SECRET_NAME}
    key: AWS_SECRET_ACCESS_KEY
EOF
$ cat <<'EOF' > ${KEDA_SCALEDOBJECT_TEMPLATE_FILE}
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: app-${NS_NAME}
  namespace: ${NS_NAME}
spec:
  scaleTargetRef:
    name: ${KEDA_DEPLOYMENT_NAME}
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: ${KEDA_TRIGGER_NAME}
    metadata:
      queueURL: ${QueueURL}
      awsRegion: "${REGION}"
      queueLength: "5"
EOF
$ cat <<'EOF' > ${DEPLOYMENT_TEMPLATE_FILE}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${KEDA_DEPLOYMENT_NAME}
  namespace: ${NS_NAME}
spec:
  selector:
    matchLabels:
      service: ${KEDA_DEPLOYMENT_NAME}
  replicas: 1
  template:
    metadata:
      labels:
        service: ${KEDA_DEPLOYMENT_NAME}
    spec:
      containers:
      - image: ${SQS_IMAGE}
        name: ${CONATNER_NAME}
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: ${SECRET_NAME}
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: ${SECRET_NAME}
              key: AWS_SECRET_ACCESS_KEY
        - name: AWS_DEFAULT_REGION
          value: ${REGION}
EOF

Then execute the following script to create

$ cat <<'EOF' > create_queue_with_object.sh
#!/usr/bin/env bash
kubectl create namespace ${NS_NAME}
kubectl create secret generic ${SECRET_NAME} --namespace=${NS_NAME} --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
for file in ${KEDA_SCALEDOBJECT_TEMPLATE_FILE} ${DEPLOYMENT_TEMPLATE_FILE} ${KEDA_TRIGGER_TEMPLATE}; do
    cat ${file} | envsubst > ${file}_rendered && kubectl apply -f ${file}_rendered
done 
EOF
$ chmod +x create_queue_with_object.sh && ./create_queue_with_object.sh

After deployment check all objects is created

$ kubectl get triggerauthentications.keda.sh -n scale-by-aws-sqs
NAME                                PODIDENTITY   SECRET                     ENV   VAULTADDRESS
keda-trigger-auth-aws-credentials                 scale-by-aws-sqs-secrets
$ kubectl get scaledobjects.keda.sh -n scale-by-aws-sqs
NAME                   SCALETARGETKIND      SCALETARGETNAME         MIN   MAX   TRIGGERS        AUTHENTICATION                      READY   ACTIVE   FALLBACK   AGE
app-scale-by-aws-sqs   apps/v1.Deployment   keda-scale-by-aws-sqs               aws-sqs-queue   keda-trigger-auth-aws-credentials   True    False    False      5m39s
$ kubectl get horizontalpodautoscalers.autoscaling -n scale-by-aws-sqs
NAME                            REFERENCE                          TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-app-scale-by-aws-sqs   Deployment/keda-scale-by-aws-sqs   0/5 (avg)   1         100       1          6m1s

To troubleshoot we can look at our app itself logs and Keda operator logs too

$ kubectl logs -f -n keda $(kubectl get pods -n keda | grep keda-operator | awk '{ print $1 }')

Before starting I have scaled to `0` my deployment app because, it is handled quickly than create of queue count in AWS SQS and then, I have executed the following command to increase the size of queue. Trigger will be happened and increase the size of POD when `5` queue will be waiting on AWS SQS queue:

$ kubectl scale deployment --replicas 0 -n $NS_NAME $KEDA_DEPLOYMENT_NAME
$ for x in {1..200}; do
    result=$(aws sqs send-message --message-body="Queue message number ${x}" --queue-url=$QueueURL --region=$REGION)
    echo ${result}
done

After deployment open tmux session and look at the pod counts in real time and try to increase the size of sqs queue in another terminal

$ watch -n2 "kubectl get pods -n ${NS_NAME} | wc -l"
$ watch -n2 "kubectl get pods -n ${NS_NAME}"

We can see real simulation in the following video

I hope it will be useful.