Scaling I/O Bound Microservices based on RPS

Don't Scale High-I/O Microservices on CPU—Use Request-Based Scaling (KEDA) Instead

Imagine a modern banking microservice handling user payments. It calls multiple downstream services per request. Under heavy traffic, CPU usage may remain low while HTTP 504 timeouts spike dramatically.

CPU-Based Scaling in Kubernetes

The Hidden Trap of CPU-Based Autoscaling

In synchronous runtimes, each request occupies a worker thread. When waiting on external APIs, threads consume 0% CPU, misleading HPA into thinking pods are idle while they are actually starved.

The Solution: Scale on Workload Demand, Not CPU

• Requests Per Second (RPS): Incoming HTTP traffic rate.
• Active Executing Threads / Busy Connections: Number of requests actively processed.

Designing the Math for RPS-Based Scaling

Using Little's Law, throughput per pod = (Max Target Threads) / (Average Downstream API Latency). Example: 200 threads, 70% safe capacity (140 threads), 0.5s latency → 280 RPS per pod.

Implementation Strategies with KEDA & Prometheus

Strategy 1: Scale by Incoming Requests Per Second (RPS)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payment-service-scaler
  namespace: banking
spec:
  scaleTargetRef:
    name: payment-service-deployment
  minReplicaCount: 4
  maxReplicaCount: 50
  cooldownPeriod: 180
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc:9090
        query: |
          sum(rate(http_requests_total{app="payment-service"}[2m])) 
          / 
          count(kube_pod_info{pod=~"payment-service-.*"})
        threshold: '280'

Strategy 2: Scale by Active Busy Threads (The Golden Metric)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payment-service-thread-scaler
  namespace: banking
spec:
  scaleTargetRef:
    name: payment-service-deployment
  minReplicaCount: 4
  maxReplicaCount: 50
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc:9090
        query: avg(tomcat_threads_busy{app="payment-service"})
        threshold: '140'

Conclusion

Autoscaling is not one-size-fits-all. Compute-heavy apps scale on CPU. I/O-heavy apps scale on Requests or Active Connections to prevent thread starvation and cascading timeouts.