Site Reliability Engineering (SRE)

Q1: What happens internally when a request hits a Load Balancer before reaching an application server?

Ans: The load balancer terminates the client connection, inspects the request, and applies routing rules. It may perform SSL/TLS termination, health checks, and session persistence. Based on algorithms like round-robin, least connections, or weighted routing, it forwards the request to a healthy backend server. It also rewrites headers, manages retries, and ensures high availability.

Q2: How would you troubleshoot a Kubernetes pod stuck in CrashLoopBackOff?

Ans: 1. Run `kubectl describe pod ` to check events. 2. Inspect logs with `kubectl logs --previous`. 3. Verify container image and entrypoint. 4. Check liveness/readiness probes. 5. Ensure resource limits are sufficient. 6. Debug application code or configuration causing repeated crashes.

Q3: A deployment was successful but users report 502/503 errors. What could be the possible reasons?

Ans: 502/503 errors indicate gateway or service unavailability. Possible causes: pods not ready due to failing readiness probes, misconfigured service selectors, backend crash, insufficient resources, or load balancer health checks failing. Network policies or firewall rules may also block traffic.

Q4: What steps would you take if a CI/CD pipeline suddenly starts failing after working for months?

Ans: 1. Review recent code/config changes. 2. Check pipeline logs for errors. 3. Validate credentials, tokens, and secrets. 4. Inspect dependency updates or expired certificates. 5. Roll back to last working version. 6. Re-run pipeline with debug mode enabled.

Q5: How does Horizontal Pod Autoscaler (HPA) decide when to scale pods?

Ans: HPA monitors metrics like CPU, memory, or custom metrics via Metrics Server. It compares current utilization against target thresholds. If utilization exceeds the target, HPA increases replicas; if below, it scales down. Scaling decisions are averaged over a stabilization window to avoid flapping.

Q6: Your CPU usage spikes to 95% in production after a new release. How will you investigate?

Ans: 1. Check application logs for infinite loops or heavy queries. 2. Use `top` or `kubectl top pods` to identify resource hogs. 3. Profile code with APM tools. 4. Roll back release if necessary. 5. Validate resource requests/limits in Kubernetes. 6. Monitor DB queries or external API calls causing bottlenecks.

Q7: How do you implement zero downtime deployments in Kubernetes?

Ans: Use rolling updates with `kubectl rollout`, configure readiness probes, and ensure new pods are healthy before terminating old ones. Apply strategies like blue-green or canary deployments. Use service mesh or ingress controllers for traffic shifting.

Q8: A Docker container exits immediately after starting. How do you debug it?

Ans: Run `docker logs ` to inspect errors. Check entrypoint/command in Dockerfile. Use `docker run -it /bin/sh` to start interactively. Validate dependencies and environment variables. Ensure the process runs in foreground.

Q9: What is the difference between Infrastructure as Code and Configuration Management?

Ans: Infrastructure as Code (IaC) provisions infrastructure resources (servers, networks, storage) using tools like Terraform or CloudFormation. Configuration Management manages software and system configurations on provisioned infrastructure using tools like Ansible, Puppet, or Chef.

Q10: If a database suddenly becomes slow, what metrics would you check first?

Ans: Check CPU, memory, and disk I/O usage. Inspect query execution time, slow query logs, connection pool saturation, lock contention, and cache hit ratio. Monitor replication lag if applicable.

Q11: Your Prometheus alerts are firing continuously even though the system looks healthy. What could be wrong?

Ans: Possible issues: misconfigured alert thresholds, incorrect PromQL queries, stale metrics, or clock skew between nodes. Validate scrape intervals and ensure metrics reflect actual system state.

Q12: A Kubernetes node goes into NotReady state. What troubleshooting steps would you take?

Ans: 1. Run `kubectl describe node `. 2. Check kubelet logs. 3. Verify network connectivity and DNS. 4. Inspect resource usage (CPU/memory/disk). 5. Restart kubelet or node if necessary. 6. Check cloud provider health if managed cluster.

Q13: What is the difference between Liveness Probe and Readiness Probe in Kubernetes?

Ans: Liveness Probe checks if the container is alive; if it fails, Kubernetes restarts the container. Readiness Probe checks if the container is ready to serve traffic; if it fails, the pod is removed from service endpoints but not restarted.

Q14: How would you secure secrets in a CI/CD pipeline?

Ans: Use secret managers like AWS Secrets Manager, HashiCorp Vault, or Kubernetes Secrets. Avoid hardcoding secrets in code. Encrypt environment variables, restrict access via RBAC, and rotate credentials regularly.

Q15: How do you investigate memory leaks in a containerized application?

Ans: Monitor memory usage with `kubectl top pods`. Use profiling tools (Heap dump, pprof). Check for unclosed connections or objects. Apply resource limits to prevent node exhaustion. Restart pods as a temporary fix while debugging root cause.

Q16: Your Terraform deployment failed halfway. How do you recover from the inconsistent state?

Ans: Run `terraform plan` to identify drift. Use `terraform refresh` to sync state. Manually fix resources if needed. Apply again after resolving conflicts. Use remote state backends to avoid corruption.

Q17: What are the key SLIs, SLOs, and SLAs you track for a production system?

Ans: SLIs: latency, error rate, throughput, availability. SLOs: 99.9% uptime, <200ms response time. SLAs: contractual guarantees to customers, e.g., 99.5% availability per month.

Q18: A microservice suddenly starts returning high latency. What tools would you use to debug it?

Ans: Use distributed tracing (Jaeger, Zipkin), APM tools (Datadog, New Relic), and Prometheus metrics. Inspect logs, DB queries, and network latency. Check resource usage and external dependencies.

Q19: How would you design a highly available system that can handle traffic spikes during peak sales?

Ans: Use load balancers, auto-scaling groups, and multi-region deployment. Apply caching (Redis, CDN), database sharding, and message queues. Implement failover strategies and chaos testing.

Q20: If multiple monitoring alerts fire at the same time, how do you prioritize incident response?

Ans: Prioritize based on customer impact, severity, and SLA breaches. Focus on alerts affecting availability and critical services first. Use incident management tools to correlate alerts and reduce noise.