Kubernetes Scenario Based

Q1: How do you recover a Pod stuck in permanent failure if InitContainer Failure with restartPolicy: Never is observed?

Ans: When an InitContainer fails under `restartPolicy: Never`, the Pod will remain in a failed state and Kubernetes will not retry. Troubleshooting involves checking logs (`kubectl logs -c `), verifying commands, and ensuring dependencies are available before the main container starts. Recovery options include deleting and recreating the Pod, adjusting the InitContainer logic, or changing the restartPolicy to `OnFailure` so Kubernetes retries automatically. In production, use CI/CD validation to catch InitContainer misconfigurations early.

Q2: What happens when you delete a specific Pod in a StatefulSet?

Ans: Deleting `pod-1` in a StatefulSet causes Kubernetes to recreate it with the same identity (same name, ordinal index, and PersistentVolumeClaim). Other Pods remain unaffected. This ensures stable network identities and persistent storage. Troubleshooting involves checking PVC reattachment and ensuring the volume is not corrupted. If the Pod fails to come back, inspect StatefulSet events (`kubectl describe sts `) and confirm that the storage backend supports reattachment.

Q3: Why do DaemonSets tolerate NoSchedule taints automatically?

Ans: DaemonSets are designed to run system-critical Pods (like monitoring agents) across all nodes, including masters. They automatically tolerate `NoSchedule` taints on master nodes. If a DaemonSet Pod is missing, check tolerations in the manifest and inspect node taints with `kubectl describe node `. For custom taints, explicitly add tolerations in the DaemonSet spec. This ensures observability and security agents run consistently across the cluster.

Q4: How does Kubernetes handle overlapping rollouts?

Ans: When a Deployment is updated with a new image, Kubernetes triggers a new rollout and cancels any previous rollout in progress. If the rollout hangs, check status with `kubectl rollout status deployment `. Common causes include failing readiness probes or insufficient resources. To fix, configure readiness probes correctly, adjust `maxUnavailable` and `maxSurge` in the Deployment strategy, or use `kubectl rollout undo` to revert. In production, combine readiness probes with PodDisruptionBudgets to avoid downtime.

Q5: What happens if multiple containers try to bind the same port?

Ans: Containers in the same Pod share the network namespace, so they cannot bind to the same port. If attempted, one container will fail to start. Troubleshoot by checking container specs in the Pod manifest. Fix by assigning unique ports per container and using a Service or sidecar proxy to route traffic appropriately. For example, a sidecar can expose metrics on a different port while the main app uses its own.

Q6: What happens when the Metrics Server is unavailable?

Ans: If the Metrics Server is down, the Horizontal Pod Autoscaler (HPA) cannot fetch metrics and freezes at the current replica count. Troubleshoot by checking APIService health (`kubectl get apiservices`), logs (`kubectl logs -n kube-system metrics-server`), and RBAC permissions. Fix by restarting the Metrics Server, ensuring API aggregation is healthy, and monitoring cluster resource usage. In production, consider fallback scaling policies or external metrics providers.

Q7: How do Pods behave when their ServiceAccount is deleted?

Ans: When a ServiceAccount is deleted, its tokens become invalid but remain mounted in running Pods until restart. This can cause authentication failures for API calls. Troubleshoot by checking Pod logs for 401/403 errors. Fix by recreating the ServiceAccount, restarting affected Pods, or using projected ServiceAccount tokens which auto-refresh. In production, enforce RBAC policies to prevent accidental deletion of critical ServiceAccounts.

Q8: How can strict rules cause scheduling deadlocks?

Ans: Strict anti-affinity rules (`requiredDuringSchedulingIgnoredDuringExecution`) can prevent Pods from being scheduled if no nodes satisfy the constraints. Troubleshoot with `kubectl describe pod` to see scheduling events. Fix by relaxing rules using `preferredDuringSchedulingIgnoredDuringExecution`, adding more nodes, or adjusting topology spread constraints. In production, balance anti-affinity with availability requirements to avoid deadlocks.

Q9: Why does a Kubernetes PersistentVolume with ReadWriteOnce access mode allow multiple Pods to mount it only if they are scheduled on the same node?

Ans: PersistentVolumes with `ReadWriteOnce` access mode allow multiple Pods to mount them only if they run on the same node. This is due to storage backend limitations. Troubleshoot by checking PVC events (`kubectl describe pvc`). Fix by using `ReadWriteMany` volumes (like NFS, CephFS, or GlusterFS) if multi-node access is required, or co-locating Pods with node affinity. In production, choose the correct access mode based on workload requirements.

Q10: What happens if only ingress rules are defined?

Ans: If a NetworkPolicy defines only ingress rules, egress traffic remains unrestricted. This means Pods can still initiate outbound connections. Troubleshoot by reviewing policies with `kubectl get networkpolicy` and testing connectivity. Fix by explicitly defining egress rules to restrict outbound traffic. In production, combine ingress and egress policies to enforce zero-trust networking.

Q11: How do you debug CrashLoopBackOff when logs are empty?

Ans: If a Pod is in CrashLoopBackOff but logs are empty, check events with `kubectl describe pod`. The container may fail before writing logs (e.g., missing binary, bad entrypoint). Inspect the image locally, verify entrypoint commands, and check resource requests. In production, use `kubectl get events` and enable debug sidecars to capture early failures.

Q12: What are common causes if Cluster Autoscaler is not adding nodes?

Ans: If the Cluster Autoscaler does not add nodes, verify node group limits in the cloud provider. Check resource requests — if Pods request more than available per node, scaling may fail. Troubleshoot with `kubectl describe pod` and Autoscaler logs. Fix by adjusting node group limits, reducing resource requests, or enabling larger instance types. In production, monitor Autoscaler metrics to ensure responsiveness.

Q13: During rolling updates, why does downtime occur in some cases. What could be the reasons?

Ans: Downtime during rolling updates often occurs due to missing readiness probes or lack of PodDisruptionBudgets (PDB). Without readiness probes, Kubernetes may route traffic to Pods that are not ready. Without PDBs, too many Pods may be terminated simultaneously. Fix by configuring readiness probes (`httpGet`, `tcpSocket`) and defining PDBs to control disruption. In production, always combine these for zero-downtime deployments.

Q14: How do you diagnose and fix etcd performance issues?

Ans: etcd performance issues often stem from disk latency or slow fsync operations. Monitor metrics (`etcd_disk_wal_fsync_duration_seconds`) and check disk I/O. Troubleshoot by moving etcd to SSD-backed storage, increasing IOPS, or tuning OS parameters. In production, run etcd on dedicated nodes with fast disks and monitor with Prometheus/Grafana.

Q15: How can you enforce image policies or Restrict External Images in Kubernetes?

Ans: To prevent Pods from pulling external or unverified images, use Admission Controllers or OPA Gatekeeper. Configure policies to allow only trusted registries. Troubleshoot by checking audit logs and validating webhook configurations. In production, enforce signed images with tools like Cosign and integrate with CI/CD pipelines to block unapproved images.

Q16: Kubernetes Api Gateway is showing a 504 timeout via ingress. How to troubleshoot that?

Ans: The troubleshooting steps are as follow: 1. Isolate the Backend: A 504 error means the Ingress controller/API Gateway timed out waiting for a response from the backend service. Check if the backend application pods are healthy and responsive.
2. Check Pod Logs: Review the logs of the application pods to see if they are processing requests too slowly or crashing/freezing mid-request.
3. Adjust Annotations: If the application legitimately takes a long time to respond (e.g., heavy processing), increase the proxy timeout limits in your Ingress resource using annotations like nginx.ingress.kubernetes.io/proxy-read-timeout: "600".

Q17: Application work internally cannot be accessed via ingress. How can we troubleshoot it?

Ans: The troubleshooting steps are as follow: 1. Test Internal Connectivity: Spin up a temporary curl pod inside the cluster and test if you can reach the application using its internal Service DNS (e.g., curl http://service-name.namespace.svc.cluster.local:port).
2. Verify Selector Labels: Ensure the Ingress points to the correct Service name and port, and verify that the Service’s spec.selector matches the actual labels on your application Pods.
3. Check Ingress Class and Rules: Verify that the Ingress controller is actively picking up your Ingress resource (check for an assigned IP/Address in kubectl get ingress) and that the host/path rules match your request precisely.

Q18: In production pods are restarting randomly. How to troubleshoot this scenario

Ans: The troubleshooting steps are as follow: 1. Check Termination Reason: Inspect the exit code and termination reason using kubectl describe pod <pod-name>. Look for 'OOMKilled' (Exit Code 137), which indicates the pod exceeded its memory limit.
2. Review Application Logs: Check the previous instance's logs before it crashed using kubectl logs <pod-name> --previous to identify application-level unhandled exceptions or fatal errors.
3. Inspect Probes: Check if misconfigured Liveness or Readiness probes are failing repeatedly, forcing Kubernetes to kill and restart otherwise healthy pods.

Q19: How do you monitor pod CPU usage, particularly Prometheus, not in the Grafana dashboard?

Ans: The methods are as follow: 1. Prometheus Expression Browser: Access the Prometheus UI directly and run a PromQL query like sum(rate(container_cpu_usage_seconds_total{pod="<pod-name>", container!=""}[5m])) by (pod) to see real-time CPU usage.
2. Using kubectl top: If the Kubernetes Metrics Server is installed, you can quickly check CPU consumption from your terminal without Prometheus by running kubectl top pod <pod-name>.

Q20: How would you handle if a Kubernetes cluster is deleted accidentally?

Ans: If a cluster is deleted accidentally, the first step is to confirm the source of deletion using cloud provider audit logs. Next, stop any automation that may reapply destructive changes. Recovery involves re-provisioning the cluster using Infrastructure-as-Code tools like Terraform or CloudFormation, restoring etcd snapshots and persistent volume backups, and redeploying workloads via manifests or Helm charts. Preventive measures include enabling deletion protection, scheduling automated backups, enforcing RBAC to limit destructive actions, and maintaining disaster recovery runbooks.