Kubernetes Operations & Platform Engineering

This skill covers day-to-day Kubernetes operations (troubleshooting, debugging, scaling) and platform engineering (manifests, Helm, RBAC, autoscaling, monitoring).

Scripts: Always run scripts with --help first. Do not read script source unless debugging the script itself.

References: Load reference files on demand based on the task at hand. Do not pre-load all references.

Slash commands: Users can also invoke these directly:

• /k8s-skills:k8s-debug [pod] [namespace] — Diagnose a pod or deployment
• /k8s-skills:k8s-deploy [action] [name] [namespace] — Deploy, rollback, or restart
• /k8s-skills:k8s-health [namespace] — Cluster or namespace health check

Quick Command Reference

Pod Troubleshooting

Follow the diagnostic path: get → describe → logs → exec

Pod not healthy?
│
├─ Status: CrashLoopBackOff
│  ├─ Check: kubectl logs <pod> --previous
│  ├─ Check: kubectl describe pod <pod> → Events section
│  ├─ Exit code 137 (OOMKilled)?
│  │  └─ Increase memory limits, check for memory leaks
│  ├─ Exit code 1 (app error)?
│  │  └─ Read logs, fix application startup
│  └─ Probe failure?
│     └─ Adjust initialDelaySeconds, check endpoint health
│
├─ Status: ImagePullBackOff
│  ├─ Check: kubectl describe pod <pod> → image name/tag
│  ├─ Registry auth? → Verify imagePullSecrets
│  ├─ Image exists? → Check registry directly
│  └─ Network? → Can node reach registry?
│
├─ Status: Pending
│  ├─ Check: kubectl describe pod <pod> → Events
│  ├─ Insufficient resources? → kubectl describe nodes, check allocatable
│  ├─ NodeSelector/affinity mismatch? → Verify node labels
│  ├─ Taint not tolerated? → Add tolerations or remove taint
│  └─ PVC not bound? → kubectl get pvc -n <ns>
│
├─ Status: Init:Error / Init:CrashLoopBackOff
│  ├─ Check: kubectl logs <pod> -c <init-container>
│  └─ Common: DB migration failed, config dependency not ready
│
├─ Status: Evicted
│  ├─ Check: kubectl describe node <node> → Conditions
│  ├─ DiskPressure? → Clean up node disk
│  └─ MemoryPressure? → Check resource quotas, set proper requests
│
└─ Status: Terminating (stuck)
   ├─ Check: kubectl describe pod <pod> → finalizers
   ├─ Finalizer stuck? → Patch to remove finalizer
   └─ Last resort: kubectl delete pod <pod> --grace-period=0 --force

For detailed debugging workflows with step-by-step resolution for every error state, read Troubleshooting Guide.

Deployment Workflow

Standard Apply → Monitor → Verify

# 1. Validate manifest before applying
kubectl diff -f manifest.yaml

# 2. Apply the change
kubectl apply -f manifest.yaml

# 3. Monitor rollout
kubectl rollout status deployment/<name> -n <ns> --timeout=300s

# 4. Verify pods are healthy
kubectl get pods -n <ns> -l app=<name>
kubectl logs -l app=<name> -n <ns> --tail=20

# 5. Verify service endpoints
kubectl get endpoints <service> -n <ns>

Rollback

# Check revision history
kubectl rollout history deployment/<name> -n <ns>

# Rollback to previous revision
kubectl rollout undo deployment/<name> -n <ns>

# Rollback to specific revision
kubectl rollout undo deployment/<name> -n <ns> --to-revision=<N>

# Verify rollback completed
kubectl rollout status deployment/<name> -n <ns>

Rolling Update Strategy

When writing Deployment manifests, configure the update strategy:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # Extra pods during rollout
      maxUnavailable: 25%   # Max pods unavailable during rollout

For complete manifest patterns including probes, anti-affinity, and security contexts, read Manifest Patterns.

Emergency Procedures

1. Stop Bleeding — Scale to Zero

kubectl scale deployment/<name> --replicas=0 -n <ns>

2. Rollback a Bad Deployment

kubectl rollout undo deployment/<name> -n <ns>
kubectl rollout status deployment/<name> -n <ns>

3. Drain a Node for Maintenance

# Prevent new scheduling
kubectl cordon <node>

# Evict pods (respects PDB)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# After maintenance, re-enable
kubectl uncordon <node>

4. Force-Delete a Stuck Pod

# Only as last resort — try graceful delete first
kubectl delete pod <pod> -n <ns> --grace-period=0 --force

5. Apply Emergency Resource Quota

# Prevent runaway resource consumption in a namespace
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: emergency-quota
  namespace: <ns>
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "20"
EOF

6. Kill a Runaway CronJob

# Suspend the CronJob to prevent new runs
kubectl patch cronjob/<name> -n <ns> -p '{"spec":{"suspend":true}}'

# Delete active jobs
kubectl delete job -l job-name=<cronjob-name> -n <ns>

Helm Quick Reference

Values Override Precedence

Lower overrides higher:

1. Chart values.yaml (defaults)
2. -f custom-values.yaml (file override)
3. --set key=value (CLI override)

Common Flags

• --atomic — Auto-rollback on failure (implies --wait)
• --wait — Wait until all resources are healthy
• --timeout 5m — Max wait time
• --dry-run — Preview without applying
• --create-namespace — Create namespace if missing
• -f values-prod.yaml — Environment-specific values

For Helm chart structure, Helmfile, values management, and testing, read Helm Guide.

Diagnostic Scripts

Pod Diagnostics

Run bash scripts/diagnose.sh --help for full usage.

Diagnoses a specific pod or all unhealthy pods in a namespace. Checks status, events, logs, resource usage, and outputs a structured report with suggested fixes.

# Diagnose a specific pod
bash scripts/diagnose.sh -n <namespace> <pod-name>

# Diagnose all non-running pods in a namespace
bash scripts/diagnose.sh -n <namespace> --all

Cluster Health

Run bash scripts/cluster-health.sh --help for full usage.

Produces a cluster health overview: node status, resource utilization, unhealthy pods, pending PVCs, and recent warning events.

# Health check for current context
bash scripts/cluster-health.sh

# Health check for specific context
bash scripts/cluster-health.sh --context <context-name>

RBAC Audit

Run bash scripts/rbac-audit.sh --help for full usage.

Audits RBAC permissions across the cluster: finds cluster-admin bindings, wildcard permissions, default service account usage, secrets access, and unused service accounts.

# Audit entire cluster
bash scripts/rbac-audit.sh

# Audit specific namespace
bash scripts/rbac-audit.sh --namespace production

Namespace Setup

Run bash scripts/namespace-setup.sh --help for full usage.

Generates production-ready namespace setup manifests: namespace with PSS labels, ResourceQuota, LimitRange, NetworkPolicy (default deny), dedicated ServiceAccount, and RBAC.

# Generate manifests to stdout
bash scripts/namespace-setup.sh my-namespace

# Write to files
bash scripts/namespace-setup.sh my-namespace --output ./manifests/

# Apply directly (with confirmation)
bash scripts/namespace-setup.sh my-namespace --apply

Reference Files

Load these references as needed based on the task:

• [Troubleshooting Guide](./references/troubleshooting.md) — Complete debugging workflows:
• CrashLoopBackOff, OOMKilled, ImagePullBackOff detailed resolution
• Node-level issues (DiskPressure, MemoryPressure, NotReady)
• Networking failures (DNS, Service, Ingress)
• Storage issues (PVC pending, mount errors)

• [Manifest Patterns](./references/manifests.md) — Production-ready YAML templates:
• Deployment, StatefulSet, DaemonSet with best practices
• Service, Ingress, NetworkPolicy
• ConfigMap, Secret, HPA, PDB
• Resource requests/limits and QoS classes

• [Security Reference](./references/security.md) — Hardening and access control:
• RBAC Role/ClusterRole/Binding patterns
• Pod Security Standards (Restricted, Baseline, Privileged)
• Network policies for namespace isolation
• Secrets management (sealed-secrets, external-secrets)

• [Monitoring Guide](./references/monitoring.md) — Observability setup:
• Prometheus alerting rules for common failure modes
• ServiceMonitor / PodMonitor patterns
• Grafana dashboard guidelines
• Key metrics (USE method, RED method)

• [Helm Guide](./references/helm.md) — Helm operations and chart authoring:
• Chart directory structure and templating
• Values file management and overrides
• Helmfile for multi-chart deployments
• Chart testing and linting

• [GitOps Reference](./references/gitops.md) — ArgoCD and Flux:
• ArgoCD Application CRD, sync policies, app-of-apps, ApplicationSet
• Flux GitRepository, Kustomization, HelmRelease, image automation
• Environment promotion, secrets in GitOps (SOPS, sealed-secrets)
• ArgoCD vs Flux comparison and rollback patterns

• [Examples](./examples/) — Ready-to-use templates:
• Complete Helm chart for a production web application
• Helmfile config for multi-environment deployments
• Environment-specific values (dev, staging, production)

Quick Task Reference