Kubernetes Best Practices for Production

Running Kubernetes in production requires careful planning and adherence to best practices. This guide covers essential strategies for deploying and managing production-grade Kubernetes clusters.

Resource Management

1. Always Set Resource Requests and Limits

yamlCopy
resources:
  requests:
    memory: "256Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"
    cpu: "1000m"

Benefits:

• Prevents resource starvation
• Enables effective scheduling
• Improves cluster stability

2. Use Horizontal Pod Autoscaling

Automatically scale based on metrics:

yamlCopy
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

High Availability

1. Multiple Replicas

Never run single instances in production:

yamlCopy
spec:
  replicas: 3

2. Pod Disruption Budgets

Protect against disruptions:

yamlCopy
apiVersion: policy/v1
kind:PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

3. Anti-Affinity Rules

Distribute pods across nodes:

yamlCopy
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - my-app
        topologyKey: kubernetes.io/hostname

Security

1. Use Network Policies

Control traffic between pods:

yamlCopy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend

2. RBAC Configuration

Implement least privilege access:

yamlCopy
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

3. Security Contexts

Run containers with minimal privileges:

yamlCopy
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true

4. Pod Security Standards

Enforce security policies:

yamlCopy
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted

Configuration Management

1. Use ConfigMaps and Secrets

Never hardcode configuration:

yamlCopy
# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  app.properties: |
    log_level=info
    timeout=30

# Secret
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  api-key: <base64-encoded-value>

2. External Secret Management

Consider tools like:

• HashiCorp Vault
• AWS Secrets Manager
• Azure Key Vault
• Google Secret Manager

Health Checks

1. Liveness Probes

Restart unhealthy containers:

yamlCopy
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

2. Readiness Probes

Control traffic to pods:

yamlCopy
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

3. Startup Probes

Handle slow-starting applications:

yamlCopy
startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Monitoring and Logging

1. Prometheus for Metrics

Deploy Prometheus stack:

• Prometheus for metrics collection
• Grafana for visualization
• Alertmanager for notifications

2. Centralized Logging

Use ELK or EFK stack:

• Elasticsearch for storage
• Fluentd/Logstash for collection
• Kibana for visualization

3. Distributed Tracing

Implement with:

• Jaeger
• Zipkin
• OpenTelemetry

Deployment Strategies

1. Rolling Updates

Default strategy with configuration:

yamlCopy
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

2. Blue-Green Deployments

Use services to switch traffic:

yamlCopy
# Blue (current)
selector:
  app: my-app
  version: v1

# Switch to Green
selector:
  app: my-app
  version: v2

3. Canary Deployments

Gradual rollout with traffic splitting

Backup and Disaster Recovery

1. Backup Strategy

Regular backups of:

• etcd cluster
• Persistent volumes
• Configuration files

2. Tools to Consider

• Velero for cluster backup
• Kasten K10 for data management
• Cloud-native backup solutions

Cost Optimization

1. Right-Size Resources

Monitor and adjust:

• Review actual usage
• Adjust requests/limits
• Remove unused resources

2. Use Cluster Autoscaler

Automatically adjust cluster size:

yamlCopy
apiVersion: autoscaling/v1
kind: ClusterAutoscaler
spec:
  minNodes: 3
  maxNodes: 10

3. Spot/Preemptible Instances

Use for non-critical workloads:

• Significant cost savings
• Implement proper handling of interruptions

Namespace Organization

1. Environment Separation

production
staging
development

2. Team-Based Namespaces

team-api
team-frontend
team-data

3. Resource Quotas

Limit resources per namespace:

yamlCopy
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi

Update and Maintenance

1. Regular Updates

• Keep Kubernetes updated
• Update dependencies
• Patch security vulnerabilities

2. Test Before Production

• Staging environment
• Integration tests
• Load testing

3. Maintenance Windows

Plan for:

• Cluster upgrades
• Node updates
• Major changes

Common Pitfalls to Avoid

1. ❌ No resource limits
2. ❌ Running as root
3. ❌ No health checks
4. ❌ Single replica deployments
5. ❌ Hardcoded configurations
6. ❌ No monitoring
7. ❌ Neglecting security
8. ❌ No backup strategy

Conclusion

Running Kubernetes in production requires attention to detail and adherence to best practices. Focus on:

• Reliability: High availability and fault tolerance
• Security: Defense in depth
• Observability: Comprehensive monitoring and logging
• Efficiency: Resource optimization
• Maintainability: Clean, organized clusters

Start with these fundamentals, iterate based on your needs, and continuously improve your Kubernetes operations. The investment in proper setup and management pays dividends in stability, security, and operational efficiency.

---

Related Resources:

• Read about Microservices Architecture
• Learn more in our Platform Engineering Revolution
• Explore our Cloud Solutions