K8S 生产环境实战：滚动更新零停机、一键回滚与 CrashLoopBackOff 排查手册

K8S 集群（3 Master + 5 Worker）
├── Namespace: production
│   ├── Deployment: gateway (2 replicas)
│   ├── Deployment: device-service (3 replicas)
│   ├── Deployment: collect-service (3 replicas)
│   ├── Deployment: alarm-service (2 replicas)
│   ├── Deployment: watchtower-service (2 replicas)
│   └── ... 其他服务
├── Namespace: staging
│   └── ... 预发环境（每个服务 1 replica）
└── Namespace: monitoring
    ├── Prometheus
    ├── Alertmanager
    └── Grafana

apiVersion: apps/v1
kind: Deployment
metadata:
  name: device-service
  namespace: production
spec:
  replicas: 3
  revisionHistoryLimit: 5  # 保留5个历史版本，支持回滚
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # 最多同时多出1个Pod
      maxUnavailable: 0  # 更新过程中不允许有Pod不可用
  selector:
    matchLabels:
      app: device-service
  template:
    metadata:
      labels:
        app: device-service
        version: v2.3.1
    spec:
      containers:
        - name: device-service
          image: registry.company.com/kcloud/device-service:v2.3.1
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1024Mi"
​
          # 就绪探针：决定Pod何时接收流量
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30   # Spring Boot启动需要时间
            periodSeconds: 5
            failureThreshold: 3
​
          # 存活探针：决定Pod是否需要重启
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            failureThreshold: 3
​
          # 启动探针：保护慢启动的应用
          startupProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 30  # 最长等待 10+30*5=160秒启动

初始状态: [Pod-v1-a ✓] [Pod-v1-b ✓] [Pod-v1-c ✓]  (3 pods, all v1)
​
Step 1: 创建新 Pod
[Pod-v1-a ✓] [Pod-v1-b ✓] [Pod-v1-c ✓] [Pod-v2-a 启动中...]
​
Step 2: Pod-v2-a 通过就绪探针 → 开始接收流量 → 终止 Pod-v1-a
[Pod-v1-a 终止中] [Pod-v1-b ✓] [Pod-v1-c ✓] [Pod-v2-a ✓]
​
Step 3: Pod-v1-a 终止完毕，创建 Pod-v2-b
[Pod-v1-b ✓] [Pod-v1-c ✓] [Pod-v2-a ✓] [Pod-v2-b 启动中...]
​
... 重复 ...
​
最终状态: [Pod-v2-a ✓] [Pod-v2-b ✓] [Pod-v2-c ✓]  (3 pods, all v2)

# 1. 查看更新历史
$ kubectl rollout history deployment/device-service -n production
REVISION  CHANGE-CAUSE
3         image update to v2.1.0
4         image update to v2.2.0
5         image update to v2.3.0  ← 当前版本（有bug）
​
# 2. 一键回滚到上一个版本
$ kubectl rollout undo deployment/device-service -n production
deployment.apps/device-service rolled back
​
# 3. 确认回滚状态
$ kubectl rollout status deployment/device-service -n production
Waiting for deployment "device-service" rollout to finish: 1 old replicas are pending termination...
deployment "device-service" successfully rolled out
​
# 4. 验证版本
$ kubectl get pods -n production -l app=device-service -o jsonpath='{.items[*].spec.containers[0].image}'
registry.company.com/kcloud/device-service:v2.2.0

# 回滚到 REVISION 3（v2.1.0）
kubectl rollout undo deployment/device-service -n production --to-revision=3

stage('Deploy to Production') {
    steps {
        sh "kubectl set image deployment/device-service device-service=${IMAGE}:${TAG} -n production"
        sh "kubectl rollout status deployment/device-service -n production --timeout=300s"
    }
}
​
stage('Smoke Test') {
    steps {
        script {
            def healthCheck = sh(
                script: "curl -sf http://device-service.production.svc/actuator/health",
                returnStatus: true
            )
            def apiCheck = sh(
                script: "curl -sf http://device-service.production.svc/api/v1/devices/health-check",
                returnStatus: true
            )
            if (healthCheck != 0 || apiCheck != 0) {
                echo "冒烟测试失败，自动回滚..."
                sh "kubectl rollout undo deployment/device-service -n production"
                sh "kubectl rollout status deployment/device-service -n production --timeout=300s"
                error("部署已自动回滚，请检查新版本")
            }
        }
    }
}

# 查看 Pod 事件
$ kubectl describe pod device-service-xyz -n production
Events:
  Type     Reason     Age   Message
  ----     ------     ----  -------
  Warning  OOMKilled  30s   Container device-service was OOMKilled
​
# 查看上一次容器的退出状态
$ kubectl get pod device-service-xyz -n production -o jsonpath='{.status.containerStatuses[0].lastState}'
{"terminated":{"exitCode":137,"reason":"OOMKilled"}}

# 查看容器日志（关键：-p 参数查看上一次崩溃的容器日志）
$ kubectl logs device-service-xyz -n production -p
...
Caused by: java.lang.IllegalArgumentException:
  Could not resolve placeholder 'spring.datasource.password' in value "${spring.datasource.password}"

$ kubectl describe pod alarm-service-xyz -n production
Events:
  Warning  Unhealthy  45s (x3 over 65s)  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 503
  Normal   Killing    45s                 kubelet  Container alarm-service failed liveness probe, will be restarted

Pod 状态: CrashLoopBackOff
        │
        ▼
kubectl describe pod → 查看 Events 和 Last State
        │
        ├── OOMKilled (exitCode=137)
        │   → 调整 memory limits 或优化应用内存使用
        │
        ├── Liveness probe failed
        │   → 检查探针配置（initialDelay是否足够、端口是否正确）
        │   → 检查应用启动时间，增加 startupProbe
        │
        ├── 其他退出码
        │   │
        │   ▼
        │   kubectl logs <pod> -p → 查看崩溃前的日志
        │   │
        │   ├── 配置错误（Missing property / DB connection failed）
        │   │   → 检查 ConfigMap / Secret / 外部依赖可达性
        │   │
        │   ├── 端口冲突（Address already in use）
        │   │   → 检查 containerPort 配置和其他容器
        │   │
        │   └── 应用 Bug（NPE / 类加载错误）
        │       → 修复代码，回滚到上一版本
        │
        └── 没有明显错误
            → kubectl exec -it <pod> -- /bin/sh 进入容器排查
            → 检查文件系统权限、挂载卷是否正常