二进制多 Master K8s 集群（v1.18）部署完成后的验证清单

多 Master 二进制部署走到最后一步，集群到底起没起来、网络通不通，需要一次完整验证。下面是按顺序过的检查项，环境为 3 master + 3 worker，版本 v1.18.6。国内拉镜像慢的问题可以自己搭镜像代理仓库解决，不在本篇展开。

节点状态与测试负载

先看节点，全部 Ready 且版本一致才算正常：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
k8s-master1   Ready    <none>   3h44m   v1.18.6
k8s-master2   Ready    <none>   3h43m   v1.18.6
k8s-master3   Ready    <none>   3h43m   v1.18.6
k8s-node1     Ready    <none>   3h44m   v1.18.6
k8s-node2     Ready    <none>   3h43m   v1.18.6
k8s-node3     Ready    <none>   3h43m   v1.18.6

创建一个 NodePort 类型的 Service 加 DaemonSet 测试负载，保证每个节点都有 Pod：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cat > nginx-test.yml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.16.1
        ports:
        - containerPort: 80
EOF

[root@k8s-master1 work]# kubectl apply -f nginx-test.yml
service/nginx-ds created
daemonset.apps/nginx-ds created

[root@k8s-master1 work]# kubectl get pods -o wide|grep nginx
nginx-66qf9                  1/1     Running            0          7m47s   10.68.107.193   k8s-node3   <none>           <none>
nginx-lrm4n                  1/1     Running            0          7m47s   10.68.36.67     k8s-node1   <none>           <none>
nginx-r7pln                  1/1     Running            0          7m47s   10.68.169.129   k8s-node2   <none>           <none>

Pod IP 连通性

拿到三个 Pod IP：

[root@k8s-master1 work]# kubectl get pods  -o wide -l app=nginx

NAME          READY   STATUS    RESTARTS   AGE     IP              NODE        NOMINATED NODE   READINESS GATES
nginx-66qf9   1/1     Running   0          8m24s   10.68.107.193   k8s-node3   <none>           <none>
nginx-lrm4n   1/1     Running   0          8m24s   10.68.36.67     k8s-node1   <none>           <none>
nginx-r7pln   1/1     Running   0          8m24s   10.68.169.129   k8s-node2   <none>           <none>

在所有 worker 上分别 ping 这三个 Pod IP，验证容器网络（flannel）跨节点是否互通：

[root@k8s-master1 work]# for node_ip in ${WORK_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "ping -c 1 10.68.107.193"
    ssh ${node_ip} "ping -c 1 10.68.36.67"
    ssh ${node_ip} "ping -c 1 10.68.169.129"
  done

Service IP 与端口

查服务的 Cluster IP 和 NodePort 端口：

[root@k8s-master1 work]# kubectl get svc -l app=nginx
NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx-ds   NodePort   10.254.212.37   <none>        80:32120/TCP   6m7s

输出里可见：Service Cluster IP 10.254.212.37，服务端口 80，NodePort 端口 32120。在所有节点上 curl Service IP，正常的话每个节点都返回 nginx 欢迎页：

[root@k8s-master1 work]# for node_ip in ${WORK_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "curl -s 10.254.212.37"
  done

再验证 NodePort 可达性，同样应全部输出欢迎页：

[root@k8s-master1 work]# for node_ip in ${WORK_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "curl -s ${node_ip}:32120"
  done

浏览器访问

最后从集群外拿任意节点 IP:32120 走一遍浏览器访问：

不通的时候查什么

Pod IP ping 不通，先查 flannel 和网段规划；Service IP 通而 NodePort 不通，基本是 kube-proxy 没起来或 iptables 模式问题；浏览器访问不到但节点内 curl 正常，检查节点防火墙有没有放行高位端口。IP、端口以自己的实际输出为准，别照抄数字。