二进制部署多 Master K8s 集群 v1.18（二）：Master 组件与 apiserver 高可用-星海夜航

这一篇把三个 Master 组件部署起来：kube-apiserver、kube-scheduler、kube-controller-manager。scheduler 和 controller-manager 多实例运行，靠竞争选举产生 leader，其余实例阻塞待命，leader 挂了重新选举；apiserver 无状态，用每节点一个 nginx 做 4 层代理保证可用。注意：Master 若只当管理节点可以不装 docker、kubelet、kube-proxy，但后面部署 metrics-server、istio 时会出现无法运行的情况，建议还是装上。如果没有特别说明，以下操作都在 k8s-master1 上执行，再远程分发文件和命令。

程序包与 kube-nginx 四层代理

k8s-server 包上传到 /opt/k8s/work 解压，二进制分发到所有 master；kubelet、kube-proxy 分发给全部 worker，目录 /opt/k8s/bin：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# tar -zxvf kubernetes-server-linux-amd64.tar.gz
[root@k8s-master1 work]# cd kubernetes/
[root@k8s-master1 kubernetes]# tar -zxvf kubernetes-src.tar.gz

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

[root@k8s-master1 work]# for node_ip in ${WORK_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kubernetes/server/bin/{kube-proxy,kubelet} root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

apiserver 高可用的整体思路：controller-manager、scheduler 连本机 apiserver，只要有一个实例正常就可用；集群内 Pod 走服务域名 kubernetes，由 kube-dns 解析出多个 apiserver 节点 IP；每个节点起一个 nginx 进程，后端对接全部 apiserver 实例做健康检查和负载均衡；kubelet、kube-proxy 等通过本地 nginx（监听 127.0.0.1）访问 apiserver。先下载编译 nginx，官方包 nginx-1.16.1.tar.gz，国内直连很慢，换国内源下载：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# wget https://mirrors.huaweicloud.com/nginx/nginx-1.16.1.tar.gz

[root@k8s-master1 work]# yum -y install gcc-c++
[root@k8s-master1 work]# tar -zxvf nginx-1.16.1.tar.gz

[root@k8s-master1 work]# cd nginx-1.16.1/
[root@k8s-master1 work]# mkdir nginx-prefix
[root@k8s-master1 nginx-1.16.1]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module

[root@k8s-master1 nginx-1.16.1]# make && make install

所有节点创建目录、拷贝编译好的二进制并重命名为 kube-nginx（路径按自己 configure 的安装位置填写）：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
  done

[root@k8s-master1 work]# for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    scp /opt/k8s/work/nginx-1.16.1/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  done

nginx 配置只保留 4 层透明转发，upstream backend 里的 server 列表就是各 kube-apiserver 节点 IP，按实际情况修改，然后分发到所有节点：

[root@k8s-master1 work]# cat > kube-nginx.conf <<EOF
worker_processes 1;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash $remote_addr consistent;
        server 192.168.66.62:6443        max_fails=3 fail_timeout=30s;
        server 192.168.66.63:6443        max_fails=3 fail_timeout=30s;
        server 192.168.66.64:6443        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 127.0.0.1:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  done

创建 kube-nginx 的 systemd unit 文件、分发、启动：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

[root@k8s-master1 work]# for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  done

[root@k8s-master1 work]# for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  done

检查 kube-nginx 服务运行状态：

[root@k8s-master1 work]# for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  done

部署 kube-apiserver 集群

签 apiserver 证书：CSR 的 hosts 字段列全 master 节点 IP、kubernetes 服务的 Cluster IP 和域名，生成证书和私钥后拷贝到所有 master 节点：

[root@k8s-master1 ~]# cd /opt/k8s/work/
[root@k8s-master1 work]# cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes-master",
  "hosts": [
    "127.0.0.1",
    "192.168.66.62",
    "192.168.66.63",
    "192.168.66.64",
    "${CLUSTER_KUBERNETES_SVC_IP}",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local.",
    "kubernetes.default.svc.${CLUSTER_DNS_DOMAIN}."
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "dqz"
    }
  ]
}
EOF

[root@k8s-master1 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 
  -ca-key=/opt/k8s/work/ca-key.pem 
  -config=/opt/k8s/work/ca-config.json 
  -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

[root@k8s-master1 work]# ls kubernetes*pem
kubernetes-key.pem  kubernetes.pem

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/kubernetes/cert"
    scp kubernetes*.pem root@${node_ip}:/etc/kubernetes/cert/
  done

加密配置文件（Secret 落盘加密用）创建后拷贝到各 master 的 /etc/kubernetes：

[root@k8s-master1 work]# cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/
  done

审计策略文件创建并分发：

[root@k8s-master1 work]# cat > audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch

  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get

  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update

  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get

  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'

  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses
  # for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch

  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch

  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
      
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yaml
  done

再签一套 proxy-client 证书，供后续访问 metrics-server 或 kube-prometheus 使用，生成后拷贝到所有 master：

[root@k8s-master1 work]# cat > proxy-client-csr.json <<EOF
{
  "CN": "aggregator",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "dqz"
    }
  ]
}
EOF

[root@k8s-master1 work]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem 
  -ca-key=/etc/kubernetes/cert/ca-key.pem  
  -config=/etc/kubernetes/cert/ca-config.json  
  -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client

[root@k8s-master1 work]# ls proxy-client*.pem
proxy-client-key.pem  proxy-client.pem

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp proxy-client*.pem root@${node_ip}:/etc/kubernetes/cert/
  done

apiserver 的 systemd unit 模板按节点替换变量（如 advertise-address）后分发：

[root@k8s-master1 work]# cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \
  --advertise-address=##MASTER_IP## \
  --default-not-ready-toleration-seconds=360 \
  --default-unreachable-toleration-seconds=360 \
  --feature-gates=DynamicAuditing=true \
  --max-mutating-requests-inflight=2000 \
  --max-requests-inflight=4000 \
  --default-watch-cache-size=200 \
  --delete-collection-workers=2 \
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
  --etcd-cafile=/etc/kubernetes/cert/ca.pem \
  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \
  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \
  --etcd-servers=${ETCD_ENDPOINTS} \
  --bind-address=##MASTER_IP## \
  --secure-port=6443 \
  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \
  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \
  --insecure-port=0 \
  --audit-dynamic-configuration \
  --audit-log-maxage=15 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-truncate-enabled \
  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
  --profiling \
  --anonymous-auth=false \
  --client-ca-file=/etc/kubernetes/cert/ca.pem \
  --enable-bootstrap-token-auth \
  --requestheader-allowed-names="aggregator" \
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --service-account-key-file=/etc/kubernetes/cert/ca.pem \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-admission-plugins=NodeRestriction \
  --allow-privileged=true \
  --apiserver-count=3 \
  --event-ttl=168h \
  --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \
  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \
  --kubelet-https=true \
  --kubelet-timeout=10s \
  --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \
  --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \
  --service-cluster-ip-range=${SERVICE_CIDR} \
  --service-node-port-range=${NODE_PORT_RANGE} \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

[root@k8s-master1 work]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service 
  done

[root@k8s-master1 work]# ls kube-apiserver*.service
kube-apiserver-192.168.66.62.service  kube-apiserver-192.168.66.64.service
kube-apiserver-192.168.66.63.service

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.service
  done

依次在各 master 启动服务、检查运行状态：

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
  done

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-apiserver |grep 'Active:'"
  done

[root@k8s-master1 work]# kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

检查集群整体状态：

[root@k8s-master1 work]# kubectl get all --all-namespaces
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.254.0.1   <none>        443/TCP   56s

[root@k8s-master1 work]# kubectl get componentstatuses

上图中 scheduler 与 controller-manager 显示 Unhealthy 属正常——这两个组件还没部署，装完再检查就恢复 ok。

部署高可用 kube-controller-manager

3 节点集群，启动后通过竞争选举产生 leader，其余节点阻塞，leader 不可用时再次选举。证书用在两处：与 kube-apiserver 的安全端口通信；在安全端口（https，10252）输出 prometheus 格式的 metrics。先创建证书签名请求、生成证书私钥并分发到所有 master：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cat > kube-controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.66.62",
      "192.168.66.63",
      "192.168.66.64"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-controller-manager",
        "OU": "dqz"
      }
    ]
}
EOF

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 
  -ca-key=/opt/k8s/work/ca-key.pem 
  -config=/opt/k8s/work/ca-config.json 
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager


[root@k8s-master1 work]# ls kube-controller-manager*pem
kube-controller-manager-key.pem  kube-controller-manager.pem

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/
  done

创建并分发 kubeconfig：

[root@k8s-master1 ~]# cd /opt/k8s/work

[root@k8s-master1 work]# kubectl config set-cluster kubernetes 
  --certificate-authority=/opt/k8s/work/ca.pem 
  --embed-certs=true 
  --server="https://##NODE_IP##:6443" 
  --kubeconfig=kube-controller-manager.kubeconfig

[root@k8s-master1 work]# kubectl config set-credentials system:kube-controller-manager 
  --client-certificate=kube-controller-manager.pem 
  --client-key=kube-controller-manager-key.pem 
  --embed-certs=true 
  --kubeconfig=kube-controller-manager.kubeconfig

[root@k8s-master1 work]# kubectl config set-context system:kube-controller-manager 
  --cluster=kubernetes 
  --user=system:kube-controller-manager 
  --kubeconfig=kube-controller-manager.kubeconfig

[root@k8s-master1 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" kube-controller-manager.kubeconfig > kube-controller-manager-${node_ip}.kubeconfig
    scp kube-controller-manager-${node_ip}.kubeconfig root@${node_ip}:/etc/kubernetes/kube-controller-manager.kubeconfig
  done

unit 模板创建、为各 Master 替换变量后分发：

[root@k8s-master1 work]# cat > kube-controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \
  --profiling \
  --cluster-name=kubernetes \
  --controllers=*,bootstrapsigner,tokencleaner \
  --kube-api-qps=1000 \
  --kube-api-burst=2000 \
  --leader-elect \
  --use-service-account-credentials\
  --concurrent-service-syncs=2 \
  --bind-address=0.0.0.0 \
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --client-ca-file=/etc/kubernetes/cert/ca.pem \
  --requestheader-allowed-names="aggregator" \
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
  --experimental-cluster-signing-duration=876000h \
  --horizontal-pod-autoscaler-sync-period=10s \
  --concurrent-deployment-syncs=10 \
  --concurrent-gc-syncs=30 \
  --node-cidr-mask-size=24 \
  --service-cluster-ip-range=${SERVICE_CIDR} \
  --pod-eviction-timeout=6m \
  --terminated-pod-gc-threshold=10000 \
  --root-ca-file=/etc/kubernetes/cert/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

[root@k8s-master1 work]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${NODE_IPS[i]}.service 
  done

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-controller-manager.service
  done

启动服务并检查状态，controller-manager 监听 10252 端口接收 https 请求：

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
  done

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-controller-manager|grep Active"
  done

[root@k8s-master1 work]# netstat -lnpt | grep kube-cont
tcp        0      0 192.168.66.62:10252     0.0.0.0:*               LISTEN      37321/kube-controll

查看输出的 metrics：如果启动文件里关掉了 –port=0 和 –secure-port=10252 这两个参数，只能通过 http 方式获取 metrics；若保留了这两个参数，则用后面的 https 方式访问。命令在 kube-controller-manager 节点上执行：

curl -s http://192.168.66.62:10252/metrics|head
curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem http://192.168.66.62:10252/metrics |head

curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://192.168.66.62:10252/metrics |head

通过 endpoints 查看当前 leader，可以看到 leader 为 k8s-master1：

[root@k8s-master1 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml

再检查一次集群状态：

[root@k8s-master1 work]# kubectl get cs

部署高可用 kube-scheduler

流程与前两者一致。创建证书签名请求（hosts 填写自己 master 集群的 IP）、生成证书私钥并分发到所有 master：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cat > kube-scheduler-csr.json <<EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.66.62",
      "192.168.66.63",
      "192.168.66.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-scheduler",
        "OU": "dqz"
      }
    ]
}
EOF

[root@k8s-master1 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 
  -ca-key=/opt/k8s/work/ca-key.pem 
  -config=/opt/k8s/work/ca-config.json 
  -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

[root@k8s-master1 work]# ls kube-scheduler*pem
kube-scheduler-key.pem  kube-scheduler.pem

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-scheduler*.pem root@${node_ip}:/etc/kubernetes/cert/
  done

创建 kubeconfig（内含 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书）并分发：

[root@k8s-master1 ~]# cd /opt/k8s/work

[root@k8s-master1 work]# kubectl config set-cluster kubernetes 
--certificate-authority=/opt/k8s/work/ca.pem 
--embed-certs=true 
--server="https://##NODE_IP##:6443" 
--kubeconfig=kube-scheduler.kubeconfig


[root@k8s-master1 work]# kubectl config set-credentials system:kube-scheduler 
--client-certificate=kube-scheduler.pem 
--client-key=kube-scheduler-key.pem 
--embed-certs=true 
--kubeconfig=kube-scheduler.kubeconfig


[root@k8s-master1 work]# kubectl config set-context system:kube-scheduler 
--cluster=kubernetes 
--user=system:kube-scheduler 
--kubeconfig=kube-scheduler.kubeconfig

[root@k8s-master1 work]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" kube-scheduler.kubeconfig > kube-scheduler-${node_ip}.kubeconfig
    scp kube-scheduler-${node_ip}.kubeconfig root@${node_ip}:/etc/kubernetes/kube-scheduler.kubeconfig
  done

创建 kube-scheduler 配置文件并分发到所有 master 节点：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cat <<EOF | sudo tee kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
EOF

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-scheduler.yaml root@${node_ip}:/etc/kubernetes/
  done

unit 模板创建、替换变量、分发：

[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# cat > kube-scheduler.service.template <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-scheduler
ExecStart=/opt/k8s/bin/kube-scheduler \
--config=/etc/kubernetes/kube-scheduler.yaml \
--bind-address=0.0.0.0 \
--tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \
--tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \
--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--client-ca-file=/etc/kubernetes/cert/ca.pem \
--requestheader-allowed-names="aggregator" \
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \
--requestheader-extra-headers-prefix="X-Remote-Extra-" \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--logtostderr=true \
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF

[root@k8s-master1 work]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-${MASTER_IPS[i]}.service 
  done
[root@k8s-master1 work]# ls kube-scheduler*.service

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-scheduler-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-scheduler.service
  done

启动服务、检查运行状态。scheduler 监听两个端口：10251 接收 http 请求，非安全端口、不需要认证授权；10259 接收 https 请求，安全端口；两个接口都对外提供 /metrics 和 /healthz 的访问。以下命令在 kube-scheduler 节点执行：

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-scheduler"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler"
  done

[root@k8s-master1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-scheduler|grep Active"
  done

[root@k8s-master1 work]# netstat -lnpt |grep kube-sch
tcp6       0      0 :::10251                :::*                    LISTEN      75717/kube-schedule 
tcp6       0      0 :::10259                :::*                    LISTEN      75717/kube-schedule

[root@k8s-master1 work]# curl -s http://192.168.66.62:10251/metrics |head
[root@k8s-master1 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://192.168.66.62:10259/metrics |head

查看当前 leader，是 k8s-master1：

[root@k8s-master1 work]# kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml

最后再看集群状态，scheduler 已恢复健康：

[root@k8s-master1 work]# kubectl get cs

收尾说明

至此 Master 层高可用完成：apiserver 靠每节点的 kube-nginx 轮询三个实例，controller-manager 与 scheduler 靠 leader 选举兜底。系列后续篇继续做 TLS Bootstrap、kubelet 与网络插件，最后按验证清单整体过一遍。

文章版权归作者所有，未经允许请勿转载。

THE END

二进制部署多 Master K8s 集群 v1.18（二）：Master 组件与 apiserver 高可用

程序包与 kube-nginx 四层代理

部署 kube-apiserver 集群

部署高可用 kube-controller-manager

部署高可用 kube-scheduler

收尾说明

请登录后发表评论