2043 kube-controller-manager

部署一个三实例 kube-controller-manager 的集群，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

部署策略：

  • 3 节点高可用；

  • 关闭非安全端口，在安全端口 10252 接收 https 请求；

  • 使用 kubeconfig 访问 apiserver 的安全端口；

  • 自动 approve kubelet 证书签名请求 (CSR)，证书过期后自动轮转；

  • 各 controller 使用自己的 ServiceAccount 访问 apiserver；

部署软件规划

IP

部署软件包

10.40.61.116

kube-controller-manager

10.40.58.153

kube-controller-manager

10.40.58.154

kube-controller-manager

01.创建 kube-controller-manager 证书和私钥

创建证书签名请求：

cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

说明： hosts 列表包含所有 kube-controller-manager 节点 IP； CN 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。

创建 kube-controller-manager凭证与私钥:

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

结果将产生以下几个文件：

kube-controller-manager-key.pem
kube-controller-manager.pem

02.创建kubeconfig

kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、 CA 证书和 kube-controller-manager 证书

先确定apiserver对外提供服务的地址

export KUBE_APISERVER="https://apiserver-p001.svc.gxd88.cn:6443"

  • 设置集群参数

      kubectl config set-cluster kubernetes \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-controller-manager.kubeconfig

  • 设置客户端认证参数

      kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

  • 设置上下文参数

      kubectl config set-context default \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

  • 设置默认上下文

      kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

03.service-accout证书

请使用kube-apiserver文档中生成的service-account证书， 其中kube-apiserver使用公钥， kube-controller-manager使用私钥

04.使用systemd管理kube-controller-manager

tee /etc/systemd/system/kube-controller-manager.service <<-EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/srv/kubernetes/bin/kube-controller-manager \\
  --kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \\
  --authentication-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \\
  --authorization-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \\
  --bind-address=0.0.0.0 \\
  --pod-eviction-timeout=3m0s \\
  --node-eviction-rate=0.1 \\
  --secondary-node-eviction-rate=0.1 \\
  --large-cluster-size-threshold=1 \\
  --allocate-node-cidrs=true \\
  --service-cluster-ip-range=10.244.0.0/16 \\
  --cluster-cidr=10.243.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/srv/kubernetes/pki/ca.pem \\
  --cluster-signing-key-file=/srv/kubernetes/pki/ca-key.pem \\
  --service-account-private-key-file=/srv/kubernetes/pki/service-account-key.pem \\
  --use-service-account-credentials=true \\
  --root-ca-file=/srv/kubernetes/pki/ca.pem \\
  --leader-elect=true \\
  --v=2 \\
  --logtostderr=false \\
  --log-dir=/srv/kubernetes/log
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

启动参数的请看参数详解

05.启动/停止 kube-controller-manager

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kube-controller-manager.service

kube-controller-manager can be started and stopped as follows:

sudo systemctl start kube-controller-manager.service
sudo systemctl stop kube-controller-manager.service

06.验证

查看进程是否正常

$ systemctl status kube-controller-manager |grep active
  Active: active (running) since 六 2020-04-11 16:11:58 CST; 6 days ago

查看权限:

$ kubectl describe clusterrole system:kube-controller-manager
Name:         system:kube-controller-manager
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                                  Non-Resource URLs  Resource Names             Verbs
  ---------                                  -----------------  --------------             -----
  secrets                                    []                 []                         [create delete get update]
  serviceaccounts                            []                 []                         [create get update]
  events                                     []                 []                         [create patch update]
  events.events.k8s.io                       []                 []                         [create patch update]
  endpoints                                  []                 []                         [create]
  serviceaccounts/token                      []                 []                         [create]
  tokenreviews.authentication.k8s.io         []                 []                         [create]
  subjectaccessreviews.authorization.k8s.io  []                 []                         [create]
  leases.coordination.k8s.io                 []                 []                         [create]
  endpoints                                  []                 [kube-controller-manager]  [get update]
  leases.coordination.k8s.io                 []                 [kube-controller-manager]  [get update]
  configmaps                                 []                 []                         [get]
  namespaces                                 []                 []                         [get]
  *.*                                        []                 []                         [list watch]

ClusteRole system:kube-controller-manager 的权限很小，只能创建 secret、serviceaccount 等资源对象，各 controller 的权限分散到 ClusterRole system:controller:XXX 中。

当在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。

kubectl get clusterrole|grep controller
system:controller:attachdetach-controller                              2020-04-11T07:37:53Z
system:controller:certificate-controller                               2020-04-11T07:37:53Z
system:controller:clusterrole-aggregation-controller                   2020-04-11T07:37:53Z
system:controller:cronjob-controller                                   2020-04-11T07:37:53Z
system:controller:daemon-set-controller                                2020-04-11T07:37:53Z
system:controller:deployment-controller                                2020-04-11T07:37:53Z
system:controller:disruption-controller                                2020-04-11T07:37:53Z
system:controller:endpoint-controller                                  2020-04-11T07:37:53Z
system:controller:endpointslice-controller                             2020-04-11T07:37:53Z
system:controller:expand-controller                                    2020-04-11T07:37:53Z
system:controller:generic-garbage-collector                            2020-04-11T07:37:53Z
system:controller:horizontal-pod-autoscaler                            2020-04-11T07:37:53Z
system:controller:job-controller                                       2020-04-11T07:37:53Z
system:controller:namespace-controller                                 2020-04-11T07:37:53Z
system:controller:node-controller                                      2020-04-11T07:37:53Z
system:controller:persistent-volume-binder                             2020-04-11T07:37:53Z
system:controller:pod-garbage-collector                                2020-04-11T07:37:53Z
system:controller:pv-protection-controller                             2020-04-11T07:37:53Z
system:controller:pvc-protection-controller                            2020-04-11T07:37:53Z
system:controller:replicaset-controller                                2020-04-11T07:37:53Z
system:controller:replication-controller                               2020-04-11T07:37:53Z
system:controller:resourcequota-controller                             2020-04-11T07:37:53Z
system:controller:route-controller                                     2020-04-11T07:37:53Z
system:controller:service-account-controller                           2020-04-11T07:37:53Z
system:controller:service-controller                                   2020-04-11T07:37:53Z
system:controller:statefulset-controller                               2020-04-11T07:37:53Z
system:controller:ttl-controller                                       2020-04-11T07:37:53Z
system:kube-controller-manager                                         2020-04-11T07:37:53Z

查看其中deployment controller

kubectl describe clusterrole system:controller:deployment-co
Name:         system:controller:deployment-controller
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                          Non-Resource URLs  Resource Names  Verbs
  ---------                          -----------------  --------------  -----
  replicasets.apps                   []                 []              [create delete get list patch update watch]
  replicasets.extensions             []                 []              [create delete get list patch update watch]
  events                             []                 []              [create patch update]
  events.events.k8s.io               []                 []              [create patch update]
  pods                               []                 []              [get list update watch]
  deployments.apps                   []                 []              [get list update watch]
  deployments.extensions             []                 []              [get list update watch]
  deployments.apps/finalizers        []                 []              [update]
  deployments.apps/status            []                 []              [update]
  deployments.extensions/finalizers  []                 []              [update]
  deployments.extensions/status      []                 []              [update]

07.参数详解

kube-controller-manager，k8s的大脑，大部分控制器所在，大管家，配置包括:

  • 驱逐相关

  • 高可用选主相关

  • PodIP网段

  • …

启动命令

/srv/kubernetes/bin/kube-controller-manager \
  --kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig \
  --pod-eviction-timeout=3m0s \
  --node-eviction-rate=0.1 \
  --secondary-node-eviction-rate=0.1 \
  --large-cluster-size-threshold=1 \
  --address=0.0.0.0 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.244.0.0/16 \
  --cluster-cidr=10.243.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/srv/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/srv/kubernetes/pki/ca-key.pem \
  --service-account-private-key-file=/srv/kubernetes/pki/service-account-key.pem \
  --use-service-account-credentials=true \
  --root-ca-file=/srv/kubernetes/pki/ca.pem \
  --leader-elect=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/srv/kubernetes/log

参数用途说明

--leader-elect=true

开启选举。

利用etcd的强一致性，可以用来组件的选主，kube-controller-manager就是利用这个特性实现的高可用。

高可用条件: kube-controller-manager数量大于等于2即可。

--pod-eviction-timeout=3m0s

驱逐超时，默认5分钟，配置为3分钟，从controller感知到节点挂了开始计时。

--node-eviction-rate=0.1

一级驱逐速率，这里两个0.1相当于每个节点每10秒钟驱逐一个Pod。

Number of nodes per second on which pods are deleted in case of node failure when a zone is healthy (see –unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters.

当开启多可用区时，一级驱逐速率只有在zone健康的时候生效；当非多可用区的时候，zone代表整个集群。

--secondary-node-eviction-rate=0.1

二级驱逐速率，这里两个0.1相当于每个节点每10秒钟驱逐一个Pod。

Number of nodes per second on which pods are deleted in case of node failure when a zone is unhealthy (see –unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. This value is implicitly overridden to 0 if the cluster size is smaller than –large-cluster-size-threshold.

--large-cluster-size-threshold=1

Number of nodes from which NodeController treats the cluster as large for the eviction logic purposes. –secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller.

设置一个"大"集群的阈值，多大为大集群，默认为50，当集群节点数小于该值的时候，二级驱逐速率就是0，不驱逐！

--unhealthy-zone-threshold=0.55

Fraction of Nodes in a zone which needs to be not Ready (minimum 3) for zone to be treated as unhealthy.

Zone被认为不健康的条件，大于55%的节点故障（NodeReady的节点数大于等于3）.

举例:

比如现在ZoneB有20个节点，那么当20 * 0.55 = 11个以上节点挂了后，那么这个Zone就被认为不健康了，此时一级驱逐速率也就不生效了；由于总结点数大于1，那么二级速率有效，所以整个集群会以10s/pod/node进行驱逐。

08.Q&A

Q:

 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"

A:

Q:

No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.

A: 需要在启动命令中添加如下配置

  --authentication-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig 
  --authorization-kubeconfig=/srv/kubernetes/kubeconfig/kube-controller-manager.kubeconfig
上一页2042 kube-scheduler下一页205 Node节点部署及维护

最后更新于