摘要：
VIP需要与主机位于同一局域网中！
K8S部署笔记
一、集群环境说明
| 主机名 | IP地址 | 说明 |
|---|---|---|
| k8s-master01 | 192.168.1.107 | master节点 |
| k8s-master02 | 192.168.1.108 | master节点 |
| k8s-master03 | 192.168.1.109 | master节点 |
| k8s-master-lb（在master节点） | 192.168.1.236 | keepalived虚拟IP |
| k8s-node01 | 192.168.1.110 | worker节点 |
| k8s-node02 | 192.168.1.111 | worker节点 |
| 配置信息 | 备注 |
|---|---|
| 系统版本 | CentOS 7.9 |
| Docker版本 | 19.03.x |
| Pod网段 | 172.168.0.0/12 |
| Service网段 | 10.96.0.0/12 |
注意：
VIP（虚拟IP）不要和公司内网IP重复，首先去ping一下，不通才可用。VIP需要和主机在同一个局域网内！
二、基础环境配置（以下操作所有节点都得执行）
2.1、配置hosts解析
[root@k8s-master01 ~]# cat /etc/hosts
192.168.1.107 k8s-master01
192.168.1.108 k8s-master02
192.168.1.109 k8s-master03
192.168.1.236 k8s-master-lb # 如果不是高可用集群，该IP为Master01的IP
192.168.1.110 k8s-node01
192.168.1.111 k8s-node02
2.2、更换yum源码
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
2.3、安装常用工具
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y
2.4、关闭防火墙、selinux、dnsmasq、swap
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
关闭swap分区
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
2.5、时间同步配置
# 安装ntpdate
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y
# 更改时区
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# 设置定时任务同步时间
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
*/5 * * * * ntpdate time2.aliyun.com
2.6、优化Linux
ulimit -SHn 65535
vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
# 然后重启Linux
reboot
2.7、所有节点升级系统并重启，此处升级没有升级内核，下节会单独升级内核：
# CentOS7需要升级，CentOS8可以按需升级系统
yum update -y --exclude=kernel* && reboot
三、内核升级
3.1、配置免密登录（Master01上）
Master01节点免密钥登录其他节点，安装过程中生成配置文件和证书均在Master01上操作，集群管理也在Master01上操作，阿里云或者AWS上需要单独一台kubectl服务器。密钥配置如下：
# 一直回车就行
ssh-keygen -t rsa
for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
3.2、下载安装所有的源码文件：（Master01上）
cd /root/ ; git clone https://github.com/dotbalo/k8s-ha-install.git
3.3、下载升级所需安装包（Master01上）
CentOS7 需要升级内核至4.18+，本地升级的版本为4.19
# 在master01节点下载内核
cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
# 从master01节点传到其他节点：
for i in k8s-master02 k8s-master03 k8s-node01 k8s-node02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done
3.4、内核升级（所有节点）
cd /root && yum localinstall -y kernel-ml*
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
# 检查默认内核是不是4.19
grubby --default-kernel /boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64
# 所有节点重启，然后检查内核是不是4.19
reboot
[root@k8s-node02 ~]# uname -a
Linux k8s-node02 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
3.5、安装ipvsadm（所有节点）
yum install ipvsadm ipset sysstat conntrack libseccomp -y
所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：
vim /etc/modules-load.d/ipvs.conf
# 加入以下内容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack # 4.18 改成这个nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
# 然后执行
systemctl enable --now systemd-modules-load.service
3.6、开启一些k8s集群中必须的内核参数，配置k8s内核（所有节点）
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
# 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载
reboot
[root@k8s-master01 ~]# lsmod | grep --color=auto -e ip_vs -e nf_conntrack
ip_vs_ftp 16384 0
nf_nat 32768 1 ip_vs_ftp
ip_vs_sed 16384 0
ip_vs_nq 16384 0
ip_vs_fo 16384 0
ip_vs_sh 16384 0
ip_vs_dh 16384 0
ip_vs_lblcr 16384 0
ip_vs_lblc 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs_wlc 16384 0
ip_vs_lc 16384 0
ip_vs 151552 24 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack 143360 2 nf_nat,ip_vs
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 4 nf_conntrack,nf_nat,xfs,ip_vs
四、基本组件安装
本节主要安装的是集群中用到的各种组件，比如Docker-ce、Kubernetes各组件等。
4.1、安装Docker-ce 19.03（所有节点）
yum install docker-ce-19.03.* -y
4.1.1温馨提示：
由于新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
4.1.2、所有节点设置开机自启动Docker
systemctl daemon-reload && systemctl enable --now docker
4.2、安装k8s组件（所有节点）
所有节点安装最新版本kubeadm：
yum list kubeadm.x86_64 --showduplicates | sort -r
yum install kubeadm -y
默认配置的pause镜像使用gcr.io仓库，国内可能无法访问，所以这里配置Kubelet使用阿里云的pause镜像：
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF
设置Kubelet开机自启动：
systemctl daemon-reload
systemctl enable --now kubelet
五、高可用组件安装（所有Master节点）
（注意：如果不是高可用集群，haproxy和keepalived无需安装）
5.1、通过yum安装HAProxy和KeepAlived
yum install keepalived haproxy -y
5.2、配置HAProxy
所有Master节点配置HAProxy（详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同）：
mkdir /etc/haproxy
[root@k8s-master01 ~]# cat /etc/haproxy/haproxy.cfg
global
maxconn 2000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 15s
timeout http-keep-alive 15s
frontend monitor-in
bind *:33305
mode http
option httplog
monitor-uri /monitor
frontend k8s-master
bind 0.0.0.0:16443
bind 127.0.0.1:16443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server k8s-master01 192.168.1.107:6443 check
server k8s-master02 192.168.1.108:6443 check
server k8s-master03 192.168.1.109:6443 check
5.3、配置KeepAlived
注意每个节点的IP和网卡（interface参数）
Master01节点的配置：
mkdir /etc/keepalived
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface ens33
mcast_src_ip 192.168.1.107
virtual_router_id 51
priority 101
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.236
}
track_script {
chk_apiserver
}
}
Master02节点的配置：
mkdir /etc/keepalived
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.1.108
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.236
}
track_script {
chk_apiserver
}
}
Master03节点的配置：
mkdir /etc/keepalived
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.1.109
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.236
}
track_script {
chk_apiserver
}
}
5.4、所有master节点配置KeepAlived健康检查文件：
cat > /etc/keepalived/check_apiserver.sh << EFO
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep haproxy)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
EFO
5.4.1、脚本授权
chmod +x /etc/keepalived/check_apiserver.sh
5.4.2、启动haproxy和keepalived
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived
5.5、如果安装了keepalived和haproxy，需要测试keepalived是否是正常的
# 测试VIP（任意节点上）
[root@k8s-node02 ~]# ping 192.168.1.236 -c 4
PING 192.168.1.236 (192.168.1.236) 56(84) bytes of data.
64 bytes from 192.168.1.236: icmp_seq=1 ttl=64 time=0.923 ms
64 bytes from 192.168.1.236: icmp_seq=2 ttl=64 time=0.433 ms
64 bytes from 192.168.1.236: icmp_seq=3 ttl=64 time=0.554 ms
64 bytes from 192.168.1.236: icmp_seq=4 ttl=64 time=0.457 ms
--- 192.168.1.236 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3061ms
rtt min/avg/max/mdev = 0.433/0.591/0.923/0.198 ms
# 测试haproxy（任意节点上）
[root@k8s-node01 ~]# telnet 192.168.1.236 16443
Trying 192.168.1.236...
Connected to 192.168.1.236.
Escape character is '^]'.
Connection closed by foreign host.
# 排查思路
如果ping不通且telnet没有出现 ] ，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等
所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld
所有节点查看selinux状态，必须为disable：getenforce
master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy
master节点查看监听端口：netstat -lntp
六、 集群初始化
官方初始化文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
Master01：（# 注意，如果不是高可用集群，192.168.1.236:16443改为master01的地址，16443改为apiserver的端口，默认是6443，注意更改v1.18.5自己服务器kubeadm的版本：kubernetesVersion）
6.1、Master01节点创建kubeadm-config.yaml配置（Master01上）
[root@k8s-master01 ~]# cat /root/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: 7t2weq.bjbawausm0jaxury
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.1.107
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master01
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
certSANs:
- 192.168.1.236
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.1.236:16443
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.0 # 改成对应的版本
networking:
dnsDomain: cluster.local
podSubnet: 172.168.0.0/12 # 如果跟公司ip冲突得改
serviceSubnet: 10.96.0.0/12 # 如果跟公司ip冲突得改
scheduler: {}
更新kubeadm文件的命令（Master01上）
# 更新之后的文件是new.yaml
[root@k8s-master01 ~]# kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml
6.2、将new.yaml文件复制到其他节点（Master01上）
scp new.yaml k8s-master02:/root/
scp new.yaml k8s-master03:/root/
scp new.yaml k8s-node01:/root/
scp new.yaml k8s-node02:/root/
6.3、下载镜像（Master01、02、03上）
[root@k8s-master01 ~]# kubeadm config images pull --config /root/new.yaml
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
6.4、设置开机自启动kubelet（所有节点）
systemctl enable --now kubelet
6.5、Master01节点初始化，初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他Master节点加入Master01即可：（Master01上）
kubeadm init --config /root/new.yaml --upload-certs
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
# 初始化成功以后，会产生Token值，用于其他节点加入时使用，因此要记录下初始化成功生成的token值（令牌值）：
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
# 这个就是要在另外2个master节点执行的加入集群的命令
kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
--control-plane --certificate-key b66aa49bb32a780ac5b58841c50a7767a9638a9d313e23c9413218473e527ec2
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
# 这个就是要在另外2个node节点执行的加入集群的命令
kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
6.6、如果初始化失败，重置后再次初始化，命令如下：
kubeadm reset -f ; ipvsadm --clear ; rm -rf ~/.kube
6.7、Master01节点配置环境变量，用于访问Kubernetes集群（Master01上）
cat <<EOF >> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc
# 查看节点状态：
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane,master 3m24s v1.20.0
6.8、采用初始化安装方式，所有的系统组件均以容器的方式运行并且在kube-system命名空间内，此时可以查看Pod状态：
[root@k8s-master01 ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-54d67798b7-8w5hd 0/1 Pending 0 4m18s <none> <none> <none> <none>
coredns-54d67798b7-vb2ll 0/1 Pending 0 4m18s <none> <none> <none> <none>
etcd-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
kube-apiserver-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
kube-controller-manager-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
kube-proxy-5bws8 1/1 Running 0 4m18s 192.168.1.107 k8s-master01 <none> <none>
kube-scheduler-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
七、高可用Master
7.1、matser02节点执行
[root@k8s-master02 ~]# kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
--control-plane --certificate-key b66aa49bb32a780ac5b58841c50a7767a9638a9d313e23c9413218473e527ec2
# 成功显示，如果想在另外的master节点使用kubectl命令，就执行以下操作。
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
# matser01节点查看，已经成功加入集群
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane,master 16m v1.20.0
k8s-master02 NotReady control-plane,master 2m21s v1.20.0
7.2、如果证书失效、采用以下方法重新生成证书（master01上）
# Token过期后生成新的token：
[root@k8s-master01 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.1.236:16443 --token 8k8qzk.d43ed9gfgw1st3xi --discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
# Master需要生成--certificate-key
[root@k8s-master01 ~]# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
43c5695789c0dc4433f480a05683d55887e836b71b452b407138d8dd54cad937
7.2.1、把master03节点加入集群（master03上）
# 执行（kubeadm token create --print-join-command）后生成的命令+执行（kubeadm init phase upload-certs --upload-certs）生成的key替换到参数（--control-plane --certificate-key）然后在master03执行
kubeadm join 192.168.1.236:16443 --token 8k8qzk.d43ed9gfgw1st3xi
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
--control-plane --certificate-key 43c5695789c0dc4433f480a05683d55887e836b71b452b407138d8dd54cad937
7.3、把node节点加入集群（node01、02上）
Node节点上主要部署公司的一些业务应用，生产环境中不建议Master节点部署系统组件之外的其他Pod，测试环境可以允许Master节点部署Pod以节省系统资源。
# 只要执行（kubeadm token create --print-join-command）后生成的命令
kubeadm join 192.168.1.236:16443 --token 8k8qzk.d43ed9gfgw1st3xi
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
# 看到以下的就是成功了
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
# 查看master01上查看节点信息
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane,master 33m v1.20.0
k8s-master02 NotReady control-plane,master 19m v1.20.0
k8s-master03 NotReady control-plane,master 10m v1.20.0
k8s-node01 NotReady <none> 116s v1.20.0
k8s-node02 NotReady <none> 113s v1.20.0
报错：
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty
[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
[ERROR Port-10250]: Port 10250 is in use
# /etc/kubernetes/manifests 删除这个目录
# 10250停了这端口
八、 Calico组件的安装（master01执行）
8.1、切换分支
[root@k8s-master01 ~]# cd /root/k8s-ha-install && git checkout manual-installation-v1.20.x && cd calico/
Already on 'manual-installation-v1.20.x'
8.2、修改calico-etcd.yaml的以下位置
sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.1.107:2379,https://192.168.1.108:2379,https://192.168.1.109:2379"#g' calico-etcd.yaml
ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '
'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '
'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '
'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml
sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`
sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.1.0/16"@ value: '"${POD_SUBNET}"'@g' calico-etcd.yaml
[root@k8s-master01 calico]# kubectl apply -f calico-etcd.yaml
secret/calico-etcd-secrets created
configmap/calico-config created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
查看容器状态（需等待几分钟）
[root@k8s-master01 dashboard]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5f6d4b864b-khq4h 1/1 Running 0 13m
calico-node-5tvxh 1/1 Running 0 13m
calico-node-kffn7 1/1 Running 0 13m
calico-node-lltfs 1/1 Running 0 13m
calico-node-nhgn8 1/1 Running 0 13m
coredns-54d67798b7-8w5hd 1/1 Running 0 117m
coredns-54d67798b7-vb2ll 1/1 Running 0 117m
etcd-k8s-master01 1/1 Running 0 117m
etcd-k8s-master02 1/1 Running 0 104m
kube-apiserver-k8s-master01 1/1 Running 0 117m
kube-apiserver-k8s-master02 1/1 Running 0 104m
kube-controller-manager-k8s-master01 1/1 Running 1 117m
kube-controller-manager-k8s-master02 1/1 Running 0 104m
kube-proxy-5bws8 1/1 Running 0 117m
kube-proxy-pbqjc 1/1 Running 0 104m
kube-proxy-tpwbt 1/1 Running 0 86m
kube-proxy-vbpc5 1/1 Running 0 86m
kube-scheduler-k8s-master01 1/1 Running 1 117m
kube-scheduler-k8s-master02 1/1 Running 0 104m
metrics-server-545b8b99c6-hkgnz 1/1 Running 0 2m38s
九、Metrics部署
在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。
9.1、将Master01节点的front-proxy-ca.crt复制到所有Node节点（Master01上）
# 其他节点自行拷贝
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node01:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node02:/etc/kubernetes/pki/front-proxy-ca.crt
9.2、安装metrics server
[root@k8s-master01 calico]# cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm/
[root@k8s-master01 metrics-server-0.4.x-kubeadm]# kubectl create -f comp.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
9.3、查看状态
[root@k8s-master01 metrics-server-0.4.x-kubeadm]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master01 374m 18% 1158Mi 62%
k8s-master02 354m 17% 1027Mi 54%
k8s-master03 104m 2% 1082Mi 28%
k8s-node01 188m 9% 773Mi 41%
k8s-node02 446m 22% 803Mi 43
10、Dashboard部署
Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。
10.1、安装指定版本dashboard
[root@k8s-master01 metrics-server-0.4.x-kubeadm]# cd /root/k8s-ha-install/dashboard/
[root@k8s-master01 dashboard]# kubectl create -f .
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
# 查看所有容器状态
[root@k8s-master01 dashboard]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5f6d4b864b-khq4h 1/1 Running 0 16m
kube-system calico-node-5tvxh 1/1 Running 0 16m
kube-system calico-node-kffn7 1/1 Running 0 16m
kube-system calico-node-lltfs 1/1 Running 0 16m
kube-system calico-node-nhgn8 1/1 Running 0 16m
kube-system coredns-54d67798b7-8w5hd 1/1 Running 0 121m
kube-system coredns-54d67798b7-vb2ll 1/1 Running 0 121m
kube-system etcd-k8s-master01 1/1 Running 0 121m
kube-system etcd-k8s-master02 1/1 Running 0 107m
kube-system kube-apiserver-k8s-master01 1/1 Running 0 121m
kube-system kube-apiserver-k8s-master02 1/1 Running 0 107m
kube-system kube-controller-manager-k8s-master01 1/1 Running 1 121m
kube-system kube-controller-manager-k8s-master02 1/1 Running 0 107m
kube-system kube-proxy-5bws8 1/1 Running 0 121m
kube-system kube-proxy-pbqjc 1/1 Running 0 107m
kube-system kube-proxy-tpwbt 1/1 Running 0 90m
kube-system kube-proxy-vbpc5 1/1 Running 0 90m
kube-system kube-scheduler-k8s-master01 1/1 Running 1 121m
kube-system kube-scheduler-k8s-master02 1/1 Running 0 107m
kube-system metrics-server-545b8b99c6-hkgnz 1/1 Running 0 6m25s
kubernetes-dashboard dashboard-metrics-scraper-7645f69d8c-hftvd 1/1 Running 0 2m31s
kubernetes-dashboard kubernetes-dashboard-78cb679857-zksm9 1/1 Running 0 2m32s
10.2、更改dashboard的svc为NodePort
# 将type的ClusterIP更改为NodePort（如果已经为NodePort忽略此步骤）：
[root@k8s-master01 dashboard]# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
type: NodePort
10.3、登录dashboard
# 查看dashboard信息，端口映射
[root@k8s-master01 dashboard]# kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.111.114.182 <none> 443:32209/TCP 10m
# 页面访问，VIP:映射的端口
https://192.168.1.236:32209/
# 获取token
[root@k8s-master01 dashboard]# kubectl -n kube-system describe secret
十一、一些必须的配置更改
11.1、将Kube-proxy改为ipvs模式，因为在初始化集群的时候注释了ipvs配置，所以需要自行修改一下：（master01上）
# 查看原来的mode模式
[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
iptables
# 修改
[root@k8s-master01 ~]# kubectl edit cm kube-proxy -n kube-system
mode: "ipvs" # 44行
# 更新Kube-Proxy的Pod
[root@k8s-master01 ~]# kubectl patch daemonset kube-proxy -p "{"spec":{"template":{"metadata":{"annotations":{"date":"`date +'%s'`"}}}}}" -n kube-system
daemonset.apps/kube-proxy patched
# 再次查看mode模式
[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
ipvs
十二、注意事项
注意：kubeadm安装的集群，证书有效期默认是一年。master节点的kube-apiserver、kube-scheduler、kube-controller-manager、etcd都是以容器运行的。可以通过kubectl get po -n kube-system查看。
启动和二进制不同的是，
kubelet的配置文件在/etc/sysconfig/kubelet和/var/lib/kubelet/config.yaml
其他组件的配置文件在/etc/Kubernetes/manifests目录下，比如kube-apiserver.yaml，该yaml文件更改后，kubelet会自动刷新配置，也就是会重启pod。不能再次创建该文件
Kubeadm安装后，master节点默认不允许部署pod，可以通过以下方式打开：
# 查看Taints：
[root@k8s-master01 ~]# kubectl describe node -l node-role.kubernetes.io/master= | grep Taints
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
# 删除Taint：
[root@k8s-master01 ~]# kubectl taint node -l node-role.kubernetes.io/master node-role.kubernetes.io/master:NoSchedule-
node/k8s-master01 untainted
node/k8s-master02 untainted
node/k8s-master03 untainted
[root@k8s-master01 ~]# kubectl describe node -l node-role.kubernetes.io/master= | grep Taints
Taints: <none>
Taints: <none>
Taints: <none>