基于CentOS-7.6部署k8s-1.24.0，containerd作为CRI，nerdctl作为容器管理CLI
文章目录
- 1. 各节点基础环境准备
- 1.1. 关闭防火墙以及SELinux
- 1.2. 配置时间同步服务
- 1.3. 配置yum源
- 1.4. 配置系统装载的内核模块以及相关参数
- 1.5. 安装相应的软件包
- 2. master节点配置以及初始化
- 3. worker节点配置以及初始化
由于Kubernetes默认已经不支持docker作为容器运行时驱动了，倒不是完全不能使用docker作为CRI驱动，像1.21、1.20、1.19的时候，直接安装完docker并且启动docker服务之后，就可以执行初始化操作，新版的kubernetes-1.22以及后续的版本，无法直接这样使用了。
所以，此处直接使用containerd.io这个软件包提供的containerd作为CRI的驱动，这个软件包也会顺带安装crictl这个软件包，其中提供了crictl这个命令，这个命令也可以实现简单的镜像以及容器管理操作，但是功能不完善，比如无法给镜像打标签，也无法保存镜像以及重新装载归档的镜像。所以建议使用nerdctl命令作为容器和镜像的管理客户端。
1. 各节点基础环境准备
1.1. 关闭防火墙以及SELinux
需要将所有的节点都关闭防火墙以及SELinux。
关闭防火墙
[root@c7u6s17 ~]# systemctl disable --now firewalld [root@c7u6s17 ~]#关闭SELinux
[root@c7u6s17 ~]# sed -i.bak -re 's/^(SELINUX=).*/\1disabled/' /etc/selinux/config [root@c7u6s17 ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
1.2. 配置时间同步服务
除此之外，还要保证各个节点之间的时间保持同步。在各个集群上分别启动时间同步服务chronyd即可。如果某个节点不能自动完成时间同步，则可以手动指定集群中某个节点作为时间同步服务器，并将其他待同步节点的chronyd配置文件中指定该节点为时间同步服务器。
关于时间同步的示例如下：
[root@c7u6s17 ~]# systemctl status chronyd ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled) Active: active (running) since 五 2022-09-02 22:33:30 CST; 1 day 17h agoDocs: man:chronyd(8)man:chrony.conf(5) Main PID: 2845 (chronyd) CGroup: /system.slice/chronyd.service└─2845 /usr/sbin/chronyd9月 03 18:12:20 c7u6s17 chronyd[2845]: Source 108.59.2.24 replaced with 119.28.183.184 9月 03 20:21:25 c7u6s17 chronyd[2845]: Can't synchronise: no selectable sources 9月 03 23:39:53 c7u6s17 chronyd[2845]: Selected source 119.28.183.184 9月 03 23:56:33 c7u6s17 chronyd[2845]: Selected source 144.76.76.107 9月 04 08:57:15 c7u6s17 chronyd[2845]: Forward time jump detected! 9月 04 08:57:15 c7u6s17 chronyd[2845]: Can't synchronise: no selectable sources 9月 04 09:00:30 c7u6s17 chronyd[2845]: Selected source 84.16.73.33 9月 04 09:01:36 c7u6s17 chronyd[2845]: Source 144.76.76.107 replaced with 193.182.111.143 9月 04 09:05:54 c7u6s17 chronyd[2845]: Selected source 119.28.183.184 9月 04 13:50:36 c7u6s17 chronyd[2845]: Can't synchronise: no selectable sources Hint: Some lines were ellipsized, use -l to show in full. [root@c7u6s17 ~]#配置文件为：
[root@c7u6s17 ~]# vim /etc/chrony.conf # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). server 0.centos.pool.ntp.org iburst server 1.centos.pool.ntp.org iburst server 2.centos.pool.ntp.org iburst server 3.centos.pool.ntp.org iburst# Record the rate at which the system clock gains/losses time. driftfile /var/lib/chrony/drift# Allow the system clock to be stepped in the first three updates # if its offset is larger than 1 second. makestep 1.0 3# Enable kernel synchronization of the real-time clock (RTC). rtcsync# Enable hardware timestamping on all interfaces that support it. #hwtimestamp *# Increase the minimum number of selectable sources required to adjust # the system clock. #minsources 2# Allow NTP client access from local network. #allow 192.168.0.0/16# Serve time even if not synchronized to a time source. #local stratum 10# Specify file containing keys for NTP authentication. #keyfile /etc/chrony.keys# Specify directory for log files. logdir /var/log/chrony# Select which information is logged. #log measurements statistics tracking查看时间同步状态，具体如下：
[root@c7u6s17 ~]# chronyc tracking Reference ID : 771CB7B8 (119.28.183.184) Stratum : 3 Ref time (UTC) : Sun Sep 04 02:57:45 2022 System time : 0.000623874 seconds slow of NTP time Last offset : +0.000213936 seconds RMS offset : 0.015058505 seconds Frequency : 0.713 ppm slow Residual freq : -0.014 ppm Skew : 3.080 ppm Root delay : 0.083240315 seconds Root dispersion : 0.083911069 seconds Update interval : 1030.7 seconds Leap status : Normal [root@c7u6s17 ~]# [root@c7u6s17 ~]# chronyc sources 210 Number of sources = 4 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^? 119.28.183.184 2 10 0 279m -8114us[-7900us] +/- 66ms ^? 162.159.200.123 3 10 0 271m +24ms[ +24ms] +/- 197ms ^? 84.16.73.33 1 10 0 280m -4978us[-4760us] +/- 109ms ^? 193.182.111.143 2 10 0 274m -15ms[ -15ms] +/- 279ms [root@c7u6s17 ~]#
1.3. 配置yum源
接下来需要准备yum源，用到的源包括：epel.repo, kubernetes.repo, docker-ce.repo。各个配置文件的内容分别如下：
epel.repo镜像源
[root@c7u6s17 ~]# cat /etc/yum.repos.d/epel.repo [epel] name=Extra Packages for Enterprise Linux 7 - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch failovermethod=priority enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7[epel-debuginfo] name=Extra Packages for Enterprise Linux 7 - $basearch - Debug #baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=1[epel-source] name=Extra Packages for Enterprise Linux 7 - $basearch - Source #baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=1 [root@c7u6s17 ~]#kubernetes.reop镜像源
[root@c7u6s17 ~]# cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 #repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg [root@c7u6s17 ~]#docker-ce.repo镜像源
[root@c7u6s17 ~]# cat /etc/yum.repos.d/docker-ce.repo [docker-ce-stable] name=Docker CE Stable - $basearch baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable enabled=1 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg[docker-ce-stable-debuginfo] name=Docker CE Stable - Debuginfo $basearch baseurl=https://download.docker.com/linux/centos/$releasever/debug-$basearch/stable enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg[docker-ce-stable-source] name=Docker CE Stable - Sources baseurl=https://download.docker.com/linux/centos/$releasever/source/stable enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg[docker-ce-test] name=Docker CE Test - $basearch baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/test enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg
1.4. 配置系统装载的内核模块以及相关参数
需要装载相应的模块，并且指定相应的内核参数，具体如下：
指定内核模块
[root@c7u6s17 ~]# cat /etc/modules-load.d/k8s.conf overlay br_netfilter [root@c7u6s17 ~]# [root@c7u6s17 ~]# modprobe overlay [root@c7u6s17 ~]# modprobe br_netfilter [root@c7u6s17 ~]# modinfo overlay filename: /lib/modules/3.10.0-957.el7.x86_64/kernel/fs/overlayfs/overlay.ko.xz alias: fs-overlay license: GPL description: Overlay filesystem author: Miklos Szeredi <miklos@szeredi.hu> retpoline: Y rhelversion: 7.6 srcversion: 1CF79904F61F2BFDF6F89EC depends: intree: Y vermagic: 3.10.0-957.el7.x86_64 SMP mod_unload modversions signer: CentOS Linux kernel signing key sig_key: B7:0D:CF:0D:F2:D9:B7:F2:91:59:24:82:49:FD:6F:E8:7B:78:14:27 sig_hashalgo: sha256 parm: check_copy_up:uint parm: ovl_check_copy_up:Warn on copy-up when causing process also has a R/O fd open parm: redirect_max:ushort parm: ovl_redirect_max:Maximum length of absolute redirect xattr value parm: redirect_dir:bool parm: ovl_redirect_dir_def:Default to on or off for the redirect_dir feature parm: redirect_always_follow:bool parm: ovl_redirect_always_follow:Follow redirects even if redirect_dir feature is turned off parm: index:bool parm: ovl_index_def:Default to on or off for the inodes index feature [root@c7u6s17 ~]# [root@c7u6s17 ~]# modinfo br_netfilter filename: /lib/modules/3.10.0-957.el7.x86_64/kernel/net/bridge/br_netfilter.ko.xz description: Linux ethernet netfilter firewall bridge author: Bart De Schuymer <bdschuym@pandora.be> author: Lennert Buytenhek <buytenh@gnu.org> license: GPL retpoline: Y rhelversion: 7.6 srcversion: C4DE536495D55C12BA6A8A8 depends: bridge intree: Y vermagic: 3.10.0-957.el7.x86_64 SMP mod_unload modversions signer: CentOS Linux kernel signing key sig_key: B7:0D:CF:0D:F2:D9:B7:F2:91:59:24:82:49:FD:6F:E8:7B:78:14:27 sig_hashalgo: sha256 [root@c7u6s17 ~]#配置内核参数
[root@c7u6s17 ~]# cat /etc/sysctl.d/90-k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 [root@c7u6s17 ~]# [root@c7u6s17 ~]# sysctl --system * Applying /usr/lib/sysctl.d/00-system.conf ... net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... kernel.yama.ptrace_scope = 0 * Applying /usr/lib/sysctl.d/50-default.conf ... kernel.sysrq = 16 kernel.core_uses_pid = 1 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.default.promote_secondaries = 1 net.ipv4.conf.all.promote_secondaries = 1 fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /etc/sysctl.d/90-k8s.conf ... net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 * Applying /etc/sysctl.d/99-sysctl.conf ... * Applying /etc/sysctl.conf ... [root@c7u6s17 ~]#如果不指定上述内核参数，在执行
kubeadm init命令初始化集群的时候，会有相关的错误提示。
1.5. 安装相应的软件包
无论是worker节点还是master节点，都需要安装如下软件包：
kubeadm, kubectl, kubelet，由于此处使用containerd作为容器运行时驱动，所以还需要安装containerd.io这个软件包。默认不指定版本的时候，都是安装仓库中的最新版。下面的安装中，kubelet, kubeadm, kubectl会采用指定版本的方式进行安装。
[root@c7u6s18 ~]# rpmkeys --import https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg [root@c7u6s18 ~]# rpmkeys --import https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg [root@c7u6s18 ~]# vim /etc/yum.repos.d/kubernetes.repo [root@c7u6s18 ~]# yum install -y kubeadm-1.24.0 kubectl-1.24.0 kubelet-1.24.0 Installed: kubeadm.x86_64 0:1.24.0-0 kubectl.x86_64 0:1.24.0-0 kubelet.x86_64 0:1.24.0-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-7.el7 cri-tools.x86_64 0:1.24.2-0 kubernetes-cni.x86_64 0:0.8.7-0 libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@c7u6s18 ~]# yum install -y containerd.io当完成containerd.io软件包的安装之后，需要修改containerd的配置文件，具体如下：
[root@c7u6s17 ~]# cat /etc/containerd/config.toml # Copyright 2018-2022 Docker Inc. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.#disabled_plugins = ["cri"] # 这一行默认是开启的，表示禁用cri插件，必须将这一行注释掉。#root = "/var/lib/containerd" #state = "/run/containerd" #subreaper = true #oom_score = 0#[grpc] # address = "/run/containerd/containerd.sock" # 套接字文件路径 # uid = 0 # gid = 0#[debug] # address = "/run/containerd/debug.sock" # uid = 0 # gid = 0 # level = "info" [plugins] [plugins."io.containerd.grpc.v1.cri"]sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7" # 这一行指定了也没起到应有的效果 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]SystemdCgroup = true # 指定cri运行时驱动使用systemd作为控制组管理工具[plugins."io.containerd.grpc.v1.cri".containerd]snapshotter = "overlayfs" [root@c7u6s17 ~]# # 启动containerd服务 [root@c7u6s17 ~]# systemctl start containerd [root@c7u6s17 ~]# systemctl status containerd ● containerd.service - containerd container runtime Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled) Active: active (running) since 六 2022-09-03 17:41:53 CST; 22h ago Docs: https://containerd.io Main PID: 30883 (containerd) CGroup: /system.slice/containerd.service ├─ 1137 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 58f22581d9248...├─ 1151 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id f25a8e5492a11...├─ 1187 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 157aa7b74cdbd... ├─ 1211 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 23d0eb7630c2b...├─ 1550 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 93b439ff22ece...├─22340 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1d12eb5d8f1ac... ├─22739 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1dcecf7ad418b...├─22741 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id bb569e9634ed3...├─30883 /usr/bin/containerd ├─kubepods-burstable-pod91c8cd68211465eb88ba2431bd893902.slice:cri-containerd:8 626d4555b28951383de87dee0cd72772526b7aaeb175de913e6d2032139e646│ └─23707 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/schedul... ├─kubepods-burstable-pod106fa15975eff62f04fe698d27a8a2a9.slice:cri-containerd:5 380c46f899c0d81496c14980869bd87446b0d172530436f0234ee1cdf7f6e3d│ └─23687 kube-controller-manager --allocate-node-cidrs=true --authenticatio... ├─kubepods-burstable-podce8839ae_1457_4118_ae04_8a3644e29f79.slice:cri-containe rd:f9533554de1548b1e04ec6dc4476b9213fd9f8ebd996d73c5f503af0ba4d42e5│ └─22855 /coredns -conf /etc/coredns/Corefile ├─kubepods-burstable-podbab2c94c_d2d3_486a_a59a_f0dfaf4a0e60.slice:cri-containe rd:7d0f0652cd4c716be7e38f1f52c7efe8ad02bfaa4b2de5166fa768188c3bb573│ └─22832 /coredns -conf /etc/coredns/Corefile此外，也需要修改kubelet的配置文件，具体如下：
[root@c7u6s17 ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS=--cgroup-driver=systemd [root@c7u6s17 ~]#
由于此处没有安装docker服务以及docker-cli，所以也就导致无法使用docker命令管理容器以及镜像，配合containerd这个CRI一起用于管理容器和镜像的命令行工具为nerdctl，以及实验环境的crictl，建议使用nerdctl这个命令行工具，功能上基本可以完全取代docker命令行工具。其安装部署比较简单，具体如下：
具体参见https://github.com/containerd/nerdctl，下载好release之后，将其解压到/usr/local/目录下，并将这个目录添加到PATH环境变量。
[root@c7u6s18 ~]# mkdir nerdctl-0.22.2 [root@c7u6s18 ~]# tar zxvf nerdctl-0.22.2-linux-amd64.tar.gz -C nerdctl-0.22.2 [root@c7u6s18 ~]# mv nerdctl-0.22.2 /usr/local/ [root@c7u6s18 ~]# cd /usr/local/nerdctl-0.22.2/ [root@c7u6s18 /usr/local/nerdctl-0.22.2/]# pwd >> /etc/profile.d/nerdctl.sh [root@c7u6s18 ~]# . /etc/profile [root@c7u6s18 ~]# which nerdctl /usr/local/nerdctl-0.22.2/nerdctl [root@c7u6s18 ~]#由于上述命令需要跟containerd的套接字文件进行通信，所以还需要通过该命令的
-a选项指定该套接字的路径。通过下面的方式，指定了别名之后，就不需要总是在命令行中手动指定套接字文件的具体路径了。具体如下：[root@c7u6s19 nerdctl-0.22.2]# cat /etc/profile.d/nerdctl.sh export PATH=/usr/local/nerdctl-0.22.2:$PATH alias nerdctl='nerdctl -a unix:///var/run/containerd/containerd.sock' [root@c7u6s19 nerdctl-0.22.2]# . /etc/profile [root@c7u6s19 nerdctl-0.22.2]# which nerdctl alias nerdctl='nerdctl -a unix:///var/run/containerd/containerd.sock'/usr/local/nerdctl-0.22.2/nerdctl [root@c7u6s19 nerdctl-0.22.2]# nerdctl ns ls NAME CONTAINERS IMAGES VOLUMES LABELS [root@c7u6s19 nerdctl-0.22.2]#nerdctl中也使用了名称空间的概念，当执行了
kubeadm init命令以及kubeadm join命令之后，会自动创建一个名为k8s.io的名称空间。此时由于还没有初始化，所以还没有名称空间。
除此之外，还需要准备flanne网络插件，需要在所有的节点上的/opt/bin/这个目录下，放上编译好的flanneld这个插件。关于这个插件的下载地址，以及安装说明，参见https://github.com/flannel-io/flannel。
下载好flanneld插件之后，将其放到/opt/bin/这个目录中。
[root@c7u6s17 flannel]# mkdir /opt/bin [root@c7u6s17 flannel]# cp flanneld-amd64 /opt/bin/flanneld这一步，在所有节点上，都需要执行。
2. master节点配置以及初始化
做完上述的操作之后，就可以着手在master节点上执行初始化操作了。由于执行初始化的时候，会去拉取镜像，所以为了加快初始化的过程，可以提前将镜像拉取下来。
拉取镜像：
[root@c7u6s17 ~]# kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers [root@c7u6s17 ~]# nerdctl ns ls # 查看名称空间信息 NAME CONTAINERS IMAGES VOLUMES LABELS k8s.io 20 43 0 [root@c7u6s17 ~]#上述输出中显示，已经包含了43个镜像，运行了20个容器。当然，上述是已经完成了集群部署之后的状态。
查看下拉取了那些镜像：
[root@c7u6s17 ~]# nerdctl -n k8s.io image ls | egrep aliyun | egrep -v none registry.aliyuncs.com/google_containers/coredns v1.8.6 5b6ec0d6de9b 38 hours ago linux/amd64 44.7 MiB 13.0 MiB registry.aliyuncs.com/google_containers/etcd 3.5.3-0 13f53ed1d91e 38 hours ago linux/amd64 288.9 MiB 97.4 MiB registry.aliyuncs.com/google_containers/kube-apiserver v1.24.0 a04522b882e9 39 hours ago linux/amd64 127.0 MiB 32.2 MiB registry.aliyuncs.com/google_containers/kube-apiserver v1.24.4 74496d788bad 38 hours ago linux/amd64 127.0 MiB 32.2 MiB registry.aliyuncs.com/google_containers/kube-controller-manager v1.24.0 df044a154e79 39 hours ago linux/amd64 117.1 MiB 29.6 MiB registry.aliyuncs.com/google_containers/kube-controller-manager v1.24.4 f9400b11d780 38 hours ago linux/amd64 117.2 MiB 29.6 MiB registry.aliyuncs.com/google_containers/kube-proxy v1.24.0 c957d602267f 39 hours ago linux/amd64 110.1 MiB 37.7 MiB registry.aliyuncs.com/google_containers/kube-proxy v1.24.4 64a04a34b31f 38 hours ago linux/amd64 110.1 MiB 37.7 MiB registry.aliyuncs.com/google_containers/kube-scheduler v1.24.0 db842a7c431f 39 hours ago linux/amd64 51.9 MiB 14.8 MiB registry.aliyuncs.com/google_containers/kube-scheduler v1.24.4 378509dd1111 38 hours ago linux/amd64 51.9 MiB 14.8 MiB registry.aliyuncs.com/google_containers/pause 3.6 3d380ca88645 22 hours ago linux/amd64 668.0 KiB 294.7 KiB registry.aliyuncs.com/google_containers/pause 3.7 bb6ed397957e 39 hours ago linux/amd64 696.0 KiB 304.0 KiB [root@c7u6s17 ~]#
接下来执行初始化操作。
具体如下：
[root@c7u6s17 ~]# kubeadm config print init-defaults > kubeadm-init.yml [root@c7u6s17 ~]# vim kubeadm-init.yml [root@c7u6s17 ~]# kubeadm init --config kubeadm-init.yml [init] Using Kubernetes version: v1.24.0 [preflight] Running pre-flight checks[WARNING Hostname]: hostname "node" could not be reached[WARNING Hostname]: hostname "node": lookup node on 192.168.122.1:53: server misbehaving[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' error execution phase preflight: [preflight] Some fatal errors occurred:[ERROR CRI]: container runtime is not running: output: E0903 01:25:16.677810 11256 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" time="2022-09-03T01:25:16+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher [root@c7u6s17 ~]# systemctl enable kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@c7u6s17 ~]#上述的初始化过程总是出错，查看/var/log/messages文件内容，其中包含的最多u多的错误记录如下：
Sep 3 15:24:53 c7u6s17 kubelet: E0903 15:24:53.807340 17248 kubelet.go:2419] "Error getting node" err="node \"c7u6s17\" not found"再有就是关于sandbox_image无法拉取的问题，具体如下：
Sep 3 15:27:23 c7u6s17 kubelet: E0903 15:27:23.895185 17248 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"k8s.gcr.io/pause:3.6\": failed to pull image \"k8s.gcr.io/pause:3.6\": failed to pull and unpack image \"k8s.gcr.io/pause:3.6\": failed to resolve reference \"k8s.gcr.io/pause:3.6\": failed to do request: Head \"https://k8s.gcr.io/v2/pause/manifests/3.6\": dial tcp: lookup k8s.gcr.io on 192.168.122.1:53: server misbehaving" pod="kube-system/kube-scheduler-c7u6s17"上述提示无法拉取k8s.gcr.io/pause:3.6这个镜像，由于部署的时候，指定的是阿里云的镜像仓库，所以下载的镜像标签都是阿里云的，而不是k8s.gcr.io。所以需要将下载的镜像重新打标签。
由于使用的containerd作为CRI，所以并没有启动docker服务，也就无法使用docker命令管理镜像以及容器，但是与containerd配对的应用有两个：crictl以及nerdctl，前者为debug用的命令，所以支持的功能和操作都比较有限；后者支持的命令和操作则更多一些。比如下面的给镜像重新打标签的操作，crictl命令就不支持，但是nerdctl命令就支持。重新打标签的操作如下：
[root@c7u6s17 ~]# nerdctl -n k8s.io tag registry.aliyuncs.com/google_containers/pause:3.6 k8s.gcr.io/pause:3.6 [root@c7u6s17 ~]# nerdctl -n k8s.io image ls | egrep pause k8s.gcr.io/pause 3.6 3d380ca8864520 seconds ago linux/amd64 668.0 KiB 294.7 KiB registry.aliyuncs.com/google_containers/pause 3.6 3d380ca8864513 minutes ago linux/amd64 668.0 KiB 294.7 KiB registry.aliyuncs.com/google_containers/pause 3.7 bb6ed397957e16 hours ago linux/amd64 696.0 KiB 304.0 KiB [root@c7u6s17 ~]#做完上述修改之后，重新执行初始化操作：
[root@c7u6s17 ~]# kubeadm init --config kubeadm-init.yml Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.122.36:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:2cb5d1ff4f2e6eaf42862a68f42147f84d5383ea7f500a317708e7048ede8d19至此， master节点的初始化操作就完成了。
接下来执行拷贝配置文件的操作
[root@c7u6s17 ~]# kubectl get nodes The connection to the server localhost:8080 was refused - did you specify the right host or port? [root@c7u6s17 ~]# ls -A anaconda-ks.cfg cni-plugins-linux-amd64-v1.1.1.tgz original-ks.cfg .ansible .config .pki .bash_history cri-dockerd-0.2.5-3.el7.x86_64.rpm runc.amd64 .bash_logout .cshrc .ssh .bash_profile ks.post.log .tcshrc .bashrc kubeadm-init.yml .viminfo .cache nerdctl-0.22.2-linux-amd64.tar.gz .vimrc [root@c7u6s17 ~]# mkdir .kube [root@c7u6s17 ~]# cp -i /etc/kubernetes/admin.conf .kube/config [root@c7u6s17 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION c7u6s17 NotReady control-plane 59s v1.24.0 [root@c7u6s17 ~]#此时，master节点还没有处于Ready状态，因为还没有安装合适的网络插件。
安装部署Flannel插件，具体如下：
下载flannel.yml文件，参见链接：https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
下载完成之后，基本不需要做任何修改，执行执行即可。
[root@c7u6s17 flannel]# ls flanneld-amd64 kube-flannel.yml [root@c7u6s17 flannel]# kubectl apply -f kube-flannel.yml namespace/kube-flannel created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created [root@c7u6s17 flannel]# kubectl get po No resources found in default namespace. [root@c7u6s17 flannel]# kubectl get ns NAME STATUS AGE default Active 6h6m kube-flannel Active 8s kube-node-lease Active 6h6m kube-public Active 6h6m kube-system Active 6h6m [root@c7u6s17 flannel]# kubectl get po -n kube-flannel NAME READY STATUS RESTARTS AGE kube-flannel-ds-rrslz 0/1 Init:1/2 0 29s [root@c7u6s17 flannel]# kubectl get all -n kube-flannel NAME READY STATUS RESTARTS AGE pod/kube-flannel-ds-rrslz 0/1 Init:1/2 0 45sNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/kube-flannel-ds 1 1 0 1 0 <none> 45s [root@c7u6s17 flannel]# kubectl get -n kube-flannel ds NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-flannel-ds 1 1 1 1 1 <none> 65s [root@c7u6s17 flannel]# kubectl get -n kube-flannel po NAME READY STATUS RESTARTS AGE kube-flannel-ds-rrslz 1/1 Running 0 72s [root@c7u6s17 flannel]# kubectl get nodes NAME STATUS ROLES AGE VERSION c7u6s17 Ready control-plane 6h7m v1.24.0 [root@c7u6s17 flannel]#等flannel的pod处于运行状态之后，master节点就处于Ready状态了。
为了免去在其他worker节点上重复下载镜像的操作，此时将master节点上的镜像归档保存为tar文件，然后分发给所有的worker节点。
具体如下：
[root@c7u6s17 ~]# images=`nerdctl -n k8s.io images | egrep 'gcr|aliyun' | gawk 'BEGIN{OFS=":"} {print $1,$2}' | egrep -v none | tr '\n' ' '` [root@c7u6s17 ~]# nerdctl -n k8s.io save -o k8s.images.tar $images [root@c7u6s17 ~]# ls -lFh k8s.images.tar -rw-r--r-- 1 root root 308M 9月 4 09:35 k8s.images.tar [root@c7u6s17 ~]#将这个镜像包分发到所有其他的节点上。
很重要的一步，就是需要将master节点配置为到所有其他worker节点可以通过无密码非交互式验证。具体如下：
[root@c7u6s17 ~]# ssh-keygen -t rsa -f ~/.ssh/id_rsa -P '' [root@c7u6s17 ~]# for vm in c7u6s{18,19,20}; do ssh-copy-id -o StrictHostKeyChecking=no $vm; done做完上述操作之后，就可以在将镜像包分发给所有的worker节点了。具体如下：
[root@c7u6s17 ~]# for vm in c7u6s{18,19,20}; do scp k8s.images.tar $vm:/root; done至此，就无需在所有节点上重复下载相应的镜像包了。
3. worker节点配置以及初始化
由于上述在所有节点上都执行了基础操作，所此时在worker节点上，只需要执行kubeadm join命令加入节点，然后检查其上的容器运行情况即可。
加入第一个worker节点
[root@c7u6s18 ~]# kubeadm join 192.168.122.36:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:2cb5d1ff4f2e6eaf42862a68f42147f84d5383ea7f500a317708e7048ede8d19 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.[root@c7u6s18 ~]#执行玩上述操作之后，kubelet会启动相应的静态pod资源。为此，就需要在该节点上有镜像。导入此前从master节点分发过来的镜像包。
[root@c7u6s18 ~]# nerdctl -n k8s.io load -i k8s.images.tar稍等一会儿即可看到该节点处于Ready状态。
[root@c7u6s18 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4c79a973e716 docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2 "/opt/bin/flanneld -…" 7 hours ago Up k8s://kube-flannel/kube-flannel-ds-xcg5n/kube-flannel 8c72db152c03 registry.aliyuncs.com/google_containers/kube-proxy:v1.24.0 "/usr/local/bin/kube…" 7 hours ago Up k8s://kube-system/kube-proxy-stb76/kube-proxy a9e12b69c7f2 k8s.gcr.io/pause:3.6 "/pause" 7 hours ago Up k8s://kube-system/kube-proxy-stb76 dc38a535bce9 k8s.gcr.io/pause:3.6 "/pause" 7 hours ago Up k8s://kube-flannel/kube-flannel-ds-xcg5n [root@c7u6s18 ~]#上述的几个容器都处于运行状态之后，worker节点就加入完成了。
加入第二个节点：
同样的命令：
[root@c7u6s19 ~]# kubeadm join 192.168.122.36:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:2cb5d1ff4f2e6eaf42862a68f42147f84d5383ea7f500a317708e7048ede8d19 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.[root@c7u6s19 ~]# nerdctl ns ls NAME CONTAINERS IMAGES VOLUMES LABELS k8s.io 0 0 0 [root@c7u6s19 ~]# nerdctl -n k8s.io load -i k8s.images.tar 导入完镜像之后，等一会儿，需要启动相关的容器 [root@c7u6s19 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a6abd316f02b registry.aliyuncs.com/google_containers/kube-proxy:v1.24.0 "/usr/local/bin/kube…" 54 seconds ago Up k8s://kube-system/kube-proxy-5pbqj/kube-proxy ba2e9960edec k8s.gcr.io/pause:3.6 "/pause" About a minute ago Up k8s://kube-flannel/kube-flannel-ds-9p78g cb4dea15a4db k8s.gcr.io/pause:3.6 "/pause" 55 seconds ago Up k8s://kube-system/kube-proxy-5pbqj [root@c7u6s19 ~]# [root@c7u6s17 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION c7u6s17 Ready control-plane 16h v1.24.0 c7u6s18 Ready <none> 21m v1.24.0 c7u6s19 NotReady <none> 58s v1.24.0 [root@c7u6s17 ~]#[root@c7u6s19 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7b92920f4eed docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2 "/opt/bin/flanneld -…" 5 seconds ago Up k8s://kube-flannel/kube-flannel-ds-9p78g/kube-flannel a6abd316f02b registry.aliyuncs.com/google_containers/kube-proxy:v1.24.0 "/usr/local/bin/kube…" 2 minutes ago Up k8s://kube-system/kube-proxy-5pbqj/kube-proxy ba2e9960edec k8s.gcr.io/pause:3.6 "/pause" 2 minutes ago Up k8s://kube-flannel/kube-flannel-ds-9p78g cb4dea15a4db k8s.gcr.io/pause:3.6 "/pause" 2 minutes ago Up k8s://kube-system/kube-proxy-5pbqj [root@c7u6s19 ~]# root@c7u6s17 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION c7u6s17 Ready control-plane 16h v1.24.0 c7u6s18 Ready <none> 24m v1.24.0 c7u6s19 Ready <none> 3m43s v1.24.0 [root@c7u6s17 ~]#此时，加入的两个worker节点都处于Ready状态了。
加入第三个节点：
具体如下：
[root@c7u6s20 kube-packs]# systemctl enable kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@c7u6s20 kube-packs]# systemctl enable containerd Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /usr/lib/systemd/system/containerd.service. [root@c7u6s20 kube-packs]# [root@c7u6s20 ~]# kubeadm join 192.168.122.36:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:2cb5d1ff4f2e6eaf42862a68f42147f84d5383ea7f500a317708e7048ede8d19 # 执行晚上述命令之后，还需要将此前master节点分发过来的镜像包导入 [root@c7u6s20 ~]# nerdctl -n k8s.io load -i k8s.images.tar [root@c7u6s20 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@c7u6s20 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 133ade108261 k8s.gcr.io/pause:3.6 "/pause" 1 second ago Up k8s://kube-flannel/kube-flannel-ds-w8jlm [root@c7u6s20 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 133ade108261 k8s.gcr.io/pause:3.6 "/pause" 5 seconds ago Up k8s://kube-flannel/kube-flannel-ds-w8jlm 974ce6970008 k8s.gcr.io/pause:3.6 "/pause" 1 second ago Up k8s://kube-system/kube-proxy-xjws8 aa8e4351fb51 registry.aliyuncs.com/google_containers/kube-proxy:v1.24.0 "/usr/local/bin/kube…" 1 second ago Up k8s://kube-system/kube-proxy-xjws8/kube-proxy [root@c7u6s20 ~]# nerdctl -n k8s.io container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 133ade108261 k8s.gcr.io/pause:3.6 "/pause" 8 seconds ago Up k8s://kube-flannel/kube-flannel-ds-w8jlm 974ce6970008 k8s.gcr.io/pause:3.6 "/pause" 4 seconds ago Up k8s://kube-system/kube-proxy-xjws8 aa8e4351fb51 registry.aliyuncs.com/google_containers/kube-proxy:v1.24.0 "/usr/local/bin/kube…" 4 seconds ago Up k8s://kube-system/kube-proxy-xjws8/kube-proxy [root@c7u6s20 ~]# [root@c7u6s17 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION c7u6s17 Ready control-plane 16h v1.24.0 c7u6s18 Ready <none> 33m v1.24.0 c7u6s19 Ready <none> 12m v1.24.0 c7u6s20 Ready <none> 97s v1.24.0 [root@c7u6s17 ~]#至此，包含1个master节点，3个worker节点的k8s集群就部署完成了。