1. 轻量级容器管理工具 Containerd
2. Containerd的两种安装方式
3. Containerd容器镜像管理
4. Containerd数据持久化和网络管理

1、Containerd镜像管理

1.1 Containerd容器镜像管理命令

• docker使用docker images命令管理镜像
• 单机containerd使用ctr images命令管理镜像,containerd本身的CLI
• k8s中containerd使用crictl images命令管理镜像,Kubernetes社区的专用CLI工具

获取命令帮助
# ctr --help
NAME:ctr -_______/ /______/ ___/ __/ ___/
/ /__/ /_/ /
\___/\__/_/......

获取命令帮助
# ctr images
NAME:ctr images - manage imagesUSAGE:ctr images command [command options] [arguments...]COMMANDS:check                    check existing images to ensure all content is available locallyexport                   export imagesimport                   import imageslist, ls                 list images known to containerdmount                    mount an image to a target pathunmount                  unmount the image from the targetpull                     pull an image from a remotepush                     push an image to a remotedelete, del, remove, rm  remove one or more images by referencetag                      tag an imagelabel                    set and clear labels for an imageconvert                  convert an imageOPTIONS:--help, -h  show help

1.2 查看镜像

# ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS

1.3 下载镜像

containerd支持oci标准的镜像，所以可以直接使用docker官方或dockerfile构建的镜像

# 镜像名不能简写，--all-platforms：所有平台，不加的话下载当前平台架构
# ctr images pull --all-platforms docker.io/library/nginx:alpine
# ctr images pull docker.io/library/nginx:latest# uname -a

说明：
这里ctr命令pull镜像时，不能直接把镜像名字写成`nginx:alpine`

查看已下载容器镜像
# ctr images ls

REF	TYPE	DIGEST
docker.io/library/nginx:alpine	application/vnd.docker.distribution.manifest.list.v2+json	sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3

SIZE	PLATFORMS	LABELS
9.7 MiB	linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x	-

指定平台下载容器镜像
# ctr images pull --platform linux/amd64 docker.io/library/nginx:alpine

没有镜像是无法运行容器的。

1.4 镜像挂载

方便查看镜像中包含的内容。

把已下载的容器镜像挂载至当前文件系统
# ctr images mount docker.io/library/nginx:alpine /mnt
sha256:af2fcce448e2e4451a5f4796a9bf9cb5c9b5f88e0d6d10029cada42fb9d268ac
/mnt
[root@localhost ~]# ls /mnt
bin  dev  docker-entrypoint.d  docker-entrypoint.sh  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

卸载
# umount /mnt

1.5 镜像导出

把容器镜像导出
# ctr i export --all-platforms nginx.img docker.io/library/nginx:alpine

说明
--all-platforms,导出所有平台镜像，本版本为1.6版本，1.4版本不需要添加此选项。

查看已导出容器镜像
# ls
nginx.img# ls -lh
总用量 196M-rw-r--r--  1 root root  73M 2月  18 14:48 nginx.img

1.6 镜像删除

删除指定容器镜像
# ctr image rm docker.io/library/nginx:alpine
docker.io/library/nginx:alpine再次查看容器镜像
[root@192 ~]# ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS

1.7 镜像导入

导入容器镜像
# ctr images import nginx.img
unpacking docker.io/library/nginx:alpine (sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3)...done

1.8 修改镜像tag

# ctr images tag docker.io/library/nginx:alpine nginx:alpine
nginx:alpine# 建议修改tag的名称为：镜像仓库地址/目录/镜像名

说明：
把docker.io/library/nginx:alpine 修改为 nginx:alpine

查看修改后的容器镜像
# ctr images ls
REF                            TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS
docker.io/library/nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 9.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
nginx:alpine                   application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 9.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -

修改后对容器镜像做检查比对
# ctr images check
REF                            TYPE                                                      DIGEST                                                                  STATUS         SIZE            UNPACKED
docker.io/library/nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 complete (7/7) 9.7 MiB/9.7 MiB truenginx:alpine                   application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 complete (7/7) 9.7 MiB/9.7 MiB true

2、Containerd容器管理

2.1 获取命令帮助

2.1.1 获取ctr命令帮助

[root@localhost ~]# ctr --help
NAME:ctr -_______/ /______/ ___/ __/ ___/
/ /__/ /_/ /
\___/\__/_/containerd CLI......

2.1.2 获取创建静态容器命令帮助

# ctr container --help
NAME:ctr containers - manage containersUSAGE:ctr containers command [command options] [arguments...]COMMANDS:create                   create containerdelete, del, remove, rm  delete one or more existing containersinfo                     get info about a containerlist, ls                 list containerslabel                    set and clear labels for a containercheckpoint               checkpoint a containerrestore                  restore a container from checkpointOPTIONS:--help, -h  show help

说明：使用`ctr container create `命令创建容器后，容器并没有处于运行状态，其只是一个静态的容器。这个 container 对象只是包含了运行一个容器所需的资源及配置的数据结构，例如： namespaces、rootfs 和容器的配置都已经初始化成功了，只是用户进程(本案例为nginx)还没有启动。需要使用`ctr tasks`命令才能获取一个动态容器。

2.1.3 获取动态容器命令帮助

# ctr run --help
NAME:ctr run - run a container......

说明：
使用`ctr run`命令可以创建一个静态容器并使其运行。一步到位运行容器。

2.2 查看容器

container表示静态容器，可用c缩写代表container

# ctr container ls
CONTAINER    IMAGE    RUNTIME

或

# ctr c ls
CONTAINER    IMAGE    RUNTIME

以上命令无法查看出来容器的静态的还是动态的

2.3 查看任务

task表示容器里跑的进程, 可用t缩写代表task

# ctr task ls
TASK    PID    STATUS

或

# ctr t ls
TASK    PID    STATUS

2.4 创建静态容器

# ctr c create docker.io/library/nginx:alpine nginx1

# ctr container ls
CONTAINER    IMAGE                             RUNTIME
nginx1       docker.io/library/nginx:alpine    io.containerd.runc.v2

查看容器详细信息
# ctr container info nginx1

2.5 静态容器启动为动态容器

复制containerd连接runC垫片工具至系统
# ls usr/local/bin/
containerd  containerd-shim  containerd-shim-runc-v1  containerd-shim-runc-v2  containerd-stress  crictl  critest  ctd-decoder  ctr
[root@localhost ~]# cp usr/local/bin/containerd-shim-runc-v2 /usr/bin/

启动task，即表时在容器中运行了进程，即为动态容器。
# ctr task start -d nginx1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/

说明：
-d表示daemon或者后台的意思，否则会卡住终端

查看容器所在宿主机进程，是以宿主机进程的方式存在的。
# ctr task ls
TASK      PID     STATUS
nginx1    3395    RUNNING

查看容器的进程(都是物理机的进程)
# ctr task ps nginx1
PID     INFO
3395    -
3434    -

物理机查看到相应的进程
# ps -ef | grep 3395
root       3395   3375  0 19:16 ?        00:00:00 nginx: master process nginx -g daemon off;
101        3434   3395  0 19:16 ?        00:00:00 nginx: worker process

2.6 进入容器操作

# ctr task exec --exec-id 1 nginx1 /bin/sh# ifconfig 查看网卡信息...# curl 127.0.0.1 访问本地提供的web服务
...

说明：
为exec进程设定一个id，可以随意输入，只要保证唯一即可，也可使用$RANDOM变量。

2.7 直接运行一个动态容器

# ctr run -d --net-host docker.io/library/nginx:alpine nginx2
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/

说明：
* -d 代表dameon，后台运行
* --net-host 代表容器的IP就是宿主机的IP(相当于docker里的host类型网络)

查看已运行容器
# ctr container ls
CONTAINER    IMAGE                             RUNTIME
nginx2       docker.io/library/nginx:alpine    io.containerd.runc.v2

查看已运行容器中运行的进程,既tasks
# ctr tasks ls
TASK      PID     STATUS
nginx2    4061    RUNNING

进入容器
# ctr task exec --exec-id 1 -t nginx2 /bin/sh

/ # ifconfig 
ens33     Link encap:Ethernet  HWaddr 00:0C:29:B1:B6:1Dinet addr:192.168.10.164  Bcast:192.168.10.255  Mask:255.255.255.0inet6 addr: fe80::2b33:40ed:9311:8812/64 Scope:Linkinet6 addr: fe80::adf4:a8bc:a1c:a9f7/64 Scope:LinkUP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1RX packets:55360 errors:0 dropped:0 overruns:0 frame:0TX packets:30526 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:53511295 (51.0 MiB)  TX bytes:2735050 (2.6 MiB)lo        Link encap:Local Loopbackinet addr:127.0.0.1  Mask:255.0.0.0inet6 addr: ::1/128 Scope:HostUP LOOPBACK RUNNING  MTU:65536  Metric:1RX packets:68 errors:0 dropped:0 overruns:0 frame:0TX packets:68 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:5916 (5.7 KiB)  TX bytes:5916 (5.7 KiB)virbr0    Link encap:Ethernet  HWaddr 52:54:00:E9:51:82inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0UP BROADCAST MULTICAST  MTU:1500  Metric:1RX packets:0 errors:0 dropped:0 overruns:0 frame:0TX packets:0 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

为容器中运行的网站添加网站文件
/ # echo "nginx2" > /usr/share/nginx/html/index.html
/ # exit

在宿主机上访问网站
[root@localhost ~]# curl 192.168.10.164
nginx2

2.8 暂停容器

查看容器状态
# ctr tasks ls
TASK      PID     STATUS
nginx2    4061    RUNNING

暂停容器
# ctr tasks pause nginx2

再次查看容器状态，看到其状态为PAUSED，表示停止。
# ctr tasks ls
TASK      PID     STATUS
nginx2    4061    PAUSED

[root@localhost ~]# curl http://192.168.10.164
在宿主机访问，发现不可以访问到网站

2.9 恢复容器

使用resume命令恢复容器
# ctr tasks resume nginx2

查看恢复后状态
# ctr tasks ls
TASK      PID     STATUS
nginx2    4061    RUNNING

在宿主机上访问容器中提供的服务
# curl http://192.168.10.164
nginx2

2.10 停止容器

# ctr tasks --help
......

使用kill命令停止容器中运行的进程，既为停止容器
# ctr tasks kill nginx2

查看容器停止后状态，STATUS为STOPPED
# ctr tasks ls
TASK      PID     STATUS
nginx1    3395    RUNNING
nginx2    4061    STOPPED

2.11 删除容器

# ctr tasks delete nginx2
必须先删除task，再删除容器

查看静态容器，确认其还存在于系统中
# ctr container ls
CONTAINER    IMAGE                             RUNTIME
nginx2       docker.io/library/nginx:alpine    io.containerd.runc.v2

删除容器
# ctr container delete nginx2

3、Containerd使用私有容器镜像仓库 Harbor

3.1 Harbor准备

3.2 配置Containerd使用Harbor仓库

3.2.1 Harbor主机名解析

在所有安装containerd宿主机上添加此配置信息。

# vim /etc/hosts
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.165 harbor.kubemsb.com

说明
* 192.168.10.165是harbor的IP* harbor.kubemsb.com建议用FQDN形式，如果用类似harbor这种短名，后面下载镜像会出问题

3.2.2 修改Containerd配置文件

此配置文件已提前替换过，仅修改本地容器镜像仓库地址即可。
# vim /etc/containerd/config.toml
# cat /etc/containerd/config.toml
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = -999[grpc]address = "/run/containerd/containerd.sock"uid = 0gid = 0max_recv_message_size = 16777216max_send_message_size = 16777216[debug]address = ""uid = 0gid = 0level = ""[metrics]address = ""grpc_histogram = false[cgroup]path = ""[plugins][plugins.cgroups]no_prometheus = false[plugins.cri]stream_server_address = "127.0.0.1"stream_server_port = "0"enable_selinux = falsesandbox_image = "easzlab/pause-amd64:3.2"stats_collect_period = 10systemd_cgroup = falseenable_tls_streaming = falsemax_container_log_line_size = 16384[plugins.cri.containerd]snapshotter = "overlayfs"no_pivot = false[plugins.cri.containerd.default_runtime]runtime_type = "io.containerd.runtime.v1.linux"runtime_engine = ""runtime_root = ""[plugins.cri.containerd.untrusted_workload_runtime]runtime_type = ""runtime_engine = ""runtime_root = ""[plugins.cri.cni]bin_dir = "/opt/kube/bin"conf_dir = "/etc/cni/net.d"conf_template = "/etc/cni/net.d/10-default.conf"[plugins.cri.registry][plugins.cri.registry.mirrors][plugins.cri.registry.mirrors."docker.io"]endpoint = ["https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"][plugins.cri.registry.mirrors."gcr.io"]endpoint = ["https://gcr.mirrors.ustc.edu.cn"][plugins.cri.registry.mirrors."k8s.gcr.io"]endpoint = ["https://gcr.mirrors.ustc.edu.cn/google-containers/"][plugins.cri.registry.mirrors."quay.io"]endpoint = ["https://quay.mirrors.ustc.edu.cn"][plugins.cri.registry.mirrors."harbor.kubemsb.com"]   在此处添加,在镜像加速器下面添加这一段endpoint = ["http://harbor.kubemsb.com"][plugins.cri.x509_key_pair_streaming]tls_cert_file = ""tls_key_file = ""[plugins.diff-service]default = ["walking"][plugins.linux]shim = "containerd-shim"runtime = "runc"runtime_root = ""no_shim = falseshim_debug = false[plugins.opt]path = "/opt/containerd"[plugins.restart]interval = "10s"[plugins.scheduler]pause_threshold = 0.02deletion_threshold = 0mutation_threshold = 100schedule_delay = "0s"startup_delay = "100ms"

重启containerd，以便于重新加载配置文件。
# systemctl daemon-reload
# systemctl restart containerd

3.2.3 ctr下载镜像

下载容器镜像
# ctr images pull --platform linux/amd64 docker.io/library/nginx:latest

说明:
* --platform linux/amd64 指定系统平台，也可以使用--all-platforms指定所有平台镜像。

查看已下载容器镜像
# ctr images ls
REF                              TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                                                                          LABELSdocker.io/library/nginx:latest   application/vnd.docker.distribution.manifest.list.v2+json sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 54.1 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x                            -

3.2.4 ctr上传镜像

上传到Harbor library公有项目

重新生成新的tag
# ctr images tag docker.io/library/nginx:latest harbor.kubemsb.com/library/nginx:latest
harbor.kubemsb.com/library/nginx:latest

查看已生成容器镜像
# ctr images ls
REF                                     TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                                                                          LABELS
docker.io/library/nginx:latest          application/vnd.docker.distribution.manifest.list.v2+json sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 54.1 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x                            -
harbor.kubemsb.com/library/nginx:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 54.1 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x                            -

推送容器镜像至Harbor
# ctr images push --platform linux/amd64 --plain-http -u admin:密码 harbor.kubemsb.com/library/nginx:latest

说明:* 先tag再push
* 因为我们harbor是http协议，不是https协议，所以需要加上`--plain-http`
* `--user admin:Harbor12345`指定harbor的用户名与密码

输出：
manifest-sha256:0fd68ec4b64b8dbb2bef1f1a5de9d47b658afd3635dc9c45bf0cbeac46e72101: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:dd025cdfe837e1c6395365870a491cf16bae668218edb07d85c626928a60e478:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.5 s                                                                    total:  9.3 Ki (18.1 KiB/s)

下载已上传容器镜像
# ctr images pull --plain-http harbor.kubemsb.com/library/nginx:latest