研发工程师玩转Kubernetes——Node亲和性requiredDuringSchedulingIgnoredDuringExecution几种边界实验
在《研发工程师玩转Kubernetes——使用Node特性定向调度Pod》中，我们提到requiredDuringSchedulingIgnoredDuringExecution只有在规则被满足的时候才能执行调度。本节我们将测试几种边界情况，看看Kubernetes的行为。
没有满足的条件
假设我们测试的Node都没有Label:not_exist=“”，于是我们在清单中要求必须有这个Label，来测试这个边界。
# nginx_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deployment
spec:replicas: 1selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: not_existoperator: Invalues:- ""containers:- name: nginx-containerimage: nginxports:- containerPort: 80
执行下面的指令
kubectl create -f nginx_deployment.yaml
deployment.apps/nginx-deployment created
观察
Pod的情况
kubectl get pod --watch -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-6b5d69bc9d-94vgl 0/1 Pending 0 0s <none> <none> <none> <none>
nginx-deployment-6b5d69bc9d-94vgl 0/1 Pending 0 0s <none> <none> <none> <none>
Deployment的情况
kubectl get deployments.apps --watch -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx-deployment 0/1 1 0 59s nginx-container nginx app=nginx
可以看到Pod一直处于Pending状态，也没调度到任何Node上。
多个nodeSelectorTerms
为了进行这个测试，我们给UbuntuB和UbunutC设置对应的Label。
kubectl label nodes ubuntub name:ubuntub
node/ubuntub labeled
kubectl label nodes ubuntuc name=ubuntuc
node/ubuntuc labeled
我们使用下面指令查看下修改后的Labels。
kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
ubuntud Ready <none> 21h v1.26.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntud,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntuc Ready <none> 21h v1.26.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntuc,kubernetes.io/os=linux,microk8s.io/cluster=true,name=ubuntuc,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntub Ready <none> 21h v1.26.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntub,kubernetes.io/os=linux,microk8s.io/cluster=true,name=ubuntub,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntue Ready <none> 21h v1.26.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntue,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-worker=microk8s-worker
ubuntua Ready <none> 21h v1.27.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ubuntua,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
然后清单改成多个nodeSelectorTerms
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deployment
spec:replicas: 1selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: nameoperator: Invalues:- "ubuntub"- matchExpressions:- key: nameoperator: Invalues:- "ubuntuc"containers:- name: nginx-containerimage: nginxports:- containerPort: 80
调用下面指令创建Deployment
kubectl create -f nginx_deployment.yaml
deployment.apps/nginx-deployment created
观察
kubectl get pod --watch -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-58d4498bdd-s5fvd 0/1 Pending 0 0s <none> <none> <none> <none>
nginx-deployment-58d4498bdd-s5fvd 0/1 Pending 0 0s <none> ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 0/1 ContainerCreating 0 0s <none> ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 0/1 ContainerCreating 0 0s <none> ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 1/1 Running 0 4s 10.1.43.212 ubuntuc <none> <none>
kubectl get deployments.apps --watch -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx-deployment 0/1 0 0 0s nginx-container nginx app=nginx
nginx-deployment 0/1 0 0 0s nginx-container nginx app=nginx
nginx-deployment 0/1 0 0 0s nginx-container nginx app=nginx
nginx-deployment 0/1 1 0 0s nginx-container nginx app=nginx
nginx-deployment 1/1 1 1 4s nginx-container nginx app=nginx
可以看到Node的requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions之间是取或的关系，即只要满足其中一个条件就可以被调度到。
为了再次验证，我们可以让UbuntC驱逐这个Pod。
kubectl taint node ubuntuc node_type=worker:NoExecute
node/ubuntuc tainted
再观察
kubectl get pod --watch -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-58d4498bdd-s5fvd 1/1 Running 0 8m28s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 1/1 Terminating 0 8m28s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 1/1 Terminating 0 8m28s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 Pending 0 1s <none> <none> <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 Pending 0 1s <none> ubuntub <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 ContainerCreating 0 1s <none> ubuntub <none> <none>
nginx-deployment-58d4498bdd-s5fvd 1/1 Terminating 0 8m29s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 ContainerCreating 0 1s <none> ubuntub <none> <none>
nginx-deployment-58d4498bdd-s5fvd 0/1 Terminating 0 8m30s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 0/1 Terminating 0 8m30s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-s5fvd 0/1 Terminating 0 8m30s 10.1.43.212 ubuntuc <none> <none>
nginx-deployment-58d4498bdd-kc2fz 1/1 Running 0 4s 10.1.209.132 ubuntub <none> <none>
可以看到Pod被调度到另外一个匹配的条件对应的Node（UbuntuB）上。
被彻底驱逐
再让UbuntuB驱逐这个Pod，这样没有哪个Node可以符合条件。
kubectl taint node ubuntub node_type=worker:NoExecute
node/ubuntub tainted
再观察
kubectl get pod --watch -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-58d4498bdd-kc2fz 1/1 Terminating 0 3m30s 10.1.209.132 ubuntub <none> <none>
nginx-deployment-58d4498bdd-kc2fz 1/1 Terminating 0 3m30s 10.1.209.132 ubuntub <none> <none>
nginx-deployment-58d4498bdd-wjkbx 0/1 Pending 0 0s <none> <none> <none> <none>
nginx-deployment-58d4498bdd-wjkbx 0/1 Pending 0 0s <none> <none> <none> <none>
nginx-deployment-58d4498bdd-kc2fz 1/1 Terminating 0 3m30s 10.1.209.132 ubuntub <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 Terminating 0 3m31s 10.1.209.132 ubuntub <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 Terminating 0 3m31s 10.1.209.132 ubuntub <none> <none>
nginx-deployment-58d4498bdd-kc2fz 0/1 Terminating 0 3m32s 10.1.209.132 ubuntub <none> <none>
因为被驱逐，老的Pod被终止，而新的Pod因为哪个Node可以被匹配到，而变成pending状态。
取消Label
接上上步，我们使用下面指令取消UbuntuB对Pod的驱逐
kubectl taint node ubuntub node_type=worker:NoExecute-
node/ubuntub untainted
可以看到Deployment将Pod调度到UbuntuB上
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-58d4498bdd-wjkbx 0/1 ContainerCreating 0 5m20s <none> ubuntub <none> <none>
nginx-deployment-58d4498bdd-wjkbx 0/1 ContainerCreating 0 5m20s <none> ubuntub <none> <none>
nginx-deployment-58d4498bdd-wjkbx 1/1 Running 0 5m23s 10.1.209.133 ubuntub <none> <none>
然后我们使用下面指令取消UbuntuB的Label：name=unbuntb
kubectl label nodes ubuntub name-
这次Deployment不会驱逐该Pod
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-58d4498bdd-wjkbx 1/1 Running 0 11m 10.1.209.133 ubuntub <none> <none>
总结
- requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions之间是取或的关系，即只要满足其中一个条件就可以被调度到。
- 没有匹配的条件，Pod会被创建，但是处于Pending状态，不会被部署到任何一个Node上。
- 如果Pod已经在Node上运行，此时删除Node匹配上的Label，Deployment不会终止该Pod。