Kubernetes 集群calico网络故障排查思路
报错calico/node is not ready: BIRD is not ready: BGP not established with 172.16.0.20,172.16.0.30
\\calico未准备好，BGP协议不能与172.16.0.20,172.16.0.30内网IP地址连接BGP协议:边界网关协议
访问k8s的dashboard界面无法访问网站，查看pod，未知原因导致calico的Pod资源重新创建后无法启动，显示的是0/1状态
[root@k8s-master yaml]# kubectl get pod -n kube-system NAMESPACE NAME READY STATUS RESTARTS AGE ... kube-system calico-kube-controllers-578894d4cd-rsgqd 1/1 Running 0 115d kube-system calico-node-64s8s 1/1 Running 3 127d kube-system calico-node-j4t7q 1/1 Running 0 127d kube-system calico-node-n6vr4 0/1 Running 0 40s
Calico的Pod报错内容
[root@k8s-master yaml]# kubectl describe pod -n kube-system calico-node-n6vr4
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/calico-node-n6vr4 to k8s-masterNormal Pulled 41s kubelet, k8s-master Container image "calico/cni:v3.15.1" already present on machineNormal Created 41s kubelet, k8s-master Created container upgrade-ipamNormal Started 40s kubelet, k8s-master Started container upgrade-ipamNormal Pulled 40s kubelet, k8s-master Container image "calico/cni:v3.15.1" already present on machineNormal Started 39s kubelet, k8s-master Started container install-cniNormal Created 39s kubelet, k8s-master Created container install-cniNormal Pulled 39s kubelet, k8s-master Container image "calico/pod2daemon-flexvol:v3.15.1" already present on machineNormal Pulled 38s kubelet, k8s-master Container image "calico/node:v3.15.1" already present on machineNormal Started 38s kubelet, k8s-master Started container flexvol-driverNormal Created 38s kubelet, k8s-master Created container flexvol-driverNormal Created 37s kubelet, k8s-master Created container calico-nodeNormal Started 37s kubelet, k8s-master Started container calico-nodeWarning Unhealthy 27s kubelet, k8s-master Readiness probe failed: 2020-08-14 02:16:54.068 [INFO][142] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 172.16.0.20,172.16.0.30Warning Unhealthy 17s kubelet, k8s-master Readiness probe failed: 2020-08-14 02:17:04.059 [INFO][181] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 172.16.0.20,172.16.0.30Warning Unhealthy 7s kubelet, k8s-master Readiness probe failed: 2020-08-14 02:17:14.065 [INFO][207] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 172.16.0.20,172.16.0.30原因：calico没有发现实node节点实际的网卡名称
解决方法
调整calicao的网络插件的网卡发现机制，修改IP_AUTODETECTION_METHOD对应的value值。下载的官方提供的yaml文件中，ip识别策略（IPDETECTMETHOD）没有配置，即默认为first-found，这会导致一个网络异常的ip作为nodeIP被注册，从而影响node之间的网络连接。可以修改成can-reach或者interface的策略，尝试连接某一个Ready的node的IP，以此选择出正确的IP。
# 修改calicao的yaml文件，添加两行配置# - name: IP_AUTODETECTION_METHOD# value: "interface=eth1" # 根据实际网卡名称配置[root@k8s-master yaml]# vim calico.yaml...(3546行)# Cluster type to identify the deployment type- name: CLUSTER_TYPEvalue: "k8s,bgp"#新添加的配置- name: IP_AUTODETECTION_METHODvalue: "interface=eth1"# Auto-detect the BGP IP address.- name: IPvalue: "autodetect"# Enable IPIP- name: CALICO_IPV4POOL_IPIPvalue: "Always"# Enable or Disable VXLAN on the default IP pool.- name: CALICO_IPV4POOL_VXLANvalue: "Never"#重新构建kubectl apply -f calico.yaml
修复完成
[root@k8s-master yaml]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-578894d4cd-rsgqd 1/1 Running 0 115d
calico-node-6ktn4 1/1 Running 0 26m
calico-node-8k5z8 1/1 Running 0 26m
calico-node-g87hc 1/1 Running 0 1m再次访问集群的各种资源已经可以访问了