监控k8s controller和scheduler，创建serviceMonitor以及Rules
目录
一、修改kube-controller和kube-schduler的yaml文件
二、创建service、endpoint、serviceMonitor
三、Prometheus验证
四、创建PrometheusRule资源
五、Prometheus验证
直接上干货
一、修改kube-controller和kube-schduler的yaml文件
注意：修改时要一个节点一个节点的修改，等上一个修改的节点服务正常启动后再修改下个节点
kube-controller文件路径：/etc/kubernetes/manifests/kube-controller-manager.yaml
kube-scheduler文件路径：/etc/kubernetes/manifests/kube-scheduler.yamlvim /etc/kubernetes/manifests/kube-controller-manager.yaml
vim /etc/kubernetes/manifests/kube-scheduler.yaml
二、创建service、endpoint、serviceMonitor
kube-controller-monitor.yaml
apiVersion: v1
kind: Service
metadata:labels:k8s-app: kube-controller-managername: kube-controller-manage-monitornamespace: kube-system
spec:ports:- name: https-metricsport: 10257protocol: TCPtargetPort: 10257sessionAffinity: Nonetype: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:labels:k8s-app: kube-controller-managername: kube-controller-manage-monitornamespace: kube-system
subsets:
- addresses:- ip: 10.50.238.191- ip: 10.50.107.48- ip: 10.50.140.151ports:- name: https-metricsport: 10257protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:k8s-app: kube-controller-managername: kube-controller-managernamespace: kube-system
spec:endpoints:- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokeninterval: 30sport: https-metricsscheme: httpstlsConfig:insecureSkipVerify: truejobLabel: k8s-appnamespaceSelector:matchNames:- kube-systemselector:matchLabels:k8s-app: kube-controller-managerkube-scheduler-monitor.yaml
apiVersion: v1
kind: Service
metadata:labels:k8s-app: kube-schedulername: kube-scheduler-monitornamespace: kube-system
spec:ports:- name: https-metricsport: 10259protocol: TCPtargetPort: 10259sessionAffinity: Nonetype: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:labels:k8s-app: kube-schedulername: kube-scheduler-monitornamespace: kube-system
subsets:
- addresses:- ip: 10.50.238.191- ip: 10.50.107.48- ip: 10.50.140.151ports:- name: https-metricsport: 10259protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:k8s-app: kube-schedulername: kube-schedulernamespace: kube-system
spec:endpoints:- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokeninterval: 30sport: https-metricsscheme: httpstlsConfig:insecureSkipVerify: truejobLabel: k8s-appnamespaceSelector:matchNames:- kube-systemselector:matchLabels:k8s-app: kube-schedulerroot@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/serviceMonitor/kubernetes-cluster# kubectl apply -f ./
service/kube-controller-manage-monitor created
endpoints/kube-controller-manage-monitor created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
service/kube-scheduler-monitor created
endpoints/kube-scheduler-monitor created
servicemonitor.monitoring.coreos.com/kube-scheduler created三、Prometheus验证
四、创建PrometheusRule资源
kube-controller-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:annotations:meta.helm.sh/release-namespace: cattle-monitoring-systemprometheus-operator-validated: "true"generation: 3labels:app: rancher-monitoringapp.kubernetes.io/instance: rancher-monitoringapp.kubernetes.io/part-of: rancher-monitoringname: kube-controller-managernamespace: cattle-monitoring-system
spec:groups:- name: kube-controller-manager.rulerules:- alert: K8SControllerManagerDownexpr: absent(up{job="kube-controller-manager"} == 1)for: 1mlabels:severity: criticalcluster: manage-prodannotations:description: There is no running K8S controller manager. Deployments and replication controllers are not making progress.summary: No kubernetes controller manager are reachable- alert: K8SControllerManagerDownexpr: up{job="kube-controller-manager"} == 0for: 1mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is down. {{ $labels.instance }} isn't reachablesummary: kubernetes controller manager is down- alert: K8SControllerManagerUserCPUexpr: sum(rate(container_cpu_user_seconds_total{pod=~"kube-controller-manager.*",container_name!="POD"}[5m]))by(pod) > 5for: 5mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is user cpu time > 5s. {{ $labels.instance }} isn't reachablesummary: kubernetes controller 负载较高超过5s- alert: K8SControllerManagerUseMemoryexpr: sum(rate(container_memory_usage_bytes{pod=~"kube-controller-manager.*",container_name!="POD"}[5m])/1024/1024)by(pod) > 20for: 5mlabels:severity: infocluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is use memory More than 20MBsummary: kubernetes controller 使用内存超过20MB- alert: K8SControllerManagerQueueTimedelayexpr: histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job="kubernetes-controller-manager"}[5m])) by(le)) > 10for: 5mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is QueueTimedelay More than 10ssummary: kubernetes controller 队列停留时间超过10秒，请检查ControllerManagerkube-scheduler-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:annotations:meta.helm.sh/release-namespace: cattle-monitoring-systemprometheus-operator-validated: "true"generation: 3labels:app: rancher-monitoringapp.kubernetes.io/instance: rancher-monitoringapp.kubernetes.io/part-of: rancher-monitoringname: kube-schedulernamespace: cattle-monitoring-system
spec:groups:- name: kube-scheduler.rulerules:- alert: K8SSchedulerDownexpr: absent(up{job="kube-scheduler"} == 1)for: 1mlabels:severity: criticalcluster: manage-prodannotations:description: "There is no running K8S scheduler. New pods are not being assigned to nodes."summary: "all k8s scheduler is down"- alert: K8SSchedulerDownexpr: up{job="kube-scheduler"} == 0for: 1mlabels:severity: warningcluster: manage-prodannotations:description: "K8S scheduler {{ $labels.instance }} is no running. New pods are not being assigned to nodes."summary: "k8s scheduler {{ $labels.instance }} is down"- alert: K8SSchedulerUserCPUexpr: sum(rate(container_cpu_user_seconds_total{pod=~"kube-scheduler.*",container_name!="POD"}[5m]))by(pod) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: "kubernetes scheduler {{ $labels.instance }} is user cpu time > 1s. {{ $labels.instance }} isn't reachable"summary: "kubernetes scheduler 负载较高超过1s,当前值为{{$value}}"- alert: K8SSchedulerUseMemoryexpr: sum(rate(container_memory_usage_bytes{pod=~"kube-scheduler.*",container_name!="POD"}[5m])/1024/1024)by(pod) > 20for: 5mlabels:severity: infocluster: manage-prodannotations:current_value: '{{$value}}'description: "kubernetess scheduler {{ $labels.instance }} is use memory More than 20MB"summary: "kubernetes scheduler 使用内存超过20MB,当前值为{{$value}}MB"- alert: K8SSchedulerPodPendingexpr: sum(scheduler_pending_pods{job="kubernetes-scheduler"})by(queue) > 5for: 5mlabels:severity: infocluster: manage-prodannotations:current_value: '{{$value}}'description: "kubernetess scheduler {{ $labels.instance }} is Pending pod More than 5"summary: "kubernetes scheduler pod无法调度 > 5,当前值为{{$value}}"- alert: K8SSchedulerPodPendingexpr: sum(scheduler_pending_pods{job="kubernetes-scheduler"})by(queue) > 10for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }} is Pending pod More than 10summary: "kubernetes scheduler pod无法调度 > 10,当前值为{{$value}}"- alert: K8SSchedulerPodPendingexpr: sum(rate(scheduler_binding_duration_seconds_count{job="kubernetes-scheduler"}[5m])) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }}summary: "kubernetes scheduler pod 无法绑定调度有问题，当前值为{{$value}}"- alert: K8SSchedulerVolumeSpeedexpr: sum(rate(scheduler_volume_scheduling_duration_seconds_count{job="kubernetes-scheduler"}[5m])) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }}summary: "kubernetes scheduler pod Volume 速度延迟，当前值为{{$value}}"- alert: K8SSchedulerClientRequestSlowexpr: histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job="kubernetes-scheduler"}[5m])) by (verb, url, le)) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }}summary: "kubernetes scheduler 客户端请求速度延迟，当前值为{{$value}}"root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# kubectl apply -f kube-controller-rules.yaml
prometheusrule.monitoring.coreos.com/kube-apiserver-rules configured
root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# kubectl apply -f kube-scheduler-rules.yaml
prometheusrule.monitoring.coreos.com/kube-apiserver-rules configured
root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# 五、Prometheus验证
