prometheus监控k8s的metric详解-01-apiserver部分-05-其他
6. 其他
6.1 cache list
6.1.1 apiserver_cache_list_total
-
作用:表示缓存中与不同资源相关的对象的总数
-
说明:
- 查询语句:
apiserver_cache_list_total{instance="10.10.239.101:6443"}
- 查询结果
apiserver_cache_list_total{index="f:spec.nodeName",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/pods"} 14
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/apiextensions.k8s.io/customresourcedefinitions"} 10
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/apiregistration.k8s.io/apiservices"} 10
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/catalog.cattle.io/apps"} 7
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/catalog.cattle.io/clusterrepos"} 7
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/catalog.cattle.io/operations"} 7
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/certificatesigningrequests"} 8
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/clusterrolebindings"} 11
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/clusterroles"} 10
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/configmaps"} 12
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/controllerrevisions"} 6
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/controllers"} 11
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/cronjobs"} 9
6.1.2 apiserver_cache_list_fetched_objects_total
- 作用:表示从缓存中实际提取并返回给客户端的不同资源相关对象的总数
6.1.3 apiserver_cache_list_returned_objects_total
- 作用:表示从缓存中返回给客户端的不同资源相关对象的总数
6.2 client
6.2.1 apiserver_client_certificate_expiration_seconds_count
- 作用:客户端证书过期时间检查次数的计数
注意:不是服务器的证书,而是和服务器链接的客户端证书。包括比如开发电脑远程链接
-
说明:
- 查询语句:
rate(apiserver_client_certificate_expiration_seconds_count[10m])
- 查询结果
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"} 0.12374644444444445
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m02",kubernetes_io_os="linux"} 0.12567777777777778
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m03",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m03",kubernetes_io_os="linux"} 0.12477644444444445
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n01",kubernetes_io_os="linux"} 0.12544933333333333
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n02",kubernetes_io_os="linux"} 0.1262991111111111
{instance="10.10.239.101:6443",job="kubernetes-apiservers"} 4.586828611111111
{instance="10.10.239.102:6443",job="kubernetes-apiservers"} 1.2532182962962963
{instance="10.10.239.103:6443",job="kubernetes-apiservers"} 0.6262733333333332
- 说明:以最后一条为例,10分钟内,链接10.10.239.103:6443的客户端过期事件为每秒0.6件。
6.2.2 apiserver_client_certificate_expiration_seconds_bucket
- 作用:客户端证书数量累计值,放到不同的桶中
- 示例
- 查询语句
rate(apiserver_client_certificate_expiration_seconds_bucket{instance="10.10.239.103:6443",job="kubernetes-apiservers"}[10m])
- 结果
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="+Inf"} 0.6204834834834836
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="0"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="1.5552e+07"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="172800"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="1800"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="2.592e+06"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="21600"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="3.1104e+07"} 0.6204834834834836
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="345600"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="3600"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="43200"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="604800"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="7.776e+06"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="7200"} 0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="86400"} 0
- 说明
过期时间范围(秒) | 标签(le) | 每秒新增速率值 |
---|---|---|
<= 0 | le="0" | 0 |
<= 1,800(30 分钟) | le="1800" | 0 |
<= 3,600(1 小时) | le="3600" | 0 |
<= 7,200(2 小时) | le="7200" | 0 |
<= 21,600(6 小时) | le="21600" | 0 |
<= 43,200(12 小时) | le="43200" | 0 |
<= 86,400(1 天) | le="86400" | 0 |
<= 172,800(2 天) | le="172800" | 0 |
<= 345,600(4 天) | le="345600" | 0 |
<= 604,800(7 天) | le="604800" | 0 |
<= 2,592,000(30 天) | le="2.592e+06" | 0 |
<= 7,776,000(90 天) | le="7.776e+06" | 0 |
<= 15,552,000(180 天) | le="1.5552e+07" | 0 |
<= 31,104,000(1 年) | le="3.1104e+07" | 0.6204834834834836 |
无限制(+∞) | le="+Inf" | 0.6204834834834836 |
27. apiserver_client_certificate_expiration_seconds_sum
- 作用:所有客户端证书过期时间和的累计值
其实不理解这个指标有什么意义,虽然它的rate值 除以 count的rate值是证书平均过期时间,但是这种平均过期时间有什么用呢?乞丐和马云的平均值是个富翁,就可以防止乞丐饿死么?
6.3 x509 SAN字段
6.3.1 apiserver_webhooks_x509_missing_san_total
- 作用:处理的 Webhook 请求中,证书缺少 SAN(Subject Alternative Name)字段的次数
SAN 是 X.509 证书的重要字段,通常包含了证书的有效主机名或 IP 地址。如果缺少 SAN,可能会导致证书校验失败或不符合标准
- 示例:
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"} 0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m02",kubernetes_io_os="linux"} 0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m03",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m03",kubernetes_io_os="linux"} 0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n01",kubernetes_io_os="linux"} 0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n02",kubernetes_io_os="linux"} 0
apiserver_webhooks_x509_missing_san_total{instance="10.10.239.101:6443",job="kubernetes-apiservers"} 0
apiserver_webhooks_x509_missing_san_total{instance="10.10.239.102:6443",job="kubernetes-apiservers"} 0
apiserver_webhooks_x509_missing_san_total{instance="10.10.239.103:6443",job="kubernetes-apiservers"} 0
6.3.2 apiserver_kube_aggregator_x509_missing_san_total
- 作用:通过聚合器的请求中,证书缺少 SAN(Subject Alternative Name)字段的次数
6.4 其他
6.4.1 apiserver_envelope_encryption_dek_cache_fill_percent
- 作用:用于信封加密的数据加密密钥(DEK)缓存的填充百分比
6.4.2 apiserver_init_events_total
- 作用:统计了 初始化事件的总数
- 示例:
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*coordination.Lease"} 422
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.ConfigMap"} 23
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Endpoints"} 183
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Namespace"} 1646
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Node"} 3197
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.PersistentVolume"} 1
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.PersistentVolumeClaim"} 2
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Pod"} 1110
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Secret"} 31
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*discovery.EndpointSlice"} 20
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*storage.StorageClass"} 1
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*unstructured.Unstructured"} 108
6.4.3 apiserver_longrunning_gauge
- 作用:衡量长时间运行的请求的数量。
- 示例:
apiserver_longrunning_gauge{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="cluster",verb="WATCH",version="v1"} 4
apiserver_longrunning_gauge{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"} 0
6.4.4 apiserver_longrunning_requests
- 作用:统计了正在处理的长时间运行的请求总数。
- 示例:
apiserver_longrunning_requests{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="cluster",verb="WATCH",version="v1"} 4
apiserver_longrunning_requests{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"} 0
6.4.5 apiserver_registered_watchers
- 作用:有多少个客户端正在通过 watch 功能注册并监听 指定种类对象的资源的变化
- 示例:
apiserver_registered_watchers{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="Deployment",version="v1"} 4
6.4.6 apiserver_selfrequest_total
- 作用:记录发出的自请求的总数
- 示例:
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="namespaces",verb="GET"} 2218318
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="namespaces",verb="LIST"} 2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="namespaces",verb="WATCH"} 32921
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="nodes",verb="LIST"} 2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="nodes",verb="WATCH"} 32892
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="persistentvolumes",verb="LIST"} 2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="persistentvolumes",verb="WATCH"} 32903
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="poddisruptionbudgets",verb="LIST"} 1
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="pods",verb="LIST"} 2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="pods",verb="WATCH"} 32876
6.4.7 apiserver_tls_handshake_errors_total
- 作用:统计在 TLS 握手过程中发生的错误总数