當(dāng)我們使用 Prometheus 來監(jiān)控 Kubernetes 集群的時候，kube-state-metrics（KSM） 基本屬于一個必備組件，它通過 Watch APIServer 來生成資源對象的狀態(tài)指標(biāo)，它并不會關(guān)注單個 Kubernetes 組件的健康狀況，而是關(guān)注各種資源對象的健康狀態(tài)，比如 Deployment、Node、Pod、Ingress、Job、Service 等等，每種資源對象中包含了需要指標(biāo)，我們可以在官方文檔 https://github.com/kubernetes/kube-state-metrics/tree/main/docs 處進(jìn)行查看。
要安裝 KSM 也非常簡單，代碼倉庫中就包含了對應(yīng)的資源清單文件，但是在安裝的時候記得要和你的 K8s 集群版本對應(yīng)。
我這里的測試集群是 v1.25 版本的，所以我先切換到該分支：
$ git clone https://github.com/kubernetes/kube-state-metrics && cd kube-state-metrics
$ git checkout v2.7.0
$ kubectl apply -f examples/standard
該方式會以 Deployment 方式部署一個 KSM 實例：
$ kubectl get deploy -n kube-system kube-state-metrics
NAME READY UP-TO-DATE AVAILABLE AGE
kube-state-metrics 1/1 1 1 2m49s
$ kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics
NAME READY STATUS RESTARTS AGE
kube-state-metrics-548546fc89-zgkx5 1/1 Running 0 2m51s
然后只需要讓 Prometheus 來發(fā)現(xiàn) KSM 實例就可以了，當(dāng)然有很多方式，比如可以通過添加注解來自動發(fā)現(xiàn)，也可以單獨為 KSM 創(chuàng)建一個獨立的 Job，如果使用的是 Prometheus Operator，也可以創(chuàng)建 ServiceMonitor 對象來獲取 KSM 指標(biāo)數(shù)據(jù)。
這種方式對于小規(guī)模集群是沒太大問題的，數(shù)據(jù)量不大，可以正常提供服務(wù)，只需要保證 KSM 高可用就可以在生產(chǎn)環(huán)境使用了。但是對于大規(guī)模的集群來說，就非常困難了，比如我們這里有一個 8K 左右 Pod 的集群，不算特別大。
但是只通過一個 KSM 實例來提供 metrics 指標(biāo)還是非常吃力的，這個時候可能大部分情況下是獲取不到指標(biāo)的，因為 metrics 接口里面的數(shù)據(jù)量太大了。
即使偶爾獲取到了，也需要話花很長時間，要知道我們會每隔 scrape_interval 的時間都會去訪問該指標(biāo)接口的，可能前面一次請求還沒結(jié)束，下一次請求又發(fā)起了，要解決這個問題就得從 KSM 端入手解決，在 KSM 的啟動參數(shù)中我們可以配置過濾掉一些不需要的指標(biāo)標(biāo)簽：
$ kube-state-metrics -h
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
Usage:
kube-state-metrics [flags]
kube-state-metrics [command]
Available Commands:
completion Generate completion script for kube-state-metrics.
help Help about any command
version Print version information.
Flags:
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
--apiserver string The URL of the apiserver to use as a master
--config string Path to the kube-state-metrics options config file
--custom-resource-state-config string Inline Custom Resource State Metrics config YAML (experimental)
--custom-resource-state-config-file string Path to a Custom Resource State Metrics config file (experimental)
--custom-resource-state-only Only provide Custom Resource State metrics (experimental)
--enable-gzip-encoding Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.
-h, --help Print Help text
--host string Host to expose metrics on. (default "::")
--kubeconfig string Absolute path to the kubeconfig file
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
--log_file string If non-empty, use this log file (no effect when -logtostderr=true)
--log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--metric-allowlist string Comma-separated list of metrics to be exposed. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
--metric-annotations-allowlist string Comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '*' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[*]').
--metric-denylist string Comma-separated list of metrics not to be enabled. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
--metric-labels-allowlist string Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '*' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[*]'). Additionally, an asterisk (*) can be provided as a key, which will resolve to all resources, i.e., assuming '--resources=deployments,pods', '=*=[*]' will resolve to '=deployments=[*],pods=[*]'.
--metric-opt-in-list string Comma-separated list of metrics which are opt-in and not enabled by default. This is in addition to the metric allow- and denylists
--namespaces string Comma-separated list of namespaces to be enabled. Defaults to ""
--namespaces-denylist string Comma-separated list of namespaces not to be enabled. If namespaces and namespaces-denylist are both set, only namespaces that are excluded in namespaces-denylist will be used.
--node string Name of the node that contains the kube-state-metrics pod. Most likely it should be passed via the downward API. This is used for daemonset sharding. Only available for resources (pod metrics) that support spec.nodeName fieldSelector. This is experimental.
--one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
--pod string Name of the pod that contains the kube-state-metrics container. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
--pod-namespace string Name of the namespace of the pod specified by --pod. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
--port int Port to expose metrics on. (default 8080)
--resources string Comma-separated list of Resources to be enabled. Defaults to "certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
--shard int32 The instances shard nominal (zero indexed) within the total number of shards. (default 0)
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
--stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
--telemetry-host string Host to expose kube-state-metrics self metrics on. (default "::")
--telemetry-port int Port to expose kube-state-metrics self metrics on. (default 8081)
--tls-config string Path to the TLS configuration file
--total-shards int The total number of shards. Sharding is disabled when total shards is set to 1. (default 1)
--use-apiserver-cache Sets resourceVersinotallow=0 for ListWatch requests, using cached resources from the apiserver instead of an etcd quorum read.
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Use "kube-state-metrics [command] --help" for more information about a command.
可以通過 --metric-allowlist 或者 --metric-denylist 參數(shù)進(jìn)行過濾。但是如果即使過濾了不需要的指標(biāo)或標(biāo)簽后指標(biāo)接口數(shù)據(jù)仍然非常大又該怎么辦呢？
其實我們可以想象一下，無論怎么過濾，請求一次到達(dá) metrics 接口后的數(shù)據(jù)量都是非常大的，這個時候是不是只能對指標(biāo)數(shù)據(jù)進(jìn)行拆分了，可以部署多個 KSM 實例，每個實例提供一部分接口數(shù)據(jù)，是不是就可以緩解壓力了，這其實就是我們常說的水平分片。為了水平分片 kube-state-metrics，它已經(jīng)實現(xiàn)了一些自動分片功能，它是通過以下標(biāo)志進(jìn)行配置的：
- --shard (從 0 開始)
- --total-shards
分片是通過對 Kubernetes 對象的 UID 進(jìn)行 MD5 哈希和對總分片數(shù)進(jìn)行取模運算來完成的，每個分片決定是否由 kube-state-metrics 的相應(yīng)實例處理對象。不過需要注意的是，kube-state-metrics 的所有實例，即使已經(jīng)分片，也會處理所有對象的網(wǎng)絡(luò)流量和資源消耗，而不僅僅是他們負(fù)責(zé)那部分對象，要優(yōu)化這個問題，Kubernetes API 需要支持分片的 list/watch 功能。在最理想的情況下，每個分片的內(nèi)存消耗將比未分片設(shè)置少 1/n。通常，為了使 kube-state-metrics 能夠迅速返回其指標(biāo)給 Prometheus，需要進(jìn)行內(nèi)存和延遲優(yōu)化。減少 kube-state-metrics 和 kube-apiserver 之間的延遲的一種方法是使用 --use-apiserver-cache 標(biāo)志運行 KSM，除了減少延遲，這個選項還將導(dǎo)致減少對 etcd 的負(fù)載，所以我們也是建議啟用該參數(shù)的。
使用了分片模式，則最好對分片相關(guān)指標(biāo)進(jìn)行監(jiān)控，以確保分片設(shè)置符合預(yù)期，可以用下面兩個報警規(guī)則來進(jìn)行報警：
- alert: KubeStateMetricsShardingMismatch
annotations:
description: kube-state-metrics pods are running with different --total-shards configuration, some Kubernetes objects may be exposed multiple times or not exposed at all.
summary: kube-state-metrics sharding is misconfigured.
expr: |
stdvar (kube_state_metrics_total_shards{job="kube-state-metrics"}) != 0
for: 15m
labels:
severity: critical
- alert: KubeStateMetricsShardsMissing
annotations:
description: kube-state-metrics shards are missing, some Kubernetes objects are not being exposed.
summary: kube-state-metrics shards are missing.
expr: |
2^max(kube_state_metrics_total_shards{job="kube-state-metrics"}) - 1
-
sum( 2 ^ max by (shard_ordinal) (kube_state_metrics_shard_ordinal{job="kube-state-metrics"}) )
!= 0
for: 15m
labels:
severity: critical
由于手動去配置分片可能會出現(xiàn)錯誤，所以 KSM 也提供了自動分片的功能，可以通過 StatefulSet 方式來部署多個副本的 KSM，自動分片允許每個分片在 StatefulSet 中部署時發(fā)現(xiàn)其實例位置，這對于自動配置分片非常有用。所以要啟用自動分片，必須通過 StatefulSet 運行 kube-state-metrics，并通過 --pod 和 --pod-namespace 標(biāo)志將 pod 名稱和命名空間傳遞給 kube-state-metrics 進(jìn)程，如下所示：
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kube-state-metrics
namespace: kube-system
spec:
replicas: 10
selector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics
serviceName: kube-state-metrics
template:
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.8.0
spec:
automountServiceAccountToken: true
containers:
- args:
- --pod=$(POD_NAME)
- --pod-namespace=$(POD_NAMESPACE)
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0
# ......
使用這種部署分片的方法，當(dāng)你想要通過單個 Kubernetes 資源（在這種情況下為單個 StatefulSet）管理 KSM 分片時是很有用的，而不是每個分片都有一個 Deployment，這種優(yōu)勢在部署大量分片時尤為顯著。
當(dāng)然使用自動分片的部署方式也是有缺點的，主要是來自于 StatefulSet 支持的滾動升級策略，當(dāng)由 StatefulSet 管理時，一個一個地替換 pod，當(dāng) pod 先被終止后，然后再重新創(chuàng)建，這樣的升級速度較慢，也可能會導(dǎo)致每個分片的短暫停機(jī)，如果在升級期間進(jìn)行 Prometheus 抓取，則可能會錯過 kube-state-metrics 導(dǎo)出的某些指標(biāo)。
自動分片功能的示例清單在 examples/autosharding 目錄中可以找到，可以直接通過下面的命令來部署：
$ kubectl apply -k examples/autosharding
上面的命令會以 StatefulSet 方式部署 2 個 KSM 實例：
$ kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics
NAME READY STATUS RESTARTS AGE
kube-state-metrics-0 1/1 Running 0 70m
kube-state-metrics-1 1/1 Running 0 65m
可以隨便查看一個 Pod 的日志：
$ kubectl logs -f kube-state-metrics-1 -nkube-system
I0216 05:53:23.151163 1 wrapper.go:78] Starting kube-state-metrics
I0216 05:53:23.154495 1 server.go:125] "Used default resources"
I0216 05:53:23.154923 1 types.go:184] "Using all namespaces"
I0216 05:53:23.155556 1 server.go:166] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: "
W0216 05:53:23.155792 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0216 05:53:23.178553 1 server.go:311] "Tested communication with server"
I0216 05:53:23.241024 1 server.go:316] "Run with Kubernetes cluster version" major="1" minor="25" gitVersinotallow="v1.25.3" gitTreeState="clean" gitCommit="434bfd82814af038ad94d62ebe59b133fcb50506" platform="linux/arm64"
I0216 05:53:23.241169 1 server.go:317] "Communication with server successful"
I0216 05:53:23.245132 1 server.go:263] "Started metrics server" metricsServerAddress="[::]:8080"
I0216 05:53:23.246148 1 metrics_handler.go:103] "Autosharding enabled with pod" pod="kube-system/kube-state-metrics-1"
I0216 05:53:23.246233 1 metrics_handler.go:104] "Auto detecting sharding settings"
I0216 05:53:23.246267 1 server.go:252] "Started kube-state-metrics self metrics server" telemetryAddress="[::]:8081"
I0216 05:53:23.253477 1 server.go:69] levelinfomsgListening onaddress[::]:8081
I0216 05:53:23.253477 1 server.go:69] levelinfomsgListening onaddress[::]:8080
I0216 05:53:23.253944 1 server.go:69] levelinfomsgTLS is disabled.http2falseaddress[::]:8080
I0216 05:53:23.254534 1 server.go:69] levelinfomsgTLS is disabled.http2falseaddress[::]:8081
I0216 05:53:23.297524 1 metrics_handler.go:80] "Configuring sharding of this instance to be shard index (zero-indexed) out of total shards" shard=1 totalShards=2
I0216 05:53:23.411710 1 builder.go:257] "Active resources" activeStoreNames="certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
可以看到有類型 "Configuring sharding of this instance to be shard index (zero-indexed) out of total shards" shard=1 totalShards=2 這樣的日志信息，表面自動分片成功了。我們可以去分別獲取下分片的指標(biāo)數(shù)據(jù)大小，并和未分片之前的進(jìn)行對比，可以看到分片后的指標(biāo)明顯減少了，如果單個實例的指標(biāo)數(shù)據(jù)還是太大，則可以增加 StatefulSet 的副本數(shù)即可。
此外我們還可以單獨針對 pod 指標(biāo)按照每個節(jié)點進(jìn)行分片，只需要加上 --node 和 --resource 即可，這個時候我們直接使用一個 DaemonSet 來創(chuàng)建 KSM 實例即可，如下所示：
apiVersion: apps/v1
kind: DaemonSet
spec:
template:
spec:
containers:
- image: registry.k8s.io/kube-state-metrics/kube-state-metrics:IMAGE_TAG
name: kube-state-metrics
args:
- --resource=pods
- --node=$(NODE_NAME)
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
對于其他的指標(biāo)我們也可以使用 --resource 來單獨指定部署，也可以繼續(xù)使用分片的方式。總結(jié)來說就是對于大規(guī)模集群使用 kube-state-metrics 需要做很多優(yōu)化：
- 過濾不需要的指標(biāo)和標(biāo)簽
- 通過分片降低 KSM 實例壓力
- 可以使用 DaemonSet 方式單獨針對 pod 指標(biāo)進(jìn)行部署
當(dāng)然可能也有人會問，如果自己的業(yè)務(wù)指標(biāo)也超級大的情況下該怎么辦呢？當(dāng)然就得讓業(yè)務(wù)方來做支持了，首先要明確指標(biāo)數(shù)據(jù)這么大是否正常？如果需求就是如此，那么也得想辦法能夠支持分片。
