Configuring prometheus to scrap SSL etcd cluster
2 min read
In this setup we will configure the prometheus community helm chart to scrap metrics from a SSL enabled etcd.
The etcd was installed through kupespray kubernetes cluster, but you should be able to adapt to your case.
Generating the SSL Client Keys
We will need to generate client certificate keys to connect to etcd.
Kubespray install the CA pem in /etc/ssl/etcd/ssl/ directory on each node
SSH into the first etcd node and generate a new ssl client certificate. The first node is the one with the file /etc/ssl/etcd/openssl.conf, you can find the file with the command
find / -name openssl.conf
Create the client request:
openssl req -config /etc/ssl/etcd/openssl.conf -new -nodes -keyout etcd-client-prometheus.key -out etcd-client-prometheus.csr -subj "/CN=etcd-client"
Signing the request
openssl ca -config openssl.cnf -extensions etcd_client -keyfile private/ca.key -cert certs/ca.crt -out certs/etcd-client.crt -infiles etcd-client.csr
Generate the certs
openssl x509 -req -in etcd-client-prometheus.csr -CAkey /etc/ssl/etcd/ssl/ca-key.pem -CA /etc/ssl/etcd/ssl/ca.pem -CAcreateserial -days 36500 -extensions ssl_client -out etcd-client-prometheus.crt -extfile /etc/ssl/etcd/openssl.conf
curl --cert etcd-client-prometheus.crt --key etcd-client-prometheus.key https://10.8.0.2:2379/metrics
Change the ip for your server ip
Configuring Prometheus to user the certs
Create a secret to old the certs (must be in same namespace as prometheus pods)
kubectl create secret generic etcd-client-cert --from-file=./ca.pem --from-file=./etcd-client-prometheus.crt --from-file=./etcd-client-prometheus.key
Configure Prometheus community to read the secret and to use it in etcd monitoring:
## Component scraping etcd ## kubeEtcd: enabled: true ## If your etcd is not deployed as a pod, specify IPs it can be found on ## endpoints: - 10.8.0.1 - 10.8.0.2 - 10.8.0.3 ## Etcd service. If using kubeEtcd.endpoints only the port and targetPort are used ## service: enabled: true port: 2379 targetPort: 2379 # selector: # component: etcd serviceMonitor: scheme: https insecureSkipVerify: false serverName: etcd.kube-system.svc # this is per kubespray default install caFile: /etc/prometheus/secrets/etcd-client-cert/ca.pem certFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-prometheus.crt keyFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-prometheus.key prometheus: prometheusSpec: secrets: ["etcd-client-cert"]
Apply the new values, for example with the command:
helm upgrade -f values.yml prometheus prometheus-community/kube-prometheus-stack
Check your targets in prometheus and your etcd dashboard in grafana.