kube/monitoring
2024-08-21 07:20:32 +03:00
..
alertmanager.yaml monitoring: Move to topologySpreadConstraints 2024-08-21 07:11:06 +03:00
blackbox-exporter.yaml monitoring: Bump versions, use gcr mirror 2024-08-21 07:17:05 +03:00
mikrotik-exporter.yaml monitoring: Move to topologySpreadConstraints 2024-08-21 07:11:06 +03:00
node-exporter.yaml monitoring: Use gcr mirror for node exporter 2024-08-21 07:19:15 +03:00
ping-exporter.yaml monitoring: Bump versions, use gcr mirror 2024-08-21 07:17:05 +03:00
prometheus.yaml monitoring: Move to topologySpreadConstraints 2024-08-21 07:11:06 +03:00
README.md monitoring: Elaborate more about operator 2024-08-12 22:15:32 +03:00
snmp-configs.yaml Move Prometheus instance to monitoring namespace 2023-08-19 09:24:48 +03:00
snmp-exporter.yaml monitoring: Revert snmp-exporter because config file needs to be updated 2024-08-21 07:20:32 +03:00
zrepl.yaml Move Ansible directory to separate repo 2024-08-12 21:41:36 +03:00

Monitoring namespace

Prometheus is accessible at prom.k-space.ee and the corresponding AlertManager is accessible at am.k-space.ee. Both are deployed by ArgoCD from this Git repo directory using Prometheus operator.

Note that Prometheus and other monitoring stack components should use the dedicated: monitoring Kubernetes node selector to make sure the components get scheduled on mon[1-3] nodes which are hosted in a privileged VLAN where they have access to UPS SNMP targets, Mikrotik router/switch API-s etc.

To add monitoring targets inside the Kubernetes cluster make use of PodMonitor or ServiceMonitor custom resource definitions.

For external targets (ab)use the Probe CRD as seen in node-exporter.yaml or ping-exporter.yaml

Alerts are sent to #kube-prod Slack channel. The alerting rules are automatically picked up by Prometheus operator via Kubernetes manifests utilizing the operator's PrometheusRule custom resource definitions.

Sample queries:

To reconfigure SNMP targets etc:

kubectl delete -n monitoring configmap snmp-exporter
kubectl create -n monitoring configmap snmp-exporter --from-file=snmp.yml=snmp-configs.yaml

To set Slack secrets:

 kubectl create -n monitoring secret generic slack-secrets \
    --from-literal=webhook-url=https://hooks.slack.com/services/...

To set Mikrotik secrets:

 kubectl create -n monitoring secret generic mikrotik-exporter \
  --from-literal=MIKROTIK_PASSWORD='f7W!H*Pu' \
  --from-literal=PROMETHEUS_BEARER_TOKEN=$(cat /dev/urandom | base64 | head -c 30)