From 7c16f84200668df536b3ee10106dcb6c1f6c67f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lauri=20V=C3=B5sandi?= Date: Mon, 12 Aug 2024 22:15:32 +0300 Subject: [PATCH] monitoring: Elaborate more about operator --- monitoring/README.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/monitoring/README.md b/monitoring/README.md index f1ec024..f874b5f 100644 --- a/monitoring/README.md +++ b/monitoring/README.md @@ -1,11 +1,26 @@ -## Monitoring +## Monitoring namespace Prometheus is accessible at [prom.k-space.ee](https://prom.k-space.ee/) and the corresponding AlertManager is accessible at [am.k-space.ee](https://am.k-space.ee/). Both are [deployed by ArgoCD](https://argocd.k-space.ee/applications/monitoring) from this Git repo directory using Prometheus operator. -Alerts are sent to #kube-prod Slack channel +Note that Prometheus and other monitoring stack components should use the +`dedicated: monitoring` Kubernetes node selector to make sure the components +get scheduled on `mon[1-3]` nodes which are hosted in a privileged VLAN where +they have access to UPS SNMP targets, Mikrotik router/switch API-s etc. + +To add monitoring targets inside the Kubernetes cluster make use of +[PodMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#using-podmonitors) or ServiceMonitor custom +resource definitions. + +For external targets (ab)use the Probe CRD as seen in `node-exporter.yaml` +or `ping-exporter.yaml` + +Alerts are sent to #kube-prod Slack channel. The alerting rules are automatically +picked up by Prometheus operator via Kubernetes manifests utilizing +the operator's +[PrometheusRule](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/alerting.md#deploying-prometheus-rules) custom resource definitions. Sample queries: