kube/rosdump/README.md

Intro

This is how we make backups of Mikrotik device configurations using Kubernetes Cronjob. This is easy to monitor with Prometheus and integrates well with the rest of our montioring system. Also the script/manifest is less than 100 lines, easy to follow and to fix.

Note that this does not have anything to do with ecadlabs/rosdump we initially used which just generated empty commits and there was no easy way to monitor.

We also considered ytti/oxidized, but it does not export Prometheus metrics either.

Deployment

To apply changes run in this directory:

kubectl apply -n rosdump -f cronjob.yaml

To trigger cronjob:

kubectl create job -n rosdump --from=cronjob/rosdump-cronjob rosdump-job-oneshot

For alerting:

absent(kube_cronjob_status_last_successful_time{cronjob="rosdump-cronjob"})

Updating SSH public keys

Whenever Mikrotik targets are added/removed or if their SSH keys change, use following to apply changes:

(for j in $(kubectl get cm -n rosdump rosdump-config -o json | jq -r '.data.targets');  do ssh-keyscan -t rsa $j; done) > ssh_known_hosts
kubectl delete -n rosdump configmap rosdump-known-hosts
kubectl create -n rosdump configmap rosdump-known-hosts --from-file=ssh_known_hosts

Make sure strong crypto is enabled on Mikrotik side:

/ip ssh set strong-crypto=yes allow-none-crypto=no

Replacing SSH private key

This affects access to both Gitea and Mikrotik targets.

Generate new key and inject it to Kubernetes cluster:

rm -fv rosdump
ssh-keygen -P '' -b 2048 -m PEM -t rsa -f rosdump -C rosdump
kubectl delete -n rosdump secret rosdump-secrets
kubectl create -n rosdump secret generic rosdump-secrets --from-file=ssh_identity=rosdump

Proceed to replace the public key in Gitea with one from rosdump.pub