Initial commit

rosdump/README.md

@@ -0,0 +1,68 @@
+# Intro
+
+This is how we make backups of Mikrotik device configurations using Kubernetes
+Cronjob. This is easy to monitor with Prometheus and integrates well with the
+rest of our montioring system. Also the script/manifest is less than 100 lines,
+easy to follow and to fix.
+
+Note that this does not have anything to do with
+[ecadlabs/rosdump](https://github.com/ecadlabs/rosdump)
+we initially used which just generated empty commits and
+there was no easy way to monitor.
+
+We also considered [ytti/oxidized](https://github.com/ytti/oxidized),
+but it does not export Prometheus metrics either.
+
+
+# Deployment
+
+To apply changes run in this directory:
+
+```
+kubectl apply -n rosdump -f cronjob.yaml
+```
+
+To trigger cronjob:
+
+```
+kubectl create job -n rosdump --from=cronjob/rosdump-cronjob rosdump-job-oneshot
+```
+
+For alerting:
+
+```
+absent(kube_cronjob_status_last_successful_time{cronjob="rosdump-cronjob"})
+```
+
+# Updating SSH public keys
+
+Whenever Mikrotik targets are added/removed or if their SSH keys change,
+use following to apply changes:
+
+```
+(for j in $(kubectl get cm -n rosdump rosdump-config -o json | jq -r '.data.targets');  do ssh-keyscan -t rsa $j; done) > ssh_known_hosts
+kubectl delete -n rosdump configmap rosdump-known-hosts
+kubectl create -n rosdump configmap rosdump-known-hosts --from-file=ssh_known_hosts
+```
+
+Make sure strong crypto is enabled on Mikrotik side:
+
+```
+/ip ssh set strong-crypto=yes allow-none-crypto=no
+```
+
+
+# Replacing SSH private key
+
+This affects access to both Gitea and Mikrotik targets.
+
+Generate new key and inject it to Kubernetes cluster:
+
+```
+rm -fv rosdump
+ssh-keygen -P '' -b 2048 -m PEM -t rsa -f rosdump -C rosdump
+kubectl delete -n rosdump secret rosdump-secrets
+kubectl create -n rosdump secret generic rosdump-secrets --from-file=ssh_identity=rosdump
+```
+
+Proceed to replace the public key in Gitea with one from `rosdump.pub`