69 lines
1.8 KiB
Markdown
69 lines
1.8 KiB
Markdown
|
# Intro
|
||
|
|
|
||
|
|
This is how we make backups of Mikrotik device configurations using Kubernetes
|
||
|
|
Cronjob. This is easy to monitor with Prometheus and integrates well with the
|
||
|
|
rest of our montioring system. Also the script/manifest is less than 100 lines,
|
||
|
|
easy to follow and to fix.
|
||
|
|
|
||
|
|
Note that this does not have anything to do with
|
||
|
|
[ecadlabs/rosdump](https://github.com/ecadlabs/rosdump)
|
||
|
|
we initially used which just generated empty commits and
|
||
|
|
there was no easy way to monitor.
|
||
|
|
|
||
|
|
We also considered [ytti/oxidized](https://github.com/ytti/oxidized),
|
||
|
|
but it does not export Prometheus metrics either.
|
||
|
|
|
||
|
|
|
||
|
|
# Deployment
|
||
|
|
|
||
|
|
To apply changes run in this directory:
|
||
|
|
|
||
|
|
```
|
||
|
|
kubectl apply -n rosdump -f cronjob.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
To trigger cronjob:
|
||
|
|
|
||
|
|
```
|
||
|
|
kubectl create job -n rosdump --from=cronjob/rosdump-cronjob rosdump-job-oneshot
|
||
|
|
```
|
||
|
|
|
||
|
|
For alerting:
|
||
|
|
|
||
|
|
```
|
||
|
|
absent(kube_cronjob_status_last_successful_time{cronjob="rosdump-cronjob"})
|
||
|
|
```
|
||
|
|
|
||
|
|
# Updating SSH public keys
|
||
|
|
|
||
|
|
Whenever Mikrotik targets are added/removed or if their SSH keys change,
|
||
|
|
use following to apply changes:
|
||
|
|
|
||
|
|
```
|
||
|
|
(for j in $(kubectl get cm -n rosdump rosdump-config -o json | jq -r '.data.targets'); do ssh-keyscan -t rsa $j; done) > ssh_known_hosts
|
||
|
|
kubectl delete -n rosdump configmap rosdump-known-hosts
|
||
|
|
kubectl create -n rosdump configmap rosdump-known-hosts --from-file=ssh_known_hosts
|
||
|
|
```
|
||
|
|
|
||
|
|
Make sure strong crypto is enabled on Mikrotik side:
|
||
|
|
|
||
|
|
```
|
||
|
|
/ip ssh set strong-crypto=yes allow-none-crypto=no
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
# Replacing SSH private key
|
||
|
|
|
||
|
|
This affects access to both Gitea and Mikrotik targets.
|
||
|
|
|
||
|
|
Generate new key and inject it to Kubernetes cluster:
|
||
|
|
|
||
|
|
```
|
||
|
|
rm -fv rosdump
|
||
|
|
ssh-keygen -P '' -b 2048 -m PEM -t rsa -f rosdump -C rosdump
|
||
|
|
kubectl delete -n rosdump secret rosdump-secrets
|
||
|
|
kubectl create -n rosdump secret generic rosdump-secrets --from-file=ssh_identity=rosdump
|
||
|
|
```
|
||
|
|
|
||
|
|
Proceed to replace the public key in Gitea with one from `rosdump.pub`
|