kube/rosdump/README.md

# Intro

This is how we make backups of Mikrotik device configurations using Kubernetes
Cronjob. This is easy to monitor with Prometheus and integrates well with the
rest of our montioring system. Also the script/manifest is less than 100 lines,
easy to follow and to fix.

Note that this does not have anything to do with
[ecadlabs/rosdump](https://github.com/ecadlabs/rosdump)
we initially used which just generated empty commits and
there was no easy way to monitor.

We also considered [ytti/oxidized](https://github.com/ytti/oxidized),
but it does not export Prometheus metrics either.


# Deployment

To apply changes run in this directory:

```
kubectl apply -n rosdump -f application.yml
```

To trigger cronjob:

```
kubectl create job -n rosdump --from=cronjob/rosdump-cronjob rosdump-job-oneshot
```

For alerting:

```
absent(kube_cronjob_status_last_successful_time{cronjob="rosdump-cronjob"})
```

# Updating SSH public keys

Whenever Mikrotik targets are added/removed or if their SSH keys change,
use following to apply changes:

```
(for j in $(kubectl get cm -n rosdump rosdump-config -o json | jq -r '.data.targets');  do ssh-keyscan -t rsa $j; done) > ssh_known_hosts
kubectl delete -n rosdump configmap rosdump-known-hosts
kubectl create -n rosdump configmap rosdump-known-hosts --from-file=ssh_known_hosts
```

Make sure strong crypto is enabled on Mikrotik side:

```
/ip ssh set strong-crypto=yes allow-none-crypto=no
```


# Replacing SSH private key

This affects access to both Gitea and Mikrotik targets.

Generate new key and inject it to Kubernetes cluster:

```
rm -fv rosdump
ssh-keygen -P '' -b 2048 -m PEM -t rsa -f rosdump -C rosdump
kubectl delete -n rosdump secret rosdump-secrets
kubectl create -n rosdump secret generic rosdump-secrets --from-file=ssh_identity=rosdump
```

Proceed to replace the public key in Gitea with one from `rosdump.pub`