pve admin tooling

proxmox/README.md

@@ -1,6 +1,32 @@
 # Proxmox Virtual Environment
+User-facing docs: https://wiki.k-space.ee/en/hosting/proxmox
 
-## K-Space Hyper Converged CEPH setup
+## Adding new node
+1. Upgrade existing nodes.
+1. Install new nodes:
+  - Hostname `pveXX.proxmox.infra.k-space.ee`
+  - Boot disk ZRAID-1
+  - 172.21 or DHCP may be used as initial IP. Installer configuration will be overwritten by cluster join and ansible.
+1. Add `non-free-firmware` as component to `/etc/apt/sources.list` to debian (not PVE) bookworm, bookworm-updates, bookworm-security (next to `main` and `contrib`)
+1. Upgrade new nodes
+  - (unsure if needed nowdays: disabling pve-enterprise, and enabling pve-no-subscription)
+1. Add new node to DNS (secretspace/ns1) and Ansible.
+1. Apply Ansible and reboot.
+1. `$ systemctl status watchdog-mux` should say `Watchdog driver 'IPMI', version 1` and NOT `Software Watchdog`
+1. Join to cluster in UI → Datacenter.
+  - IP to use is the last, ipv6 with vmbr0 <!-- TODO: might have changed -->
+1. `$ passwd` on new node
+1. `$ vim ~/.ssh/authorized_keys` → sort the new key. **Keys are managed manually** since PVE manages the file as well.
+
+TODO: prometheus node exporter
+TODO: create-external-cluster-resources.py in pve90
+TODO: PVE backup server. We want local snapshots and offsite.
+TODO: reinstate restic for /etc and /root
+TODO: d12 discard
+
+## K-SPACE Hyper-Converged CEPH setup
+> [!WARNING]
+> K-SPACE kubernetes uses PVE's CEPH cluster, k8s pools are not visible in general PVE UI.
 
 1. Configure a mesh network