k-space.ee infrastructure

Kubernetes manifests, Ansible playbooks, and documentation for K-SPACE services.

• Repo is deployed with ArgoCD. For kubectl access, see CLUSTER.md.
• Debugging Kubernetes on Wiki
• Need help? → #kube

Jump to docs: inventory-app / cameras / doors / list of apps // all infra / network / retro / non-infra

Tip: Search the repo for kind: xyz for examples.

Supporting services

• Build Git repositories with Woodpecker¹ .
• Passmower: Authz with kind: OIDCClient (or kind: OIDCMiddlewareClient² ).
• Traefik³ : Expose services with kind: Service + kind: Ingress (TLS and DNS included).

Additional

• bind: Manage additional DNS records with kind: DNSEndpoint.
• Prometheus: Collect metrics with kind: PodMonitor (alerts with kind: PrometheusRule).
• Slack bots and Kubernetes CLUSTER.md itself.

Network

All nodes are in Infra VLAN 21. Routing is implemented with BGP, all nodes and the router make a full-mesh. Both Serice LB IPs and Pod IPs are advertised to the router. Router does NAT for outbound pod traffic. See the Calico installation for Kube side and Routing / BGP in the router. Static routes for 193.40.103.36/30 have been added in pve nodes to make them communicating with Passmower via Traefik more stable - otherwise packets coming back to the PVE are routed directly via VLAN 21 internal IPs by the worker nodes, breaking TCP.

Regenerate networkpolicy-base.yml

It's quite odd there is no better way to generate these.

regenerate-networkpolicy-base.sh

Databases / -stores:

• Dragonfly: kind: Dragonfly (replaces Redis⁴ )
• External (hyperconverged Proxmox) Rook/Ceph: rook-ceph/storage-classes.yaml (filesystem storage)⁵
• Mongo⁶ : kind: MongoDBCommunity (NAS* inventory-mongodb)
• Garage S3⁷ : buckets/credentials created with CLI and usually stored in secretspace/kube #TODO: link to docs, kube claim instead?
• MariaDB*: search for mysql, mariadb⁸ (replaces MySQL)
• Postgres*: hardcoded to harbor/application.yml
• Seeded secrets: kind: SecretClaim (generates random secret in templated format)
• Secrets in git: https://git.k-space.ee/secretspace (members personal info, API credentials, see argocd/deploy_key.pub comment)

---

This page is referenced by wiki front page as the technical documentation for infra.

---

1. Replaces Drone CI. ↩︎

2. Applications should use OpenID Connect (kind: OIDCClient) for authentication, whereever possible. If not possible, use kind: OIDCMiddlewareClient client, which will provide authentication via a Traefik middleware (traefik.ingress.kubernetes.io/router.middlewares: passmower-proxmox@kubernetescrd). Sometimes you might use both for extra security. ↩︎

3. No nginx annotations! Use kind: Ingress instead. IngressRoute is not used as it doesn't support external-dns out of the box. ↩︎

4. Redis has been replaced as redis-operatori couldn't handle itself: didn't reconcile after reboots, master URI was empty, and clients complained about missing masters. Dragonfly replaces KeyDB. ↩︎

5. Replaces Longhorn and proxmox-csi. ↩︎

6. Mongo problems: Incompatible with rawfile csi (wiredtiger.wt corrupts), complicated resizing (PVCs from statefulset PVC template). ↩︎

7. Replaces Minio S3. ↩︎

8. As of 2024-07-30 used by auth, authelia, bitwarden, etherpad, freescout, git, grafana, nextcloud, wiki, woodpecker ↩︎