k-space/kube
10
1
Fork 3
Code Issues 31 Pull Requests Activity
cb41b739cc
kube/README.md

3.9 KiB
Raw Blame History

k-space.ee infrastructure

Kubernetes manifests, Ansible playbooks, and documentation for K-SPACE services.

Jump to docs: inventory-app / cameras / doors / list of apps // all infra / network / retro / non-infra

Tip: Search the repo for kind: xyz for examples.

Supporting services

  • Build Git repositories with Woodpecker.
  • Passmower: Authz with kind: OIDCClient (or kind: OIDCMiddlewareClient1).
  • Traefik2: Expose services with kind: Service + kind: Ingress (TLS and DNS included).

Additional

  • bind: Manage additional DNS records with kind: DNSEndpoint.
  • Prometheus: Collect metrics with kind: PodMonitor (alerts with kind: PrometheusRule).
  • Slack bots and Kubernetes CLUSTER.md itself.

Network

All nodes are in Infra VLAN 21. Routing is implemented with BGP, all nodes and the router make a full-mesh. Both Serice LB IPs and Pod IPs are advertised to the router. Router does NAT for outbound pod traffic. See the Calico installation for Kube side and Routing / BGP in the router. Static routes for 193.40.103.36/30 have been added in pve nodes to make them communicating with Passmower via Traefik more stable - otherwise packets coming back to the PVE are routed directly via VLAN 21 internal IPs by the worker nodes, breaking TCP.

Databases / -stores:

  • KeyDB: kind: KeydbClaim (replaces Redis3)
  • Dragonfly: kind: Dragonfly (replaces Redis3)
  • Longhorn: storageClassName: longhorn (filesystem storage)
  • Mongo4: kind: MongoDBCommunity (NAS* inventory-mongodb)
  • Minio S3: kind: MinioBucketClaim with class: dedicated (NAS*: class: external)
  • MariaDB*: search for mysql, mariadb5 (replaces MySQL)
  • Postgres*: hardcoded to harbor/application.yml

* External, hosted directly on nas.k-space.ee

This page is referenced by wiki front page as the technical documentation for infra.

  1. Applications should use OpenID Connect (kind: OIDCClient) for authentication, whereever possible. If not possible, use kind: OIDCMiddlewareClient client, which will provide authentication via a Traefik middleware (traefik.ingress.kubernetes.io/router.middlewares: passmower-proxmox@kubernetescrd). Sometimes you might use both for extra security. ↩︎

  2. No nginx annotations! Use kind: Ingress instead. IngressRoute is not used as it doesn't support external-dns out of the box. ↩︎

  3. Redis has been replaced as redis-operatori couldn't handle itself: didn't reconcile after reboots, master URI was empty, and clients complained about missing masters. ArgoCD still hosts its own Redis. ↩︎

  4. Mongo problems: Incompatible with rawfile csi (wiredtiger.wt corrupts), complicated resizing (PVCs from statefulset PVC template). ↩︎

  5. As of 2024-07-30 used by auth, authelia, bitwarden, etherpad, freescout, git, grafana, nextcloud, wiki, woodpecker ↩︎