# Kubernetes cluster manifests ## Introduction This is the Kubernetes manifests of services running on k-space.ee domains: - [Authelia](https://auth.k-space.ee) for authentication - [Drone.io](https://drone.k-space.ee) for building Docker images - [Harbor](https://harbor.k-space.ee) for hosting Docker images - [ArgoCD](https://argocd.k-space.ee) for deploying Kubernetes manifests and Helm charts into the cluster - [camtiler](https://cams.k-space.ee) for cameras - [Longhorn Dashboard](https://longhorn.k-space.ee) for administering Longhorn storage - [Kubernetes Dashboard](https://kubernetes-dashboard.k-space.ee/) for read-only overview of the Kubernetes cluster - [Wildduck Webmail](https://webmail.k-space.ee/) Most endpoints are protected by OIDC autentication or Authelia SSO middleware. ## Cluster access General discussion is happening in the `#kube` Slack channel. For bootstrap access obtain `/etc/kubernetes/admin.conf` from one of the master nodes and place it under `~/.kube/config` on your machine. Once Authelia is working, OIDC access for others can be enabled with running following on Kubernetes masters: ```bash patch /etc/kubernetes/manifests/kube-apiserver.yaml - << EOF @@ -23,6 +23,10 @@ - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 + - --oidc-issuer-url=https://auth.k-space.ee + - --oidc-client-id=kubelogin + - --oidc-username-claim=preferred_username + - --oidc-groups-claim=groups - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname EOF sudo systemctl daemon-reload systemctl restart kubelet ``` Afterwards following can be used to talk to the Kubernetes cluster using OIDC credentials: ```bash kubectl krew install oidc-login mkdir -p ~/.kube cat << EOF > ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EVXdNakEzTXpVMU1Wb1hEVE15TURReU9UQTNNelUxTVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2J2CjY3UFlXVHJMc3ZCQTZuWHUvcm55SlVhNnppTnNWTVN6N2w4ekhxM2JuQnhqWVNPUDJhN1RXTnpUTmZDanZBWngKTmlNbXJya1hpb2dYQWpVVkhSUWZlYm81TFIrb0JBOTdLWlcrN01UMFVJRXBuWVVaaTdBRHlaS01vcEJFUXlMNwp1SlU5UDhnNUR1T29FRHZieGJSMXFuV1JZRXpteFNmSFpocllpMVA3bFd4emkxR243eGRETFZaMjZjNm0xR3Y1CnViRjZyaFBXK1JSVkhiQzFKakJGeTBwRXdhYlUvUTd0Z2dic0JQUjk5NVZvMktCeElBelRmbHhVanlYVkJ3MjEKU2d3ZGI1amlpemxEM0NSbVdZZ0ZrRzd0NTVZeGF3ZmpaQjh5bW4xYjhUVjkwN3dRcG8veU8zM3RaaEE3L3BFUwpBSDJYeDk5bkpMbFVGVUtSY1A4Q0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKNnZKeVk1UlJ1aklQWGxIK2ZvU3g2QzFRT2RNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ04zcGtCTVM3ekkrbUhvOWdTZQp6SzdXdjl3bXlCTVE5Q3crQXBSNnRBQXg2T1VIN0d1enc5TTV2bXNkYjkrYXBKMHBlZFB4SUg3YXZ1aG9SUXNMCkxqTzRSVm9BMG9aNDBZV3J3UStBR0dvdkZuaWNleXRNcFVSNEZjRXc0ZDRmcGl6V3d0TVNlRlRIUXR6WG84V2MKNFJGWC9xUXNVR1NWa01PaUcvcVVrSFpXQVgyckdhWXZ1Tkw2eHdSRnh5ZHpsRTFSUk56TkNvQzVpTXhjaVRNagpackEvK0pqVEFWU2FuNXZnODFOSmthZEphbmNPWmEwS3JEdkZzd1JJSG5CMGpMLzh3VmZXSTV6czZURU1VZUk1ClF6dU01QXUxUFZ4VXZJUGhlMHl6UXZjWDV5RlhnMkJGU3MzKzJBajlNcENWVTZNY2dSSTl5TTRicitFTUlHL0kKY0pjPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://master.kube.k-space.ee:6443 name: kubernetes contexts: - context: cluster: kubernetes user: oidc name: default current-context: default kind: Config preferences: {} users: - name: oidc user: exec: apiVersion: client.authentication.k8s.io/v1beta1 args: - oidc-login - get-token - --oidc-issuer-url=https://auth.k-space.ee - --oidc-client-id=kubelogin - --oidc-use-pkce - --oidc-extra-scope=profile,email,groups - --listen-address=127.0.0.1:27890 command: kubectl env: null provideClusterInfo: false EOF ``` For access control mapping see [cluster-role-bindings.yml](cluster-role-bindings.yml) # Technology mapping Our self-hosted Kubernetes stack compared to AWS based deployments: | Hipster startup | Self-hosted hackerspace | Purpose | |-----------------|-------------------------------------|---------------------------------------------------------------------| | AWS EC2 | Proxmox | Virtualization layer | | AWS EKS | kubeadm | Provision Kubernetes master nodes | | AWS EBS | Longhorn | Block storage for arbitrary applications needing persistent storage | | AWS NLB | MetalLB | L2/L3 level load balancing | | AWS ALB | Traefik | Reverse proxy also known as ingress controller in Kubernetes jargon | | AWS ECR | Harbor | Docker registry | | AWS DocumentDB | MongoDB | NoSQL database | | AWS S3 | Minio | Object storage | | GitHub OAuth2 | Samba (Active Directory compatible) | Source of truth for authentication and authorization | | Dex | Authelia | ACL mapping and OIDC provider which integrates with GitHub/Samba | | GitHub | Gitea | Source code management, issue tracking | | GitHub Actions | Drone | Build Docker images | | Gmail | Wildduck | E-mail | | AWS Route53 | Bind and RFC2136 | DNS records and Let's Encrypt DNS validation | | AWS VPC | Calico | Overlay network | External dependencies running as classic virtual machines: - Samba as Authelia's source of truth - Bind as DNS server ## Adding applications Deploy applications via [ArgoCD](https://argocd.k-space.ee) We use Treafik with Authelia for Ingress. Applications where possible and where applicable should use `Remote-User` authentication. This prevents application exposure on public Internet. Otherwise use OpenID Connect for authentication, see Argo itself as an example how that is done. See `kspace-camtiler/ingress.yml` for commented Ingress example. Note that we do not use IngressRoute objects because they don't support `external-dns` out of the box. Do NOT add nginx annotations, we use Traefik. Do NOT manually add DNS records, they are added by `external-dns`. Do NOT manually create Certificate objects, these should be handled by `tls:` section in Ingress. ## Cluster formation Create Ubuntu 20.04 VM-s on Proxmox with local storage. After machines have booted up and you can reach them via SSH: ```bash # Enable required kernel modules cat > /etc/modules << EOF overlay br_netfilter EOF cat /etc/modules | xargs -L 1 -t modprobe # Finetune sysctl: cat > /etc/sysctl.d/99-k8s.conf << EOF net.ipv4.conf.all.accept_redirects = 0 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sysctl --system # Disable Ubuntu caching DNS resolver systemctl disable systemd-resolved.service systemctl stop systemd-resolved rm -fv /etc/resolv.conf cat > /etc/resolv.conf << EOF nameserver 1.1.1.1 nameserver 8.8.8.8 EOF # Disable multipathd as Longhorn handles that itself systemctl mask multipathd systemctl disable multipathd systemctl stop multipathd # Disable Snapcraft systemctl mask snapd systemctl disable snapd systemctl stop snapd # Permit root login sed -i -e 's/PermitRootLogin no/PermitRootLogin without-password/' /etc/ssh/sshd_config systemctl reload ssh cat << EOF > /root/.ssh/authorized_keys sk-ecdsa-sha2-nistp256@openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBD4/e9SWYWYoNZMkkF+NirhbmHuUgjoCap42kAq0pLIXFwIqgVTCre03VPoChIwBClc8RspLKqr5W3j0fG8QwnQAAAAEc3NoOg== lauri@lauri-x13 EOF userdel -f ubuntu apt-get remove -yq cloud-init ``` Install packages, for Raspbian set `OS=Debian_11` ```bash OS=xUbuntu_20.04 VERSION=1.23 cat < /etc/apt/sources.list.d/kubernetes.list apt-get update apt-get install -yqq apt-transport-https curl cri-o cri-o-runc kubelet=1.23.5-00 kubectl=1.23.5-00 kubeadm=1.23.5-00 sudo systemctl daemon-reload sudo systemctl enable crio --now apt-mark hold kubelet kubeadm kubectl sed -i -e 's/unqualified-search-registries = .*/unqualified-search-registries = ["docker.io"]/' /etc/containers/registries.conf ``` On master: ``` kubeadm init --token-ttl=120m --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint "master.kube.k-space.ee:6443" --upload-certs --apiserver-cert-extra-sans master.kube.k-space.ee --node-name master1.kube.k-space.ee ``` For the `kubeadm join` command specify FQDN via `--node-name $(hostname -f)`. After forming the cluster add taints: ```bash for j in $(seq 1 9); do kubectl label nodes worker${j}.kube.k-space.ee node-role.kubernetes.io/worker='' done for j in $(seq 1 3); do kubectl taint nodes mon${j}.kube.k-space.ee dedicated=monitoring:NoSchedule kubectl label nodes mon${j}.kube.k-space.ee dedicated=monitoring done for j in $(seq 1 4); do kubectl taint nodes storage${j}.kube.k-space.ee dedicated=storage:NoSchedule kubectl label nodes storage${j}.kube.k-space.ee dedicated=storage done ``` On Raspberry Pi you need to take additonal steps: * Manually enable cgroups by appending `cgroup_memory=1 cgroup_enable=memory` to `/boot/cmdline.txt`, * Disable swap with `swapoff -a; apt-get purge -y dphys-swapfile` * For mounting Longhorn volumes on Rasbian install `open-iscsi` For `arm64` nodes add suitable taint to prevent scheduling non-multiarch images on them: ```bash kubectl taint nodes worker9.kube.k-space.ee arch=arm64:NoSchedule ```