Kubernetes manifests of services running on k-space.ee domains (mirrored to https://gitlab.com/k-space/kube)
Go to file
2022-09-17 08:06:42 +03:00
argocd argocd: Add members repo 2022-09-17 08:06:19 +03:00
authelia authelia: Switch to KeyDB operator 2022-08-28 11:17:30 +03:00
camtiler camtiler: Formatting, networkpolicy and labeling fixes 2022-09-15 10:12:07 +03:00
cert-manager Initial commit 2022-08-25 11:22:50 +03:00
drone Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
drone-execution Initial commit 2022-08-25 11:22:50 +03:00
elastic-system Reduce Filebeat logging verbosity 2022-09-17 08:06:42 +03:00
etherpad Initial commit 2022-08-25 11:22:50 +03:00
external-dns Initial commit 2022-08-25 11:22:50 +03:00
freescout Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
harbor Initial commit 2022-08-25 11:22:50 +03:00
keel Initial commit 2022-08-25 11:22:50 +03:00
kube-system Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
kubernetes-dashboard Initial commit 2022-08-25 11:22:50 +03:00
local-path-storage Initial commit 2022-08-25 11:22:50 +03:00
logging Reduce Filebeat logging verbosity 2022-09-17 08:06:42 +03:00
longhorn-system Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
meta-operator Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
metallb-system Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
mongodb-operator Initial commit 2022-08-25 11:22:50 +03:00
mysql-operator mysql-operator: Bump to version 8.0.30-2.0.6 2022-09-16 08:41:07 +03:00
phpmyadmin Initial commit 2022-08-25 11:22:50 +03:00
prometheus-operator prometheus-operator: Fix node exporter formatting 2022-09-12 07:44:06 +03:00
reloader Initial commit 2022-08-25 11:22:50 +03:00
rosdump Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
shared Remove symlink method of adding Redis/KeyDB instances 2022-08-28 11:24:09 +03:00
tigera-operator Initial commit 2022-08-25 11:22:50 +03:00
traefik Migrate to Prometheus Operator 2022-09-11 16:38:16 +03:00
wildduck Remove symlink method of adding Redis/KeyDB instances 2022-08-28 11:24:09 +03:00
.drone.yml Initial commit 2022-08-25 11:22:50 +03:00
.gitignore Initial commit 2022-08-25 11:22:50 +03:00
cluster-role-bindings.yml Initial commit 2022-08-25 11:22:50 +03:00
CONTRIBUTORS.md Initial commit 2022-08-25 11:22:50 +03:00
LICENSE.md Initial commit 2022-08-25 11:22:50 +03:00
README.md Initial commit 2022-08-25 11:22:50 +03:00

Kubernetes cluster manifests

Introduction

This is the Kubernetes manifests of services running on k-space.ee domains:

Most endpoints are protected by OIDC autentication or Authelia SSO middleware.

Cluster access

General discussion is happening in the #kube Slack channel.

For bootstrap access obtain /etc/kubernetes/admin.conf from one of the master nodes and place it under ~/.kube/config on your machine.

Once Authelia is working, OIDC access for others can be enabled with running following on Kubernetes masters:

patch /etc/kubernetes/manifests/kube-apiserver.yaml - << EOF
@@ -23,6 +23,10 @@
     - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
     - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
     - --etcd-servers=https://127.0.0.1:2379
+    - --oidc-issuer-url=https://auth.k-space.ee
+    - --oidc-client-id=kubelogin
+    - --oidc-username-claim=preferred_username
+    - --oidc-groups-claim=groups
     - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
     - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
     - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
EOF
sudo systemctl daemon-reload
systemctl restart kubelet

Afterwards following can be used to talk to the Kubernetes cluster using OIDC credentials:

kubectl krew install oidc-login
mkdir -p ~/.kube
cat << EOF > ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EVXdNakEzTXpVMU1Wb1hEVE15TURReU9UQTNNelUxTVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2J2CjY3UFlXVHJMc3ZCQTZuWHUvcm55SlVhNnppTnNWTVN6N2w4ekhxM2JuQnhqWVNPUDJhN1RXTnpUTmZDanZBWngKTmlNbXJya1hpb2dYQWpVVkhSUWZlYm81TFIrb0JBOTdLWlcrN01UMFVJRXBuWVVaaTdBRHlaS01vcEJFUXlMNwp1SlU5UDhnNUR1T29FRHZieGJSMXFuV1JZRXpteFNmSFpocllpMVA3bFd4emkxR243eGRETFZaMjZjNm0xR3Y1CnViRjZyaFBXK1JSVkhiQzFKakJGeTBwRXdhYlUvUTd0Z2dic0JQUjk5NVZvMktCeElBelRmbHhVanlYVkJ3MjEKU2d3ZGI1amlpemxEM0NSbVdZZ0ZrRzd0NTVZeGF3ZmpaQjh5bW4xYjhUVjkwN3dRcG8veU8zM3RaaEE3L3BFUwpBSDJYeDk5bkpMbFVGVUtSY1A4Q0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKNnZKeVk1UlJ1aklQWGxIK2ZvU3g2QzFRT2RNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ04zcGtCTVM3ekkrbUhvOWdTZQp6SzdXdjl3bXlCTVE5Q3crQXBSNnRBQXg2T1VIN0d1enc5TTV2bXNkYjkrYXBKMHBlZFB4SUg3YXZ1aG9SUXNMCkxqTzRSVm9BMG9aNDBZV3J3UStBR0dvdkZuaWNleXRNcFVSNEZjRXc0ZDRmcGl6V3d0TVNlRlRIUXR6WG84V2MKNFJGWC9xUXNVR1NWa01PaUcvcVVrSFpXQVgyckdhWXZ1Tkw2eHdSRnh5ZHpsRTFSUk56TkNvQzVpTXhjaVRNagpackEvK0pqVEFWU2FuNXZnODFOSmthZEphbmNPWmEwS3JEdkZzd1JJSG5CMGpMLzh3VmZXSTV6czZURU1VZUk1ClF6dU01QXUxUFZ4VXZJUGhlMHl6UXZjWDV5RlhnMkJGU3MzKzJBajlNcENWVTZNY2dSSTl5TTRicitFTUlHL0kKY0pjPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://master.kube.k-space.ee:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: oidc
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://auth.k-space.ee
      - --oidc-client-id=kubelogin
      - --oidc-use-pkce
      - --oidc-extra-scope=profile,email,groups
      - --listen-address=127.0.0.1:27890
      command: kubectl
      env: null
      provideClusterInfo: false
EOF

For access control mapping see cluster-role-bindings.yml

Technology mapping

Our self-hosted Kubernetes stack compared to AWS based deployments:

Hipster startup Self-hosted hackerspace Purpose
AWS EC2 Proxmox Virtualization layer
AWS EKS kubeadm Provision Kubernetes master nodes
AWS EBS Longhorn Block storage for arbitrary applications needing persistent storage
AWS NLB MetalLB L2/L3 level load balancing
AWS ALB Traefik Reverse proxy also known as ingress controller in Kubernetes jargon
AWS ECR Harbor Docker registry
AWS DocumentDB MongoDB NoSQL database
AWS S3 Minio Object storage
GitHub OAuth2 Samba (Active Directory compatible) Source of truth for authentication and authorization
Dex Authelia ACL mapping and OIDC provider which integrates with GitHub/Samba
GitHub Gitea Source code management, issue tracking
GitHub Actions Drone Build Docker images
Gmail Wildduck E-mail
AWS Route53 Bind and RFC2136 DNS records and Let's Encrypt DNS validation
AWS VPC Calico Overlay network

External dependencies running as classic virtual machines:

  • Samba as Authelia's source of truth
  • Bind as DNS server

Adding applications

Deploy applications via ArgoCD

We use Treafik with Authelia for Ingress. Applications where possible and where applicable should use Remote-User authentication. This prevents application exposure on public Internet. Otherwise use OpenID Connect for authentication, see Argo itself as an example how that is done.

See kspace-camtiler/ingress.yml for commented Ingress example.

Note that we do not use IngressRoute objects because they don't support external-dns out of the box. Do NOT add nginx annotations, we use Traefik. Do NOT manually add DNS records, they are added by external-dns. Do NOT manually create Certificate objects, these should be handled by tls: section in Ingress.

Cluster formation

Create Ubuntu 20.04 VM-s on Proxmox with local storage.

After machines have booted up and you can reach them via SSH:

# Enable required kernel modules
cat > /etc/modules << EOF
overlay
br_netfilter
EOF
cat /etc/modules | xargs -L 1 -t modprobe

# Finetune sysctl:
cat > /etc/sysctl.d/99-k8s.conf << EOF
net.ipv4.conf.all.accept_redirects  = 0
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system

# Disable Ubuntu caching DNS resolver
systemctl disable systemd-resolved.service
systemctl stop systemd-resolved
rm -fv /etc/resolv.conf
cat > /etc/resolv.conf << EOF
nameserver 1.1.1.1
nameserver 8.8.8.8
EOF

# Disable multipathd as Longhorn handles that itself
systemctl mask multipathd
systemctl disable multipathd
systemctl stop multipathd

# Disable Snapcraft
systemctl mask snapd
systemctl disable snapd
systemctl stop snapd

# Permit root login
sed -i -e 's/PermitRootLogin no/PermitRootLogin without-password/' /etc/ssh/sshd_config
systemctl reload ssh
cat << EOF > /root/.ssh/authorized_keys
sk-ecdsa-sha2-nistp256@openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBD4/e9SWYWYoNZMkkF+NirhbmHuUgjoCap42kAq0pLIXFwIqgVTCre03VPoChIwBClc8RspLKqr5W3j0fG8QwnQAAAAEc3NoOg== lauri@lauri-x13
EOF
userdel -f ubuntu
apt-get remove -yq cloud-init


Install packages, for Raspbian set OS=Debian_11

OS=xUbuntu_20.04
VERSION=1.23
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list

apt-get update
apt-get install -yqq apt-transport-https curl cri-o cri-o-runc kubelet=1.23.5-00 kubectl=1.23.5-00 kubeadm=1.23.5-00
sudo systemctl daemon-reload
sudo systemctl enable crio --now
apt-mark hold kubelet kubeadm kubectl
sed -i -e 's/unqualified-search-registries = .*/unqualified-search-registries = ["docker.io"]/' /etc/containers/registries.conf

On master:

kubeadm init --token-ttl=120m --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint "master.kube.k-space.ee:6443" --upload-certs --apiserver-cert-extra-sans master.kube.k-space.ee --node-name master1.kube.k-space.ee

For the kubeadm join command specify FQDN via --node-name $(hostname -f).

After forming the cluster add taints:

for j in $(seq 1 9); do
  kubectl label nodes worker${j}.kube.k-space.ee node-role.kubernetes.io/worker=''
done

for j in $(seq 1 3); do
  kubectl taint nodes mon${j}.kube.k-space.ee dedicated=monitoring:NoSchedule
  kubectl label nodes mon${j}.kube.k-space.ee dedicated=monitoring
done

for j in $(seq 1 4); do
  kubectl taint nodes storage${j}.kube.k-space.ee dedicated=storage:NoSchedule
  kubectl label nodes storage${j}.kube.k-space.ee dedicated=storage
done

On Raspberry Pi you need to take additonal steps:

  • Manually enable cgroups by appending cgroup_memory=1 cgroup_enable=memory to /boot/cmdline.txt,
  • Disable swap with swapoff -a; apt-get purge -y dphys-swapfile
  • For mounting Longhorn volumes on Rasbian install open-iscsi

For arm64 nodes add suitable taint to prevent scheduling non-multiarch images on them:

kubectl taint nodes worker9.kube.k-space.ee arch=arm64:NoSchedule