forked from k-space/kube
Initial commit
This commit is contained in:
258
README.md
Normal file
258
README.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# Kubernetes cluster manifests
|
||||
|
||||
## Introduction
|
||||
|
||||
This is the Kubernetes manifests of services running on k-space.ee domains:
|
||||
|
||||
- [Authelia](https://auth.k-space.ee) for authentication
|
||||
- [Drone.io](https://drone.k-space.ee) for building Docker images
|
||||
- [Harbor](https://harbor.k-space.ee) for hosting Docker images
|
||||
- [ArgoCD](https://argocd.k-space.ee) for deploying Kubernetes manifests and
|
||||
Helm charts into the cluster
|
||||
- [camtiler](https://cams.k-space.ee) for cameras
|
||||
- [Longhorn Dashboard](https://longhorn.k-space.ee) for administering
|
||||
Longhorn storage
|
||||
- [Kubernetes Dashboard](https://kubernetes-dashboard.k-space.ee/) for read-only overview
|
||||
of the Kubernetes cluster
|
||||
- [Wildduck Webmail](https://webmail.k-space.ee/)
|
||||
|
||||
Most endpoints are protected by OIDC autentication or Authelia SSO middleware.
|
||||
|
||||
|
||||
## Cluster access
|
||||
|
||||
General discussion is happening in the `#kube` Slack channel.
|
||||
|
||||
For bootstrap access obtain `/etc/kubernetes/admin.conf` from one of the master
|
||||
nodes and place it under `~/.kube/config` on your machine.
|
||||
|
||||
Once Authelia is working, OIDC access for others can be enabled with
|
||||
running following on Kubernetes masters:
|
||||
|
||||
```bash
|
||||
patch /etc/kubernetes/manifests/kube-apiserver.yaml - << EOF
|
||||
@@ -23,6 +23,10 @@
|
||||
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
|
||||
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
|
||||
- --etcd-servers=https://127.0.0.1:2379
|
||||
+ - --oidc-issuer-url=https://auth.k-space.ee
|
||||
+ - --oidc-client-id=kubelogin
|
||||
+ - --oidc-username-claim=preferred_username
|
||||
+ - --oidc-groups-claim=groups
|
||||
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
|
||||
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
|
||||
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
|
||||
EOF
|
||||
sudo systemctl daemon-reload
|
||||
systemctl restart kubelet
|
||||
```
|
||||
|
||||
Afterwards following can be used to talk to the Kubernetes cluster using
|
||||
OIDC credentials:
|
||||
|
||||
```bash
|
||||
kubectl krew install oidc-login
|
||||
mkdir -p ~/.kube
|
||||
cat << EOF > ~/.kube/config
|
||||
apiVersion: v1
|
||||
clusters:
|
||||
- cluster:
|
||||
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EVXdNakEzTXpVMU1Wb1hEVE15TURReU9UQTNNelUxTVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2J2CjY3UFlXVHJMc3ZCQTZuWHUvcm55SlVhNnppTnNWTVN6N2w4ekhxM2JuQnhqWVNPUDJhN1RXTnpUTmZDanZBWngKTmlNbXJya1hpb2dYQWpVVkhSUWZlYm81TFIrb0JBOTdLWlcrN01UMFVJRXBuWVVaaTdBRHlaS01vcEJFUXlMNwp1SlU5UDhnNUR1T29FRHZieGJSMXFuV1JZRXpteFNmSFpocllpMVA3bFd4emkxR243eGRETFZaMjZjNm0xR3Y1CnViRjZyaFBXK1JSVkhiQzFKakJGeTBwRXdhYlUvUTd0Z2dic0JQUjk5NVZvMktCeElBelRmbHhVanlYVkJ3MjEKU2d3ZGI1amlpemxEM0NSbVdZZ0ZrRzd0NTVZeGF3ZmpaQjh5bW4xYjhUVjkwN3dRcG8veU8zM3RaaEE3L3BFUwpBSDJYeDk5bkpMbFVGVUtSY1A4Q0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKNnZKeVk1UlJ1aklQWGxIK2ZvU3g2QzFRT2RNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ04zcGtCTVM3ekkrbUhvOWdTZQp6SzdXdjl3bXlCTVE5Q3crQXBSNnRBQXg2T1VIN0d1enc5TTV2bXNkYjkrYXBKMHBlZFB4SUg3YXZ1aG9SUXNMCkxqTzRSVm9BMG9aNDBZV3J3UStBR0dvdkZuaWNleXRNcFVSNEZjRXc0ZDRmcGl6V3d0TVNlRlRIUXR6WG84V2MKNFJGWC9xUXNVR1NWa01PaUcvcVVrSFpXQVgyckdhWXZ1Tkw2eHdSRnh5ZHpsRTFSUk56TkNvQzVpTXhjaVRNagpackEvK0pqVEFWU2FuNXZnODFOSmthZEphbmNPWmEwS3JEdkZzd1JJSG5CMGpMLzh3VmZXSTV6czZURU1VZUk1ClF6dU01QXUxUFZ4VXZJUGhlMHl6UXZjWDV5RlhnMkJGU3MzKzJBajlNcENWVTZNY2dSSTl5TTRicitFTUlHL0kKY0pjPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
|
||||
server: https://master.kube.k-space.ee:6443
|
||||
name: kubernetes
|
||||
contexts:
|
||||
- context:
|
||||
cluster: kubernetes
|
||||
user: oidc
|
||||
name: default
|
||||
current-context: default
|
||||
kind: Config
|
||||
preferences: {}
|
||||
users:
|
||||
- name: oidc
|
||||
user:
|
||||
exec:
|
||||
apiVersion: client.authentication.k8s.io/v1beta1
|
||||
args:
|
||||
- oidc-login
|
||||
- get-token
|
||||
- --oidc-issuer-url=https://auth.k-space.ee
|
||||
- --oidc-client-id=kubelogin
|
||||
- --oidc-use-pkce
|
||||
- --oidc-extra-scope=profile,email,groups
|
||||
- --listen-address=127.0.0.1:27890
|
||||
command: kubectl
|
||||
env: null
|
||||
provideClusterInfo: false
|
||||
EOF
|
||||
```
|
||||
|
||||
For access control mapping see [cluster-role-bindings.yml](cluster-role-bindings.yml)
|
||||
|
||||
|
||||
# Technology mapping
|
||||
|
||||
Our self-hosted Kubernetes stack compared to AWS based deployments:
|
||||
|
||||
| Hipster startup | Self-hosted hackerspace | Purpose |
|
||||
|-----------------|-------------------------------------|---------------------------------------------------------------------|
|
||||
| AWS EC2 | Proxmox | Virtualization layer |
|
||||
| AWS EKS | kubeadm | Provision Kubernetes master nodes |
|
||||
| AWS EBS | Longhorn | Block storage for arbitrary applications needing persistent storage |
|
||||
| AWS NLB | MetalLB | L2/L3 level load balancing |
|
||||
| AWS ALB | Traefik | Reverse proxy also known as ingress controller in Kubernetes jargon |
|
||||
| AWS ECR | Harbor | Docker registry |
|
||||
| AWS DocumentDB | MongoDB | NoSQL database |
|
||||
| AWS S3 | Minio | Object storage |
|
||||
| GitHub OAuth2 | Samba (Active Directory compatible) | Source of truth for authentication and authorization |
|
||||
| Dex | Authelia | ACL mapping and OIDC provider which integrates with GitHub/Samba |
|
||||
| GitHub | Gitea | Source code management, issue tracking |
|
||||
| GitHub Actions | Drone | Build Docker images |
|
||||
| Gmail | Wildduck | E-mail |
|
||||
| AWS Route53 | Bind and RFC2136 | DNS records and Let's Encrypt DNS validation |
|
||||
| AWS VPC | Calico | Overlay network |
|
||||
|
||||
|
||||
External dependencies running as classic virtual machines:
|
||||
|
||||
- Samba as Authelia's source of truth
|
||||
- Bind as DNS server
|
||||
|
||||
|
||||
## Adding applications
|
||||
|
||||
Deploy applications via [ArgoCD](https://argocd.k-space.ee)
|
||||
|
||||
We use Treafik with Authelia for Ingress.
|
||||
Applications where possible and where applicable should use `Remote-User`
|
||||
authentication. This prevents application exposure on public Internet.
|
||||
Otherwise use OpenID Connect for authentication,
|
||||
see Argo itself as an example how that is done.
|
||||
|
||||
See `kspace-camtiler/ingress.yml` for commented Ingress example.
|
||||
|
||||
Note that we do not use IngressRoute objects because they don't
|
||||
support `external-dns` out of the box.
|
||||
Do NOT add nginx annotations, we use Traefik.
|
||||
Do NOT manually add DNS records, they are added by `external-dns`.
|
||||
Do NOT manually create Certificate objects,
|
||||
these should be handled by `tls:` section in Ingress.
|
||||
|
||||
|
||||
## Cluster formation
|
||||
|
||||
Create Ubuntu 20.04 VM-s on Proxmox with local storage.
|
||||
|
||||
After machines have booted up and you can reach them via SSH:
|
||||
|
||||
```bash
|
||||
# Enable required kernel modules
|
||||
cat > /etc/modules << EOF
|
||||
overlay
|
||||
br_netfilter
|
||||
EOF
|
||||
cat /etc/modules | xargs -L 1 -t modprobe
|
||||
|
||||
# Finetune sysctl:
|
||||
cat > /etc/sysctl.d/99-k8s.conf << EOF
|
||||
net.ipv4.conf.all.accept_redirects = 0
|
||||
net.bridge.bridge-nf-call-iptables = 1
|
||||
net.ipv4.ip_forward = 1
|
||||
net.bridge.bridge-nf-call-ip6tables = 1
|
||||
EOF
|
||||
sysctl --system
|
||||
|
||||
# Disable Ubuntu caching DNS resolver
|
||||
systemctl disable systemd-resolved.service
|
||||
systemctl stop systemd-resolved
|
||||
rm -fv /etc/resolv.conf
|
||||
cat > /etc/resolv.conf << EOF
|
||||
nameserver 1.1.1.1
|
||||
nameserver 8.8.8.8
|
||||
EOF
|
||||
|
||||
# Disable multipathd as Longhorn handles that itself
|
||||
systemctl mask multipathd
|
||||
systemctl disable multipathd
|
||||
systemctl stop multipathd
|
||||
|
||||
# Disable Snapcraft
|
||||
systemctl mask snapd
|
||||
systemctl disable snapd
|
||||
systemctl stop snapd
|
||||
|
||||
# Permit root login
|
||||
sed -i -e 's/PermitRootLogin no/PermitRootLogin without-password/' /etc/ssh/sshd_config
|
||||
systemctl reload ssh
|
||||
cat << EOF > /root/.ssh/authorized_keys
|
||||
sk-ecdsa-sha2-nistp256@openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBD4/e9SWYWYoNZMkkF+NirhbmHuUgjoCap42kAq0pLIXFwIqgVTCre03VPoChIwBClc8RspLKqr5W3j0fG8QwnQAAAAEc3NoOg== lauri@lauri-x13
|
||||
EOF
|
||||
userdel -f ubuntu
|
||||
apt-get remove -yq cloud-init
|
||||
|
||||
|
||||
```
|
||||
|
||||
Install packages, for Raspbian set `OS=Debian_11`
|
||||
|
||||
```bash
|
||||
OS=xUbuntu_20.04
|
||||
VERSION=1.23
|
||||
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
|
||||
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
|
||||
EOF
|
||||
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
|
||||
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
|
||||
EOF
|
||||
|
||||
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
|
||||
curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -
|
||||
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
|
||||
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
|
||||
|
||||
apt-get update
|
||||
apt-get install -yqq apt-transport-https curl cri-o cri-o-runc kubelet=1.23.5-00 kubectl=1.23.5-00 kubeadm=1.23.5-00
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable crio --now
|
||||
apt-mark hold kubelet kubeadm kubectl
|
||||
sed -i -e 's/unqualified-search-registries = .*/unqualified-search-registries = ["docker.io"]/' /etc/containers/registries.conf
|
||||
```
|
||||
|
||||
On master:
|
||||
|
||||
```
|
||||
kubeadm init --token-ttl=120m --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint "master.kube.k-space.ee:6443" --upload-certs --apiserver-cert-extra-sans master.kube.k-space.ee --node-name master1.kube.k-space.ee
|
||||
```
|
||||
|
||||
For the `kubeadm join` command specify FQDN via `--node-name $(hostname -f)`.
|
||||
|
||||
After forming the cluster add taints:
|
||||
|
||||
```bash
|
||||
for j in $(seq 1 9); do
|
||||
kubectl label nodes worker${j}.kube.k-space.ee node-role.kubernetes.io/worker=''
|
||||
done
|
||||
|
||||
for j in $(seq 1 3); do
|
||||
kubectl taint nodes mon${j}.kube.k-space.ee dedicated=monitoring:NoSchedule
|
||||
kubectl label nodes mon${j}.kube.k-space.ee dedicated=monitoring
|
||||
done
|
||||
|
||||
for j in $(seq 1 4); do
|
||||
kubectl taint nodes storage${j}.kube.k-space.ee dedicated=storage:NoSchedule
|
||||
kubectl label nodes storage${j}.kube.k-space.ee dedicated=storage
|
||||
done
|
||||
```
|
||||
|
||||
On Raspberry Pi you need to take additonal steps:
|
||||
|
||||
* Manually enable cgroups by appending
|
||||
`cgroup_memory=1 cgroup_enable=memory` to `/boot/cmdline.txt`,
|
||||
* Disable swap with `swapoff -a; apt-get purge -y dphys-swapfile`
|
||||
* For mounting Longhorn volumes on Rasbian install `open-iscsi`
|
||||
|
||||
For `arm64` nodes add suitable taint to prevent scheduling non-multiarch images on them:
|
||||
|
||||
```bash
|
||||
kubectl taint nodes worker9.kube.k-space.ee arch=arm64:NoSchedule
|
||||
```
|
||||
Reference in New Issue
Block a user