forked from k-space/kube
.. | ||
.gitignore | ||
application.yml | ||
filebeat.yml | ||
networkpolicy-base.yml | ||
README.md | ||
zinc.yml |
Logging infrastructure
Background
Fluent Bit picks up the logs from Kubernetes workers and sends them to Graylog using GELF over TCP 12201.
Graylog ingests the logs and stores them in Elasticsearch.
Deployment
To deploy:
kubectl create namespace logging
kubectl apply -n logging -f zinc.yml -f application.yml -f filebeat.yml -f networkpolicy-base.yml
kubectl rollout restart -n logging daemonset.apps/filebeat
To set secrets:
GRAYLOG_ROOT_PASSWORD=$(cat /dev/urandom | base64 | head -c 30)
echo "Graylog admin password: $GRAYLOG_ROOT_PASSWORD"
kubectl create secret generic -n logging graylog-secrets \
--from-literal=GRAYLOG_ROOT_PASSWORD_SHA2=$(echo -en $GRAYLOG_ROOT_PASSWORD | sha256sum | cut -d" " -f1) \
--from-literal=GRAYLOG_PASSWORD_SECRET=$(cat /dev/urandom | base64 | head -c 30)
kubectl create secret generic -n logging mongodb-application-readwrite-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
kubectl create secret generic -n logging mongodb-application-readonly-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
Graylog setup
Note that Graylog is running without disk journal to prevent SSD thrashing and to save some disk space. This will be problematic when there are loads for logs coming in and ElasticSearch is unable to process the entries in timely manner. ElasticSearch default index is tuned to match the persistent volume allocated on Longhorn to prevent running out disk space on that PV.
After Graylog deployment following steps were manually performed via web interface:
- Add Syslog TCP input for external Linux hosts
- Add Syslog UDP input for Mikrotik networking gear
- Add Beats input for Kubernetes workers,
enable
Do not add Beats type as prefix
- Trusted header authentication was enabled and set to
Remote-User
https://graylog.k-space.ee/system/authentication/authenticator/edit Note that user accounts are not provisioned automatically. Users need to be manually created in Graylog with matchingUsername
. Automatic user account provisioning is supported in Graylog Enterprise version