kube/logging

Logging infrastructure

Background

Fluent Bit picks up the logs from Kubernetes workers and sends them to Graylog using GELF over TCP 12201.

Graylog ingests the logs and stores them in Elasticsearch.

Deployment

To deploy:

kubectl create namespace logging
kubectl apply -n logging -f mongodb-support.yml -f application.yml -f filebeat.yml -f networkpolicy-base.yml
kubectl rollout restart -n logging daemonset.apps/filebeat

To set secrets:

GRAYLOG_ROOT_PASSWORD=$(cat /dev/urandom | base64 | head -c 30)
echo "Graylog admin password: $GRAYLOG_ROOT_PASSWORD"
kubectl create secret generic -n logging graylog-secrets \
    --from-literal=GRAYLOG_ROOT_PASSWORD_SHA2=$(echo -en $GRAYLOG_ROOT_PASSWORD | sha256sum | cut -d" " -f1) \
    --from-literal=GRAYLOG_PASSWORD_SECRET=$(cat /dev/urandom | base64 | head -c 30)
kubectl create secret generic -n logging mongodb-application-readwrite-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
kubectl create secret generic -n logging mongodb-application-readonly-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"

Graylog setup

Note that Graylog is running without disk journal to prevent SSD thrashing and to save some disk space. This will be problematic when there are loads for logs coming in and ElasticSearch is unable to process the entries in timely manner. ElasticSearch default index is tuned to match the persistent volume allocated on Longhorn to prevent running out disk space on that PV.

After Graylog deployment following steps were manually performed via web interface:

• Add Syslog TCP input for external Linux hosts
• Add Syslog UDP input for Mikrotik networking gear
• Add Beats input for Kubernetes workers, enable Do not add Beats type as prefix
• Trusted header authentication was enabled and set to Remote-User https://graylog.k-space.ee/system/authentication/authenticator/edit Note that user accounts are not provisioned automatically. Users need to be manually created in Graylog with matching Username. Automatic user account provisioning is supported in Graylog Enterprise version