53 lines
2.0 KiB
Markdown
53 lines
2.0 KiB
Markdown
|
# Logging infrastructure
|
||
|
|
|
||
|
|
## Background
|
||
|
|
|
||
|
|
Fluent Bit picks up the logs from Kubernetes workers and sends them to Graylog
|
||
|
|
using GELF over TCP 12201.
|
||
|
|
|
||
|
|
Graylog ingests the logs and stores them in Elasticsearch.
|
||
|
|
|
||
|
|
|
||
|
|
## Deployment
|
||
|
|
|
||
|
|
To deploy:
|
||
|
|
|
||
|
|
```
|
||
|
|
kubectl create namespace logging
|
||
|
|
kubectl apply -n logging -f mongodb-support.yml -f application.yml -f networkpolicy-base.yml
|
||
|
|
kubectl rollout restart -n logging daemonset/fluent-bit
|
||
|
|
```
|
||
|
|
|
||
|
|
To set secrets:
|
||
|
|
|
||
|
|
```
|
||
|
|
GRAYLOG_ROOT_PASSWORD=$(cat /dev/urandom | base64 | head -c 30)
|
||
|
|
echo "Graylog admin password: $GRAYLOG_ROOT_PASSWORD"
|
||
|
|
kubectl create secret generic -n logging graylog-secrets \
|
||
|
|
--from-literal=GRAYLOG_ROOT_PASSWORD_SHA2=$(echo -en $GRAYLOG_ROOT_PASSWORD | sha256sum | cut -d" " -f1) \
|
||
|
|
--from-literal=GRAYLOG_PASSWORD_SECRET=$(cat /dev/urandom | base64 | head -c 30)
|
||
|
|
kubectl create secret generic -n logging mongodb-application-readwrite-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
|
||
|
|
kubectl create secret generic -n logging mongodb-application-readonly-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
## Graylog setup
|
||
|
|
|
||
|
|
Note that Graylog is running without disk journal to
|
||
|
|
prevent SSD thrashing and to save some disk space.
|
||
|
|
This will be problematic when there are loads for logs coming in and
|
||
|
|
ElasticSearch is unable to process the entries in timely manner.
|
||
|
|
ElasticSearch default index is tuned to match the persistent volume allocated
|
||
|
|
on Longhorn to prevent running out disk space on that PV.
|
||
|
|
|
||
|
|
After Graylog deployment following steps were manually performed via web interface:
|
||
|
|
|
||
|
|
* Add Syslog TCP input for external Linux hosts
|
||
|
|
* Add Syslog UDP input for Mikrotik networking gear
|
||
|
|
* Add GELF TCP input for Kubernetes workers
|
||
|
|
* Trusted header authentication was enabled and set to `Remote-User`
|
||
|
|
https://graylog.k-space.ee/system/authentication/authenticator/edit
|
||
|
|
Note that user accounts are not provisioned automatically.
|
||
|
|
Users need to be manually created in Graylog with matching `Username`.
|
||
|
|
Automatic user account provisioning is supported in Graylog Enterprise version
|