kube/logging/README.md

# Logging infrastructure

## Background

Fluent Bit picks up the logs from Kubernetes workers and sends them to Graylog
using GELF over TCP 12201.

Graylog ingests the logs and stores them in Elasticsearch.


## Deployment

To deploy:

```
kubectl create namespace logging
kubectl apply -n logging -f zinc.yml -f application.yml -f filebeat.yml -f networkpolicy-base.yml
kubectl rollout restart -n logging daemonset.apps/filebeat
```

To set secrets:

```
GRAYLOG_ROOT_PASSWORD=$(cat /dev/urandom | base64 | head -c 30)
echo "Graylog admin password: $GRAYLOG_ROOT_PASSWORD"
kubectl create secret generic -n logging graylog-secrets \
    --from-literal=GRAYLOG_ROOT_PASSWORD_SHA2=$(echo -en $GRAYLOG_ROOT_PASSWORD | sha256sum | cut -d" " -f1) \
    --from-literal=GRAYLOG_PASSWORD_SECRET=$(cat /dev/urandom | base64 | head -c 30)
kubectl create secret generic -n logging mongodb-application-readwrite-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
kubectl create secret generic -n logging mongodb-application-readonly-password --from-literal="password=$(cat /dev/urandom | base64 | head -c 30)"
```


## Graylog setup

Note that Graylog is running without disk journal to
prevent SSD thrashing and to save some disk space.
This will be problematic when there are loads for logs coming in and
ElasticSearch is unable to process the entries in timely manner.
ElasticSearch default index is tuned to match the persistent volume allocated
on Longhorn to prevent running out disk space on that PV.

After Graylog deployment following steps were manually performed via web interface:

* Add Syslog TCP input for external Linux hosts
* Add Syslog UDP input for Mikrotik networking gear
* Add Beats input for Kubernetes workers,
  enable `Do not add Beats type as prefix`
* Trusted header authentication was enabled and set to `Remote-User`
  https://graylog.k-space.ee/system/authentication/authenticator/edit
  Note that user accounts are not provisioned automatically.
  Users need to be manually created in Graylog with matching `Username`.
  Automatic user account provisioning is supported in Graylog Enterprise version