Cloud native log shipping daemon written in Go
|
||
---|---|---|
k8s/dev | ||
pkg | ||
vendor | ||
.drone.yml | ||
.gitignore | ||
Dockerfile | ||
go.mod | ||
go.sum | ||
main.go | ||
README.md | ||
skaffold.yaml |
logmower-shipper
Watches log directory for logs, and ships them to mongo.
- Running:
go run .
- Building:
go build .
Files
pkg/globals
defines globals. This is used for Prometheus namespace, and for forwarding static global CLI configuration with less clutter.main.go
serves/metrics
and runswatcher.go
.pkg/watcher
main routine; initializes (includingmongo.go
) and watches for new log files.pkg/mongo
provides statically safe and central tools to interact with the database.pkg/util
provides additional utility functions.
pkg/file
handles file lifecycle; watches files and tails them, streaming lines topkg/lines
.pkg/lines
processes lines and streams them topkg/sender
.pkg/sender
batches lines and ships them to mongo.
┌──────────────────────────────────────────────────────────────┬─────────┐
│ List directory New files created ) watcher (
│ │ │ └─────────┤
│ └► Simulate initial ─┴─► File watcher ──► Parse filename (k8s) │
│ create events ┌─◄───┘ │
│ Ignore files │─◄─┐Namespace filters │
│ ┌─ Launch file collector ◄──── based on filter ┘─◄─┘Pod prefix filters │
└─▼──────────────────────────────────────────────────────────────────────┘
│ ┌──────┐
│ Retry any downstream error with backoff ) file ( ┌──────────┬────────┐
┌─▼─────────────────────────────────────────┴──────┤ │ Submit ) sender (
│┌┴─► mongo: get offset create channels ┌─────► │ to mongo └────────┤
││ │ + chan processors │ │ ├────▲────┐ │
│└──┐───┘ ┌─────────────┘ └───┤ │ ) batcher ( │
└───▼──────────▼──────────────────────────┐ │ │ ├────▲────┼─────────┘
┌───┴────┐ ┌───────┬────────────────────┐ │ └─────► │ queue │
) tailer ( ) lines ( combine │ └────────┘ └────▲────┘
└───▼────┘ ├───────┘ partial lines │ │
│ │ ▲ │ │ {possible processing plugins}
└──────► parse k8s │ ▼ │ ▲
│ line metadata ─────┴───────►─────────────────┘
│ │
└────────────────────────────┘
Skaffold
export NS=gitdbd-5cym5
kubectl apply -f k8s/dev/mongodb.yaml -n "$NS"
kubectl apply -f k8s/dev/prom.yaml -n "$NS"
skaffold dev --namespace "$NS"
hwatch kubectl get pods --namespace "$NS"