53 lines
3.7 KiB
Markdown
53 lines
3.7 KiB
Markdown
# logmower-shipper
|
|
|
|
Watches log directory for logs, and ships them to mongo.
|
|
|
|
- Running: `go run .`
|
|
- Building: `go build .`
|
|
|
|
## Files
|
|
0. `pkg/globals` defines globals. This is used for Prometheus namespace, and for forwarding static global CLI configuration with less clutter.
|
|
1. `main.go` serves `/metrics` and runs `watcher.go`.
|
|
2. `pkg/watcher` main routine; initializes (including `mongo.go`) and watches for new log files.
|
|
- `pkg/mongo` provides statically safe and central tools to interact with the database.
|
|
- `pkg/util` provides additional utility functions.
|
|
3. `pkg/file` handles file lifecycle; watches files and tails them, streaming lines to `pkg/lines`.
|
|
4. `pkg/lines` processes lines and streams them to `pkg/sender`.
|
|
5. `pkg/sender` batches lines and ships them to mongo.
|
|
|
|
```txt
|
|
┌──────────────────────────────────────────────────────────────┬─────────┐
|
|
│ List directory New files created ) watcher (
|
|
│ │ │ └─────────┤
|
|
│ └► Simulate initial ─┴─► File watcher ──► Parse filename (k8s) │
|
|
│ create events ┌─◄───┘ │
|
|
│ Ignore files │─◄─┐Namespace filters │
|
|
│ ┌─ Launch file collector ◄──── based on filter ┘─◄─┘Pod prefix filters │
|
|
└─▼──────────────────────────────────────────────────────────────────────┘
|
|
│ ┌──────┐
|
|
│ Retry any downstream error with backoff ) file ( ┌──────────┬────────┐
|
|
┌─▼─────────────────────────────────────────┴──────┤ │ Submit ) sender (
|
|
│┌┴─► mongo: get offset create channels ┌─────► │ to mongo └────────┤
|
|
││ │ + chan processors │ │ ├────▲────┐ │
|
|
│└──┐───┘ ┌─────────────┘ └───┤ │ ) batcher ( │
|
|
└───▼──────────▼──────────────────────────┐ │ │ ├────▲────┼─────────┘
|
|
┌───┴────┐ ┌───────┬────────────────────┐ │ └─────► │ queue │
|
|
) tailer ( ) lines ( combine │ └────────┘ └────▲────┘
|
|
└───▼────┘ ├───────┘ partial lines │ │
|
|
│ │ ▲ │ │ {possible processing plugins}
|
|
└──────► parse k8s │ ▼ │ ▲
|
|
│ line metadata ─────┴───────►─────────────────┘
|
|
│ │
|
|
└────────────────────────────┘
|
|
```
|
|
|
|
## Skaffold
|
|
```bash
|
|
export NS=gitdbd-5cym5
|
|
kubectl apply -f k8s/dev/mongodb.yaml -n "$NS"
|
|
kubectl apply -f k8s/dev/prom.yaml -n "$NS"
|
|
skaffold dev --namespace "$NS"
|
|
|
|
hwatch kubectl get pods --namespace "$NS"
|
|
```
|