k-space/logmower-shipper
10
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Activity

add project structure ascii

Browse Source
This commit is contained in:
rasmus 2022-11-11 17:09:18 +02:00
parent 0b3d382742
commit a50c1446c3
1 changed files with 26 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
README.md
View File

@ -15,6 +15,32 @@ Watches log directory for logs, and ships them to mongo.
4. `pkg/lines` processes lines and streams them to `pkg/sender`.
5. `pkg/sender` batches lines and ships them to mongo.
```txt
┌──────────────────────────────────────────────────────────────┬─────────┐
│ List directory New files created ) watcher (
│ │ │ └─────────┤
│ └► Simulate initial ─┴─► File watcher ──► Parse filename (k8s) │
│ create events ┌─◄───┘ │
│ Ignore files │─◄─┐Namespace filters │
│ ┌─ Launch file collector ◄──── based on filter ┘─◄─┘Pod prefix filters │
└─▼──────────────────────────────────────────────────────────────────────┘
│ ┌──────┐
│ Retry any downstream error with backoff ) file ( ┌──────────┬────────┐
┌─▼─────────────────────────────────────────┴──────┤ │ Submit ) sender (
│┌┴─► mongo: get offset create channels ┌─────► │ to mongo └────────┤
││ │ + chan processors │ │ ├────▲────┬─────────┘
│└──┐───┘ ┌─────────────┘ └───┼─────► ) batcher (
└───▼──────────▼──────────────────────────┐ │ │ ├────▲────┤
┌───┴────┐ ┌───────┬────────────────────┐ │ └─────► │ queue │
) tailer ( ) lines ( combine │ └────────┘ └────▲────┘
└───▼────┘ ├───────┘ partial lines │ │
│ │ ▲ │ │ {possible processing plugins}
└──────► parse k8s │ ▼ │ ▲
│ line metadata ─────┴───────►─────────────────┘
│ │
└────────────────────────────┘
```
## Skaffold
```bash
export NS=gitdbd-5cym5