From a50c1446c3042a7d8b09cc287b7185729613884e Mon Sep 17 00:00:00 2001 From: rasmus Date: Fri, 11 Nov 2022 17:09:18 +0200 Subject: [PATCH] add project structure ascii --- README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/README.md b/README.md index 613a9d4..ee8ed87 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,32 @@ Watches log directory for logs, and ships them to mongo. 4. `pkg/lines` processes lines and streams them to `pkg/sender`. 5. `pkg/sender` batches lines and ships them to mongo. +```txt +┌──────────────────────────────────────────────────────────────┬─────────┐ +│ List directory New files created ) watcher ( +│ │ │ └─────────┤ +│ └► Simulate initial ─┴─► File watcher ──► Parse filename (k8s) │ +│ create events ┌─◄───┘ │ +│ Ignore files │─◄─┐Namespace filters │ +│ ┌─ Launch file collector ◄──── based on filter ┘─◄─┘Pod prefix filters │ +└─▼──────────────────────────────────────────────────────────────────────┘ + │ ┌──────┐ + │ Retry any downstream error with backoff ) file ( ┌──────────┬────────┐ +┌─▼─────────────────────────────────────────┴──────┤ │ Submit ) sender ( +│┌┴─► mongo: get offset create channels ┌─────► │ to mongo └────────┤ +││ │ + chan processors │ │ ├────▲────┬─────────┘ +│└──┐───┘ ┌─────────────┘ └───┼─────► ) batcher ( +└───▼──────────▼──────────────────────────┐ │ │ ├────▲────┤ +┌───┴────┐ ┌───────┬────────────────────┐ │ └─────► │ queue │ +) tailer ( ) lines ( combine │ └────────┘ └────▲────┘ +└───▼────┘ ├───────┘ partial lines │ │ + │ │ ▲ │ │ {possible processing plugins} + └──────► parse k8s │ ▼ │ ▲ + │ line metadata ─────┴───────►─────────────────┘ + │ │ + └────────────────────────────┘ +``` + ## Skaffold ```bash export NS=gitdbd-5cym5