Cloud native log shipping daemon written in Go
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
rasmus e73c30689c update deps 2 weeks ago
k8s/dev k8s: remove NoExecute 2 weeks ago
pkg update deps 2 weeks ago
vendor update deps 2 weeks ago
.drone.yml add drone? 3 weeks ago
.gitignore it is shipping and parsing 3 weeks ago
Dockerfile don't copy . in Dockerfile 3 weeks ago
README.md accurarcy edit in readme drawing 2 weeks ago
go.mod update deps 2 weeks ago
go.sum update deps 2 weeks ago
main.go change metrics port to 8000 3 weeks ago
skaffold.yaml add skaffold and fix bugs 3 weeks ago

README.md

logmower-shipper

Watches log directory for logs, and ships them to mongo.

  • Running: go run .
  • Building: go build .

Files

  1. pkg/globals defines globals. This is used for Prometheus namespace, and for forwarding static global CLI configuration with less clutter.
  2. main.go serves /metrics and runs watcher.go.
  3. pkg/watcher main routine; initializes (including mongo.go) and watches for new log files.
    • pkg/mongo provides statically safe and central tools to interact with the database.
    • pkg/util provides additional utility functions.
  4. pkg/file handles file lifecycle; watches files and tails them, streaming lines to pkg/lines.
  5. pkg/lines processes lines and streams them to pkg/sender.
  6. pkg/sender batches lines and ships them to mongo.
┌──────────────────────────────────────────────────────────────┬─────────┐
│ List directory      New files created                        ) watcher (
│ │                    │                                       └─────────┤
│ └► Simulate initial ─┴─► File watcher ──► Parse filename (k8s)         │
│     create events                              ┌─◄───┘                 │
│                                 Ignore files   │─◄─┐Namespace filters  │
│ ┌─ Launch file collector ◄──── based on filter ┘─◄─┘Pod prefix filters │
└─▼──────────────────────────────────────────────────────────────────────┘
  │                                         ┌──────┐
  │ Retry any downstream error with backoff ) file ( ┌──────────┬────────┐
┌─▼─────────────────────────────────────────┴──────┤ │  Submit  ) sender (
│┌┴─► mongo: get offset     create channels  ┌─────► │ to mongo └────────┤
││      │                 + chan processors  │     │ ├────▲────┐         │
│└──┐───┘      ┌─────────────┘           └───┤     │ ) batcher (         │
└───▼──────────▼──────────────────────────┐  │     │ ├────▲────┼─────────┘
┌───┴────┐ ┌───────┬────────────────────┐ │  └─────► │  queue  │
) tailer ( ) lines (      combine       │ └────────┘ └────▲────┘
└───▼────┘ ├───────┘   partial lines    │                 │
    │      │            ▲       │       │   {possible processing plugins}
    └──────► parse k8s  │       ▼       │                 ▲
           │ line metadata ─────┴───────►─────────────────┘
           │                            │
           └────────────────────────────┘

Skaffold

export NS=gitdbd-5cym5
kubectl apply -f k8s/dev/mongodb.yaml -n "$NS"
kubectl apply -f k8s/dev/prom.yaml -n "$NS"
skaffold dev --namespace "$NS"

hwatch kubectl get pods --namespace "$NS"