Hi everyone, I’ve been building my own log search server because I wasn’t satisfied with any of the alternatives out there and wanted a project to learn rust with. It still needs a ton of work but wanted to share what I’ve built so far.

The repo is up here: https://codeberg.org/Kryesh/crystalline

and i’ve started putting together some documentation here: https://kryesh.codeberg.page/crystalline/

There’s a lot of features I plan to add to it but I’m curious to hear what people think and if there’s anything you’d like to see out of a project like this.

Some examples from my lab environment:

events view searching for SSH logins from systemd journals and syslog events:

counting raw event size for all indices:

performance is looking pretty decent so far, and it can be configured to not be too much of a resource hog depending on use case, some numbers from my test install:

  • raw events ingested: ~52 million
  • raw event size: ~40GB
  • on disk size: ~5.8GB

Ram usage:

  • not running searches ingesting 600MB-1GB per day it uses about 500MB of ram
  • running the ssh search examples above brings it to about 600MB of ram while the search is running
  • running last example search getting the size of all events (requires decompressing the entire event store) peaked at about 3.5GB of ram usage
  • Kryesh@lemmy.worldOPEnglish
    0·
    3 days ago

    Thanks! definitely aiming for a stupid easy installation/management for the app itself; but in my experience getting a wide range of supported log sources is no small feat. I’ve been using fluentbit to handle collection from different sources and using the following has been working well for me:

    • docker ‘journald’ log driver
    • fluentbit ‘systemd’ input
    • fluentbit ‘http’ output like the one in the readme

    with that setup you can search for container logs by name which works great with compose:

    or process logs from an nginx container like this to see traffic from external hosts:

    I’ll add a more complete example to the docs, but if you look in the repo there’s a complete example for receiving and ingesting syslog that you can run with just “docker compose up”

    • PlusMinus@lemmy.worldEnglish
      0·
      2 days ago

      Maybe you should add OTLP support? I don’t know how you are ingesting from Fluentbit at the moment, but I think with OTLP basically any log source can be integrated either through the fluentbit OTLP plugin or an OTEL collector.