this post was submitted on 21 Nov 2024
39 points (97.6% liked)

Linux

48738 readers
1410 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
 

Or any other log files/output? I'm open to any solution, but what I would like...

...is something where I can just click on a word or select some text and say "filter that out"

Something that colors different log levels differently, preferably automatically.

Something that can parse the "columns" and give me a nice quick list of values, like different unit names to filter out/solely include.

Something that lets me choose a time and go there. Something that lets me select only a specific timeframe of logs.

I know this can probably be done by going in/out of journalctl, recalling the last command and adding specific filter options... but it just feels slow. It's so many keypresses when I could just right click on the word and -> "Filter out/Search for" or something.

top 18 comments
sorted by: hot top controversial new old
[–] [email protected] 18 points 1 month ago* (last edited 1 month ago) (1 children)

tbh my go to command is just.. journalctl -fe -u service

ex :
journalctl -fe -u jellyfin
journalctl -fe -u nordvpnd

so I'd also like to know the answer to this question. my other go to is dumping journalctl to text files and parsing with grep and awk and creating my own reports with that parsed information.

grep -E is my favorite, I love regex capturing groups.

[–] [email protected] 8 points 1 month ago (1 children)

Apparently, less also has a feature built-in to filter out lines based on keywords:

https://raymii.org/s/snippets/Exclude_lines_in_less_or_journalctl.html#%3A%7E%3Atext=Once+your%2Cterm (skip the first paragraph, past those three links)

[–] [email protected] 3 points 1 month ago

That's great to know!

[–] [email protected] 6 points 1 month ago (1 children)

I sometimes pipe journalctl into lnav, but it never works quite as well as i really want...

lnav is pretty cool and does mostly what you are describing.

uuhhh maybe here? https://lnav.org/

[–] [email protected] 1 points 1 month ago

lnav is my go to regarding logs exploration, can't work without it.

[–] [email protected] 5 points 1 month ago

It might be a bit overkill but I use Grafana to do this (with Loki). It's a pretty involved setup as well, but you can filter and search by content, or date/time. It's doable on a desktop but mainly servers use it

[–] [email protected] 5 points 1 month ago (1 children)

If you are on gnome,.gnome logs do most of the things you want (if I recall correctly, some years since I run gnome)

https://apps.gnome.org/Logs/

[–] villainy 5 points 1 month ago

Maybe something like Logdy? It does have specific documentation around journalctl

[–] [email protected] 5 points 1 month ago

I don't know of any graphical tools that let you do this, but generally, if you want to search for specific terms/times/commands or anything of that sort, piping journalctl into grep (and optionally grep into less) is pretty effective at finding stuff.

[–] MorphiusFaydal 2 points 1 month ago (1 children)

Centralized logging like Graylog or Grafana Loki can help with a lot of this.

[–] [email protected] 1 points 1 month ago

I tried to setup Loki but the documentation was atrocious. Everything was outdated, referring to tools that were marked deprecated but documentation for the replacements just didn't exist.

[–] [email protected] 2 points 1 month ago

journalctl -fu servicename

If I am concerned about a specific service, and can trigger the problematic behavior.

In my head it stands for "fuck you" ☺️

[–] [email protected] 1 points 1 month ago* (last edited 1 month ago) (1 children)

Sounds like you want a siem like Wazuh. Its agent can collect journald logs from any number of systems. It also has a gui you can interact with to parse logs.

[–] [email protected] 4 points 1 month ago* (last edited 1 month ago) (1 children)

Well, just a monitoring stack, like for example Grafana, would probably be more suitable for this specific task (if we're doing central hosting/collection).

Kind of my main recommendation is to use something with OpenTelemetry. It's pretty much the standard protocol for transferring logs, traces and metrics, so if you set everything up with that, then you can swap out the visualization software with less pain.
Here's a guide for Grafana + OpenTelemetry Collector: https://grafana.com/docs/loki/latest/send-data/otel/

[–] [email protected] 2 points 1 month ago

I'm seconding this recommendation

[–] [email protected] 1 points 1 month ago (1 children)
systemctl status <service>
[–] [email protected] 2 points 1 month ago

That is good but only shows the last 10-15 lines of the log, unless there is an arg to expand that, or a command to follow the log. I am aware of neither.

I usually use your suggested command to check if a service is up, then if it isn't, use journalctl to find out why.