Data Engineering

259 readers

1 users here now

News and discussion on Data Engineering topics

founded 1 year ago

MODERATORS

[email protected]

Visualising Kafka Internals (lemmy.ml)

submitted 1 year ago by [email protected] to c/[email protected]

2 comments fedilink hide all child comments

I plan to run a few tests to determine if Kafka is suitable for a certain usecase I have in mind.

My idea is to run a local cluster of Kafka servers (either VMs or containers), produce/consume a series of messages, observe a bunch of metrics (Prometheus & Grafana) and custom business logic outcomes.

What are some good tools to record and visualise the internals of Kafka cluster?

I'm looking for things like consumer lag, topic replication, possibly tracing messages, ...

Originally posted on https://mastodon.social/@bahmanm/110662538718523380

top 2 comments

sorted by: hot top controversial new old

[–] [email protected] 2 points 1 year ago (1 children)

JMX Exporter and Kafka Exporter

[–] [email protected] 1 points 1 year ago

Thanks 🙏 That's what I've gathered so far too. I will keep this post, well, posted in case I stumble upon something else.