News and discussion on Data Engineering topics
What needs to be added for 2023
I do like to have a look at these type of overviews, but I highly doubt they're actually useful to anyone. Stuff on there is so random it'll probably confuses the people this is intended for more than it helps..
what could be a better approach?
Hmm good question.. I do like overviews with tooling/technology; what is new, what is left behind, what is considered standard at the moment, etc. It's just that, since DE is such a broad field, I think it would be better to let the field you're active in determine what to focus on.
If the goal is to provide junior DE's with some guidelines, I prefer to educate them on what would functionally work best to solve a problem, so they can make the right decision on what tool to pick (where in practice, this is likely already decided; you just work with what's there).
Isn’t Apache Arrow a table format rather than a batch processing tool?
You could add SQLMesh as an alternative to dbt.
thanks for suggesting sqlmesh. regarding arrow, have you read this - https://arrow.apache.org/docs/format/Columnar.html
btw, you've spelt Engineering wrong in the sidebar.
thanks for pointing out. fixed.