Ahh - the joys of monorepo.
this post was submitted on 29 Oct 2024
98 points (97.1% liked)
Technology
59291 readers
4752 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
The algorithm checked only the last 16 characters of the path, so package differences were not considered.
that is such a C thing to do.
It's not as stupid as this blog post makes it sound. This was a hashing function that was intentionally taking the end of the path as the most significant part. This just impacts the order of objects in a pack file, and the size of the compression window needed to compress it.
It's not actually mistaking one file for another, and their proposed solution is not better in all situations.
yeah i'm not saying it is stupid or something, but this kind of optimisation is found in C code very often, where people take the first|last N of something because it is most likely good enough :D
maybe I'm missing something but wouldn't this show up in a diff before pushing?
The culprit: name-hash collisions. With files like changelog.md and changelog.json, Git found all kinds of differences with each commit even though there often were none, which gradually added 173GB of unnecessary bloat. However, the changelogs were in completely different packages. The algorithm checked only the last 16 characters of the path, so package differences were not considered.
thanks for the reply, but i think i got that. from the linked article:
For example, if you changed repo/packages/foo/CHANGELOG.json, when git was getting ready to do the push, it was generating a diff against repo/packages/bar/CHANGELOG.json! This meant we were in many occasions just pushing the entire file again and again, which could be 10s of MBs per file in some cases, and you can imagine in a repo our size, how that would be a problem.
but wouldn't these erroneous diffs not show up in git diff? it seems that they were pushing (maybe automatically?)without inspecting the diffs first