It's not so much that ActivityPub can't scale up, so much as that for one, as I've understood it that's not really been desirable anyway (undermines the point of decentralization/distribution), and for two, it starts getting bogged down as you already recognize. It also runs into similar, if not worse, cost problems to operate as ATProto's full network approaches are now.
ActivityPub is more suited to scaling across multiple instances/sites than up, and I'd argue that's its strength. It unintentionally has an implosion threshold to counter centralization in terms of cost and performance.
On the other hand, ATProto's advantage is that it enables scaling up while also enabling better data portability. I'm aware of work on this with ActivityPub as well, but it's still very early stages. My thinking is that there may be some ways to work with both to push towards their similar shared aims in terms of an open social web, with more flexibility in moving between spaces and adjusting experiences to better find what one wants from these different spaces.
I understand the hesitation, which is why I've been trying to monitor its developments closely. Hence why I linked the example of someone testing out a small network ATProto relay, and why I also dug up this post about self-hosting different parts of the ATProto infrastructure the other day.
From what I've observed, there's no pushback against people doing so, and the only things stopping people are the usual: time, costs, knowledge, motivation, etc. For the first step to really happen at all there have to be people with the resources and motivation to do so, which is always the tricky part. In a small way part of my OP is intending to encourage anyone with both to give it a shot, as I lack some of the necessary resources to try it myself.