If it doesn’t belong to them, they should relinquish it. Simple as that
Ignoring the rest of your comme to focus on this part because I love it when people declare complex issues "simple" and give 1 dimensional solutions to it.
How do you determine if it doesn't belong to them?
For example I think most people would say if they bought the artifacts legitimately, then they belong to them right?
What about cases when they legitimately buy artifacts from people who themselves acquired them I legitimately? Or how do you even determine legitimacey? Is someone finding a historical object mean it belongs to them and they can do what they like with it? What if it's on public land? What if it's private land and they are working on it and find it? What about when ownership of the land is disputed?
What about cases like the rosetta stone, that was found ina rubble heap by French forces and eventually given to Britain as part of war confessions. Should that frenchman have left it in a pile, doomed to be destroyed because it doesn't belong to him?
It's not British museum but the koh-i-noor diamond that's part of the crown jewels, often claimed that the UK should give it back. Who do you give it back to?
Do you give it to India as the successor state of the kingdom of Punjab who handed over to the British after they lost a war? Do you give it back to Kashmir as the successor of Jammu who the Punabs stole it from? Do you give it to Pakistan as it was once the property of the Sikh empire? Or do you give it to Iran as it was first record in the possession of Nader Shah?
While I would say the Elgin marbles are a case that err on the side of they should be returned, again reality is not as simple and black and white as people like you want it to be.
When Elgin took the marbles the entire area was, by all accounts, in shambles and people were burning pieces of these statues to aquire lime to build with. And that was one of the reasons Elgin decided to remove the marbles, so they are preserved today because they were removed.
Then there's also the argument that they do in fact legally belong to the UK, Elgin got permission from both the ottoman empire and local authorities in Athens to remove the sculptures. Then the actions were also ratified when Elgin was twice given permissions by the ottomans to ship the statues out of Athens. No historian worth there salt will tell you they are stolen. That is a position held by Greeks, people pressured by them and people that have seen a tumble post or reddit article about the marbles and gone full dunning-kruger, thinking they know the intricacies of the whole situation.