A) Your conspiracy theory of this being some great plot to pump up AI to control society is not at all what's suggested by the article.
B) The article says that the Times blocked one specific bot (among others, but this is the one they named). That bot was not run by the Internet Archive but by Amazon. The Amazon bot did indeed feed the information on to the Archive - but it also fed all that information into their Alexa AI.
The bot belonged to Alexa Internet, a web traffic analysis company co-founded by Brewster Kahle, who went on to create the Internet Archive right after Alexa. Alexa Internet went on to be acquired by Amazon in 1999 — its trademark name was later used for Amazon’s signature voice-activated assistant — and was eventually sunset in 2022.
C) The Internet Archive decided to ignore the robots.txt setting anyway:
By 2017, however, the Internet Archive announced its intention to stop abiding by the dictates of a site’s robots.txt. While the Internet Archive had already been disregarding the robots.txt for military and government sites, the new update expanded the move to disregard robots.txt for all sites. Instead, website owners could make manual exclusion requests by email.