Warhammer 40k

40k is a wargame. Players, generally two, bring their armies to the table and battle against each other. As opposed to D&D which is an RPG with a collaborative element with combat against DM controlled enemies that are designed to be beaten. Here's a battle video, chosen basically at random to show what a game looks like.

40k relies on having a board and terrain, unlike D&D which can all be done in by talking ("the theater of the mind" if you are feeling pretentious). There is no DM in 40k, just players both following the rules with no hidden rolls behind screens, making it less narrative focused and more straight combat focused. The sidebar here has links to the current rules, and the core rules and quick start guide will be most relevant.

As for cost, it is more of a moneysink than D&D by the nature of making armies rather than single characters, and on top of the Games Workshop has infamously high pricing. There's a lot you can do to mitigate Games Workshop's pricing. Investing in paints from other companies is a big one right off the bat. GW makes some good paints, but more often than not you can find the same or better from another company for a lower price. For models, depending on whatever faction catches your eye, you can buy used models on Ebay, scratch build/modify your own, or buy 3rd party approximations. Those last two options are no good in official Games Workshop tournaments, but if you just want to play at independent stores or with friends, I highly recommend it. Is there a faction that you have in mind to try out?

If you want an RPG in the 40k universe, you can try Inquisitor, which has free rules here. It's designed for 54mm miniatures, but it is not difficult to convert to play with the more common 28mm.

If you are wanting to try out a 40k style skirmish but feeling overwhelmed by the rules, you can also check out third party rules, like Grimdark Future Firefight which is easily compatible with 40k miniatures, but uses it's own unique rules. It gives the flavor of a skirmish wargame, which although different in details can give you a general idea what to expect.