They could serve similar purposes. In terms of maturity nostr is younger. Here are the main differences from the point of view of nostr:
- In nostr there is no registration, your identity is your public key that you generate by yourself (lose that and you cannot recover it). You can connect to a bunch of different nostr relays with the same key, or use different ones.
- AFAIK nostr does NOT do end to end encrypted for group chat. But it does support end to end encryption for direct messages
- nostr does not do video/audio calls
- nostr does not host your images/files, you just put some URL in your messages
At its core nostr is a basic protocol where you send messages to a relay server and the relay passes them along to other people when they request them. And on top of those messages people implement extensions for features, full length posts, payments, etc. The are notions of followers and subscriptions (like twitter) but those are just tiny messages where you ask the relay for messages from person A or B. The list of specifications is here https://github.com/nostr-protocol/nips
Finally there are a few different nostr implementations for relays, clients and web interfaces. Some of them do not implement all the features, so you may need to shop around a bit if your are looking for some fancy features (check https://github.com/vishalxl/Nostr-Clients-Features-List).
Also some nostr highlights which I think don't have equivalent in matrix (but deserve nerd points)
- message expiration dates - the relay removes them after the deadline
- nostr has builtin proof of work to dissuade spam by forcing the client to do some computation before posting
- you can do reposts across relays or share relay addresses to people in another relay