Key points off the top of my head:

• Both Signal and Matrix have solid cryptography. (In fact, the Matrix Megolm protocol is built upon Signal's Olm protocol.)
• Signal's client is open-source, but not its server. Matrix clients and servers are open-source. (Note that open server code is less important than open client code when end-to-end encryption is used.)
• Both have had security audits, and presumably will have more in the future.
• Both have cross-platform support. Matrix has a greater variety of clients, and last time I checked, the desktop clients were better than Signal's.
• Signal prides itself on minimizing metadata exposure. Matrix started later, and hasn't yet moved all metadata (e.g. reactions, group membership?) to an encrypted channel. (Note that metadata protection can't ever be complete for either one of them, because an attacker that can gain access to the servers could just as well gain access to the routers, and correlate packets at the network level.)
• Signal is a centralized service (hosted by Amazon last time I checked), meaning all traffic flows through one place and can be shut down or monitored there. Matrix is federated, much like Lemmy.
• Signal does not offer self-hosting. Matrix does. (One might do this to be sure they always have a copy of their past messages, or to be independent of public server outages, or to be the sole keeper of data and metadata in conversations between local users.)
• Signal requires a phone number, which makes having an anonymous account challenging for many people. Matrix doesn't require anything, but some Matrix servers require an email address for account recovery purposes.
• Signal apparently integrates cryptocurrency payments. Matrix does not.

The most obvious security vulnerability that I can think of is that the person you are communicating with (or, conceivably, oneself, as well) is using an insecure/compromised application that may be leaking information.

Indeed, end-to-end encryption is unable to protect the endpoints. That's true of both messengers.

Since you brought it up, note that smartphones generally run system-level software that is controlled by the OS maker, meaning that Google or Apple have the ability to monitor those endpoints. Signal's approach to addressing this is to offer a Google-free build of their app, if you can find it, buried on their web site and with a scary-looking warning against using it. Matrix publishes their apps on F-Droid, so there's an easy Google-free option. Of course, most of the people using either messenger will have downloaded from the big app stores, so even if your phone and app are Google-free, it's likely that most of your contacts' phones are not. I believe Matrix has an advantage here, simply because their Google-free builds are easier to find and keep updated, and are therefore probably in use by more people.

Signal goes very far to protect even the tiniest bits of metadata.

For example see sealed sender, private contact discovery and group v2.

On the other hand, matrix stores your profile info, group membership, and ongoing conversation metadata in plaintext, some of them replicated across homeservers. In addition to metadata that matrix doesn't encrypt, they also do not encrypt some actual data like emoji reactions.

Edit: clarified that conversations are not in plaintext. My wording what confusing as hell sorry.

Matrix is less secure than Signal. While Signal and Matrix use the same encryption, Matrix doesn't encrypt everything. This includes: message sender, message timestamps, reactions, members, read receipts, etc. All of this data can be accessed by the homeserver admin. On Matrix, you should assume that only the message content itself (text and attachments) is encrypted. Your account data is also not protected, you have to trust your homeserver admin. Signal is designed not to trust the server. It's important to consider your threat model. Matrix doesn't require a phone number, which makes it better for anonymity, but Signal has better security.

This is a good explanation of Matrix's metadata leaks: https://web.archive.org/web/20210618055112/http://serpentsec.1337.cx/matrix

Matrix can't be forced to put backdoors into their software because they are not s company. Signal can.

There are several different questions here that run the gamut of security related questions, not just privacy.

If you want a lay persons answer, the gist of it is both are secure enough for the vast majority of situations. If you don't have nation states after you, use what is most convenient for your primary use case.

If you are worried about nation states... You need to do your own technical analysis, and if you are incapable of doing so... You should not be trusting random idiots on the web.

That said... The biggest difference is that signal is more secure by default, forcing end to end encryption on all communications but you have to trust a closed source private organization. Arguable the signal protocol is probably provably more secure with perfect forward secrecy (the double ratchet algorithm is legit).

Matrix has a significantly larger public surface area for non nation state level actors to observe and Messaging is arguably slower... But you have much more freedom with how much trust you give entities. Unfortunately, more of the security is up to you and your contacts which makes it less secure. (We on average are terrible at security). Matrix is probably also more resilient as a network.

Finally, Signal does not provide any anonymity whatsoever. I'm fact, it directly ties your messaging to other highly personal information. (But nobody other than you and the other participants can read any of the messages). Provide you know your contacts in real life, you can prove message authorship (you know for a fact that the message was written on their device and nobody other than the intended recipients read it).

Matrix does provide anonymous access. If this is something you need... This is the answer.

Why not Signal?

for decentralyzed private and anonymous messaging, try Session. its like cross between matrix and signal. it is supposed to hide metadata, not just message content. #notsponsored

Threema is very security focused at the expense of user experience. You lost your private key? Tough luck, here's a new account and everyone will have to verify your new account again. I have had conversations with friends over five or six of their accounts because they use threema just like WhatsApp. I stopped recommending threema to folks without a basic technical understanding. Signal is more than good enough for their needs.

Signal is a smartphone app, so you can trust it as much as you trust smartphones. :) It is more reliable and user friendly however, which is important too.

https://securemessagingapps.com