this post was submitted on 10 Jul 2023
240 points (100.0% liked)

Lemmy Administration

64 readers
2 users here now

Anything about running your own Lemmy instance. Including how to install it, maintain and customise it.

Be sure to check out the docs: https://join-lemmy.org/docs/en/administration/administration.html

If you have any problems, describe them here and we will try to help you fixing them.

founded 4 years ago
MODERATORS
 

UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.

Original post follows:


This post is intended as a central place that admins can reference regarding the XSS incident from this morning.

What happened?

A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.

Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.

Am I vulnerable?

If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.

I had custom emojis on my instance, what should I do?

This should be enough to mitigate now:

  1. Remove custom emoji
DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;
  1. Rotate your JWT secret (invalidates all current login sessions)
-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();
  1. Restart Lemmy server

If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)

Legal

If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402

More context:

https://github.com/LemmyNet/lemmy-ui/issues/1895

https://github.com/LemmyNet/lemmy-ui/pull/1897

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 29 points 1 year ago (2 children)

Thanks for posting. There really should be a button which allows the admins to log everyone out for crisis situations like there I think

[–] [email protected] 22 points 1 year ago (1 children)

Changing the JWT secret does this. So instead of a button, its a line of code, making it less likely to be done by mistake.

[–] [email protected] 7 points 1 year ago (1 children)

True as that is, not every admin has DB access and those with DB access might be AFK.

[–] [email protected] 13 points 1 year ago (1 children)

Surely this exploit proves that it's best to minimise the number of administrative actions available in the UI?

[–] [email protected] 6 points 1 year ago

Destructive actions, sure. This would not be one such actions and it would be self-defeating.

[–] [email protected] 5 points 1 year ago* (last edited 1 year ago) (13 children)

In crisis situations like this, the last thing you want is to rely on some portion of the application UI working.

load more comments (13 replies)
[–] [email protected] 23 points 1 year ago (2 children)

It seems to me that the scope of this could have been mitigated with a simple privilege separation policy for admin server accounts but I see a lot of (what looks like) server admins using that account as their daily driver.

Also, lemmy-ui should post a security advisory to their github.

[–] spacedancer 6 points 1 year ago (1 children)

Welp, that is some simple lapse of separation-of-duties principles that shouldn't have been.

[–] [email protected] 5 points 1 year ago (1 children)

I don't pretend to know the demographics of lemmy server admins, but my gut says it's predominantly hobbyists and devops types, rather than grizzled system admins.

[–] MigratingtoLemmy 4 points 1 year ago (1 children)

People in DevOps should know better than reusing their accounts. It doesn't take a seasoned system admin to know about basic OPSEC and separation amongst profiles.

[–] [email protected] 3 points 1 year ago (1 children)

Agreed. I'm pointing it out in hopes that becomes one of the takeaways of this incident.

(Unrelated, why are you marked at a bot account?)

[–] MigratingtoLemmy 1 points 1 year ago (3 children)

Wait, why am I marked as a bot account?

This is the first time I've heard about this. Whom should I ask for help?

load more comments (3 replies)
[–] CMahaff 23 points 1 year ago* (last edited 1 year ago) (2 children)

This bug also needs to be fixed so that a user logging out / changing password actually correctly invalidates their JWT token: https://github.com/LemmyNet/lemmy/issues/3364

I suspect that last night even if an admin knew their account was compromised, they probably couldn't do anything about it without DB access.

[–] AlmightySnoo 5 points 1 year ago* (last edited 1 year ago)

This bug also needs to be fixed so that a user logging out / changing password actually correctly invalidates their JWT token: https://github.com/LemmyNet/lemmy/issues/3364

Okay it is insane that that issue is 2 weeks old and was not prioritized and because of that the hacker was gifted 2 weeks to prepare an attack.

I love Lemmy and am grateful for the work of the devs. But I see huge issues with the LemmyNet governance.

Previously it was captchas that were removed because one of the two main devs had a strong personal belief that captchas are "useless" and wanted to impose that belief upon everyone, which then led to an enormous wave of bots with the 0.18.0 as captchas were removed. I'm glad he was then convinced that he was wrong to remove them and then took steps to revert that decision, but it took too long and many instances suffered performance problems by being forced to stay with the 0.17.6 version because it had captchas.

Right now the same thing happens again with issues about security being left unanswered for two weeks. I believe right now all feature developments should be paused and a security audit of the whole code base should be the #1 priority.

This is just very bad. Proper governance and prioritization would have avoided exposing minors to lemon party porn and other disgusting content.

I can only imagine how helpless the admin whose account was compromised felt if she didn't have access to the database to invalidate those tokens, and it could have been prevented if that issue was properly prioritized.

[–] [email protected] 6 points 1 year ago

Thanks for the info.

[–] [email protected] 5 points 1 year ago (1 children)

My personal instance has no custom emoji's, yet the SQL to remove onload= from content removed 16 effected comments. Does that mean federated instances are effected?

[–] [email protected] 7 points 1 year ago (1 children)

This does not mean your instance was affected. You're just cleaning up comments which may or may not have successfully worked on other instances by abusing the emoji bug there.

[–] [email protected] 5 points 1 year ago

Roger okay. Thanks for clearing that up.

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago) (2 children)

I'm curious why the UI update to 0.18.2-rc.1 isn't mentioned anywhere in this thread. I understood that updating the UI was a fix for this exploit.

[–] [email protected] 4 points 1 year ago (1 children)

My post predates the new versions, but I am editing the post now!

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

Awesome, thank you!

edit: you may want to specify that it's lemmy-ui that has the update, not lemmy.

[–] [email protected] 2 points 1 year ago

We'll likely do a lemmy release tomorrow with this fix included.

[–] fututio_enjoyer 3 points 1 year ago (2 children)

Stolen cookies gave attackers access to all private messages

Aren't "private messages" not really private on lemmy anyway?

[–] [email protected] 7 points 1 year ago* (last edited 1 year ago)

Under normal circumstances, they are only visible to:

  • sender (and whoever has DB access on the sender's instance)
  • recipient (and whoever has DB access on the recipient's instance)
  • in case a private message is reported, all admins of the reporter's instance

It is still considered a breach of user data if such messages are leaked.

[–] A_Random_Idiot 2 points 1 year ago (1 children)

"private" messages are not private anywhere.

[–] [email protected] 11 points 1 year ago

unless they're E2EE

[–] [email protected] 2 points 1 year ago (2 children)

How did the hackers get the cookies in the first place? Compromised devices on the clients?

[–] devnill 3 points 1 year ago (1 children)

🅴🆁🅶🆈 1•

How did the hackers get the cookies in the first place? Compromised devices on the clients?

I'm not affiliated with lemmy.world or any other instance but I do software and can explain some of the jargon above. XSS is the abbreviation of one of the most common attacks we see on websites, cross-site scripting. This attack works by having some vulnerable code which arbitrarily executes some javascript on a users browser.

In this case, the attacker seems to have found a vulnerability where a specially crafted character is executed when users read posts or comments containing it. In this case, it was especially bad because of how passwords are stored. When you log in to pretty much any website, passwords are stored in the form of cookies; small pieces of data that are passed back and forth to the web server and the client automatically. Usually, these cookies are set to not be readable by javascript, but in this case, it appears that that flag was not set. This allowed the XSS exploit to be sent back to a computer which was set up to grab these cookies.

One thing to note is that although the cookies were stolen, our passwords wouldn't have immediately been compromised. Our login cookies store jwt tokens; a cryptographically signed message proving that we provided a password previously. That's not to say that further escalation isn't possible, but its hard to say for certain how far the hack went before being noticed.

[–] [email protected] 1 points 1 year ago

Thank you for the detailed explanation :)

load more comments
view more: next ›