UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.

Original post follows:

This post is intended as a central place that admins can reference regarding the XSS incident from this morning.

What happened?

A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.

Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.

Am I vulnerable?

If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.

I had custom emojis on my instance, what should I do?

This should be enough to mitigate now:

1. Remove custom emoji

DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;

2. Rotate your JWT secret (invalidates all current login sessions)

-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();

3. Restart Lemmy server

If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)

Legal

If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402

More context:

https://github.com/LemmyNet/lemmy-ui/issues/1895

https://github.com/LemmyNet/lemmy-ui/pull/1897

Hey folks!

So far, I have been the sole admin at lemm.ee. Most reports I receive don't really require any action from me - they are handled directly by moderators of the communities the reports originate from. Still, there is a chance that any reported post might contain content which would need to be purged from lemm.ee servers, so even reports that don't need any action should still be checked by an admin.

The volume of incoming issues has been quite manageable so far, but I have noticed a steady increase in daily reports lately. Additionally, there is nobody covering the report queue while I am sleeping or otherwise unable to access my computer.

I would like to ensure that there is a bit more redundancy in the admin team, so I am looking to potentially add one or two admins. Specifically, I am looking for somebody who would be willing to share the following responsibilities (copied from our administration policy):

Admins

• Ensure that there are no communities on lemm.ee which break lemm.ee rules
• Ban lemm.ee users who break our rules on other instances
• Ban users who consistently break rules across multiple communities
• Purge illegal content from lemm.ee

Note: I am not looking for help with system administration work at this time, this is strictly about administration within Lemmy itself.

Please be aware that being an admin is unfortunately quite a thankless job - if you're doing your job well, then most people won't even realize you're doing anything. OTOH, if you make mistakes, there will likely be many users calling you out in public. The main motivation for joining the admin team would need to be a desire to help build and maintain this instance as a great home for yourself and others.

If there is anybody who would be interested in helping out even despite the above disclaimer, please leave a comment with the following info:

• On a typical day, during what hours are you active on lemm.ee (with timezone info)
• Do you have any previous experience with moderation/administration
• Are you in agreement with the current state of the lemm.ee administration policy

Context

There have been a lot of posts and comments recently about Facebook entering the fediverse, and how different instances will handle it. Many people have asked me to commit to pre-emptively defederating from Threads before they even implement ActivityPub.

The lemm.ee federation policy states that it's not a goal for lemm.ee to curate content for our users, but we will certainly defederate any server which aims to systematically break our rules. I want to point out here that Facebook makes essentially all of its money from advertising, and lemm.ee has a no advertising rule - basically, Facebook has a built-in financial incentive to break our rules. ActivityPub has no protections against advertising, so it's likely we will end up having to eventually defederate from Threads just for this reason alone.

However, I would still like to get a feel for how many people in our instance are actually excited for potential federation with Threads. While personally I feel that any theoretical pros are by far outweighed by cons, I do want to use this opportunity to see how much of the community disagrees with me. I am not intending to run this instance as a democracy (sorry if anybody is disappointed by that), but I would still like to have a clear picture of user feedback for potentially major decisions such as this one. This is why I am asking every user who wants lemm.ee to federate with Facebook to please downvote this post.

Here are some reasons why I personally believe that Threads will have a negative effect on the fediverse

• As mentioned above, Facebook is completely driven by ad revenue. There is nothing stopping them from sending out ads as posts/comments with artificially inflated scores, which would ensure that their ads end up on the "all" page of federated servers.
• Threads already has more users than all Lemmy instances combined. Even if their algorithms don't apply to the rest of the fediverse directly, they can still completely dictate what the "all" page will look like for all instances by simply controlling what their own users see and vote on.
• Moderation does not seem to be a priority for Threads so far, meaning that they would create massive moderation workloads for smaller instances.
• In general, Facebook has shown countless times that they don't have their users best interests in mind. They view users as something to exploit for revenue. There are probably ways they are already thinking about hurting the fediverse that we can't even imagine yet.

By the way, we're not really in any rush today with our decision regarding federation

• Threads does not have ActivityPub support yet today
• Even if they add ActivityPub support, their UX is geared towards Mastodon-like usage - it seems unlikely that there would ever be proper interoperability between Threads and Lemmy
• We don't really know what to defederate from - it's completely possible that "threads.net" will not be their ActivityPub domain at all.

So go ahead and downvote if you feel defederation would be a mistake, and feel free to share your thoughts in the comments! It would be super helpful to me if folks who are in favor of federating with Threads could leave a comment explaining their reasoning.

Update:

By now, it's clear that there is a group of users who are in favor of federating with Threads. The breakdown is like this (based on downvotes):

• lemm.ee users: 136 in favor of federating with Threads
• Others: 288 in favor of federating with Threads

While it seems to be a minority, it's still quite a few users. There is no way to please all users in this situation - any decision I make will certainly inconvenience some of you, and I apologize for that.

A big thanks to everybody who has shared opinions and arguments in comments so far. I think there are several well written comments that have been unfairly downvoted, but I have personally read all comments and tried to respond to several as well. I will keep reading them as they come in.

The main facts I am working with right now are as follows:

• The majority of lemm.ee users are strongly opposed to immediately federating with Threads
• Facebook has a proven track record of exploiting users (and a built-in financial incentive to do so)
• We currently lack proper federation/moderation tools to allow us to properly handle rule breaking content from Facebook

Considering all of the above, I believe the initial approach for lemm.ee should be to defederate Threads, and then monitor the situation for a period of time to determine if federating with them in the future is a realistic option

In order to federate with them, the following conditions would need to be fulfilled:

• There needs to be actual interoperability between Threads and Lemmy
• Threads needs to prove that they are not flooding instances with rule-breaking content (mainly ads and bigotry for lemm.ee)
• There needs to be a mechanism to prevent feed manipulation by Threads algorithms (potentially this means discarding all incoming votes from Threads)

Note: this is an initial list, subject to change as we learn more about Threads.

Again, I realize this approach won't please everybody, but I really believe it's the best approach on a whole for now. Please feel free to keep adding comments and keep the discussion going if you think there is something I have not considered.

I'm a huge fan of SNES-era RPGs (and modern games which build on that style), so this looks like something I would really enjoy. There's even a big sale on Steam, I'm definitely going to try it out.

Thanks for helping me discover something new! 😃

👋 to all the newcomers, let me know if you need any help getting settled in!

Hey lemmings!

I wanted to share a quick update about our recent performance issues and how I have addressed them.

The last 24h have been a bit rough for lemm.ee.

Last night, I spent some time debugging federation issues with lemmy.world. We managed to significantly improve the situation - lemmy.world content is now reaching lemm.ee with a very high success rate - but this has had the effect of increasing incoming federation traffic on our servers significantly.

Additionally, we have been seeing steadily increasing normal user traffic over the past week, which is awesome from a community standpoint, but of course means that our servers have to do more work to keep up with all the new people.

To top things off, today there appeared a badly configured instance in the network, which was effectively launching a DoS attack against lemm.ee for several hours. Most likely it was unintentional, but unfortunately the end result was a sudden increase in our server load.

All these factors combined resulted in a really bad experience for most lemm.ee users today. Page load times have consistently been spiking into as much as 10 seconds or more for the whole day:

In fact, a lot of page loads just timed out with errors.

Fortunately, it seems I have managed to clear up the problems!

I have put a bunch of mitigations in place, and after monitoring the situation for the past hour, it seems that our performance issues have been resolved for now. So hopefully, you can enjoy browsing lemm.ee again without it feeling like torture!

Here are specific steps I took:

• I have doubled the hardware resources for our backend servers and database.
• I purchased a Cloudflare pro subscription for lemm.ee for 1 year. This took out a considerable chunk of my budget for lemm.ee, but in return it will allow me to analyze and optimize our cache usage to a far greater extent. I am already seeing vastly reduced load times for cacheable content (try opening https://lemm.ee a few times in a row as a logged out user - it should be blazing fast now!)
• I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.

Of course, all of the above is costly. Luckily, lemm.ee users have been very generous with donations in the month of June, and in fact a significant amount of donors have opted for monthly recurring contributions. This all gives me the confidence to increase our spending for now, and I am currently expecting to NOT increase my personal planned contribution of 150€/month, as the increased costs so far are entirely being covered by donations!

Let me take this opportunity to thank the sponsors who made the upgrades possible! All lemm.ee users are now enjoying better performance thanks to you, I could not have done it without you awesome people.

On a final note, I just want to say that I hope a lot of these issues can be solved by optimizations in Lemmy software itself in the future. I have been personally contributing several optimizations to the Lemmy codebase, and I know many others are focused on optimizations as well. Just throwing extra resources at the problem will probably not be a sustainable solution for very long 😅. But I am optimistic that we are moving in the right direction with the software changes, and we'll be enjoying reduced resource needs before long.

That's all I wanted to share today, I wish you all a great weekend!

Update: The maintenance has been completed!

Welcome to 0.18.1! Hopefully you'll notice some much needed UI tweaks as well as several performance optimizations.

I am still going through things and making sure everything looks good, but so far, I have not detected any major issues (fingers crossed). If you do have any problems, please comment below!

Original annoucenemnt below:

0.18.1 is coming!

As I mentioned in a previous post, this upgrade will require brief downtime. I will try to keep it as short as possible, but I can't guarantee an exact window, so if you really can't wait for lemm.ee to be back online, then the best advice I can give is to check back every 5 minutes or so.

As with any update, please be prepared for unexpected side-effects. If we find any major issues, then we can always roll back to good old 0.17.4. If we just find some minor issues, then most likely the overall experience will still be better on 0.18.1, so in that case we stay on 0.18.1 and try to get any potential issues fixed in 0.18.2. In either case, if you see anything weird after the upgrade, please report in the comments!

Tee, mis minu koduni viib, on iga suvi paksult rattureid täis. See tee on piisavalt kitsas, et kui 2 autot + rattur kolmekesi kõrvuti on, siis ratturil on suht suur oht peegliga pihta saada ja kraavi sõita.

Mul tuleb vähemalt korra nädalas ette olukord, kus hakkan mõnele ratturile tagant järgi jõudma, samal ajal näen, et vastassuunast tuleb auto. On aru saada, et see auto kulgeb rahulikult enne ratturist ja minust mööda, kui ma ise ratturini jõuan. Valmistun juba suunatuld sisse panema, et vastassuuna vabanedes kohe ratturist möödasõitu teha... ja siis saan aru, et vastutulev auto on hakkanud pidurit vajutama. Pidurdab hoo täpselt nii palju maha, et me jõuame enam-vähem samal hetkel ratturini. Mis mul muud üle jääb, pidurdan siis ka hoo maha, ja ootan ratturi taga, et tee möödasõiduks vabaneks. Tulemus on see, et kõik osapooled (peale ratturi) kaotavad aega ja kulutavad pisut rohkem kütust jne.

Eriti hull on siis, kui selle vastutuleva auto taga on pikem kolonn, kes siis keegi enne kiirendada ei saa, kui esimene jälle gaasipedaali üles on leidnud.

Mis värk sellega on? Kui ise vastassuunas ratturit näen, siis ei tule küll pähe, et peaks hoo maha pidurdama 😃

/rant over

Administration, moderation, and federation policy for lemm.ee

This post aims to clarify principles for how administration and federation is done on lemm.ee. It is intended to be an overview of general guidelines, not a formal set of rules.

Instance rules

This instance (like most others) has a set of rules which are always visible on the sidebar of the front page. All users of this instance are expected to follow these rules in all of their activities, including:

• Community moderation
• Posting
• Commenting

⚠️ Our rules apply even when you're posting in a community on another instance. For example, this means that you're not allowed to post advertisement spam using your lemm.ee account on any other instance (even if that other instance has no rules).

Each community hosted on lemm.ee is free to have additional rules in addition to our instance wide rules, but instance rules supercede any community rules and must always be enforced.

Responsibilities

Admins

• Ensure that there are no communities on lemm.ee which break lemm.ee rules
• Ban lemm.ee users who break our rules on other instances
• Ban users who consistently break rules across multiple communities
• Purge illegal content from lemm.ee

Moderators

• Ensure that posts and comments in their communities don't break rules
• Ban users from their communities for consistently breaking rules
• Ensure that they only provide accurate and clear reasons for mod actions

Users

• Downvote low quality content
• Report rule violations

⚠️ Admins are not responsible for censoring content from other instances.

In exceptional cases (illegal or extremely disturbing content), admins will step in and purge the content from lemm.ee servers, but in general it is undestood that our instance rules do not apply to external users on other instances, and censoring and curating external instances for our users is not a general goal for lemm.ee admins.

Federation

Lemmy is a federated network, so a lot of content will be posted on other instances. It is possible to limit which instances lemm.ee is federated with, this is called defederation.

Defederating another instance has the following effects:

• Our users will stop seeing new posts and comments from users of the defederated instance (on all instances)
• Users of the defederated instance will stop seeing new posts and comments from our users
• Users of the defederated instance will be prevented from participating in communities hosted on lemm.ee

As mentioned above, it is not a goal for lemm.ee to censor and curate external instances. While there are certainly instances which contain content that wouldn't be allowed on this instance, breaking our rules outside of this instance is not by itself enough of a reason for us to defederate other instances.

As a result, defederation is relatively rare on lemm.ee. You can read a more about our approach to defederation in this post.

That being said, we will defederate any instance which is directly harming lemm.ee users. This is up to the discretion of our admins. Some concrete examples of instances which we would defederate:

• An instance which has a 2:1 ratio of bots to users 🤖
• An instance which is focused on creating spam in the network
• An instance which systematically allows large groups of users to break lemm.ee rules in communities hosted on lemm.ee
• An instance which is knowingly spreading CSAM into the federated network

What should I do if I see content I don't like on another instance?

• If it's low quality content, you should always downvote ⬇️
• If you think it breaks local rules for the community or instance, then report it and local admins/mods will deal with it
  • Your reports will also reach lemm.ee admins, so if it's about illegal content, then we can purge it from lemm.ee servers
• If it's just some user being a prick, then you can block that specific user (lemm.ee admins will not take action in case of external users posting on external communities)
• If it's a community dedicated to being awful in some way, then you can block that specific community

Hello, lemmings!

I want to write a quick post about the recent wave of spam users on the federated network, and what steps I am taking to protect lemm.ee.

TL;DR:

• Tens of thousands of bots are signing up on small unprotected Lemmy instances. lemm.ee has not been targeted so far.
• To protect lemm.ee users from spam, I am going to start defederating such instances immediately.
• If spam bots start signing up on lemm.ee in the future, I will be (temporarily) closing new sign-ups until we have better tools to deal with bots.

Read on for more details!

Background

In the past few days, the growth of Lemmy user counts across the whole network has increased exponentially:

While there’s no question that this growth includes a big amount of real people coming over from Reddit, unfortunately, there is also a huge amount of automated sign-ups by bots.

For now, lemm.ee has not been affected by automated sign-ups. Bots seem to be avoiding instances which employ some or all of the following protections:

• E-mail verification
• Captcha on sign-up
• Sign-up applications with manual review

Currently, lemm.ee employs e-mail verifiaction and captchas.

There is a large amount of instances out there which don’t employ any of these protections. These are the instances the bots are mainly targeting. Most of these instances seem to be very small and not very active (often having <10 organic users and very few communities or posts). Some of these instances have taken notice of the bots and have begun taking steps to remove the bots and tighten up their sign-ups, but the majority have done nothing to combat the situation.

If you’re interested, I am maintaining a (non-comprehensive) list of most likely affected instances here. I have been updating it every now and then since yesterday in hopes of seeing positive change, but unfortunately, the situation seems to be getting worse.

Up until yesterday, these bots were mostly just quietly sitting there, but as of today, the bots have started posting spam. I have already been moderating several cases of automated spam, but I can only do this reactively.

Current solution: defederating spambot-infested instances

As I have mentioned previously in other threads, I do not really want to defederate any legitimate instances, but I will defederate instances which are actively making Lemmy worse for lemm.ee users. It seems clear in this case that the bots are planning to create a bad experience for all legitimiate users, and that the only way to really limit the effect of these bots is to defederate the instances where they are joining uncontrollably.

This is a lose-lose situation - if we don’t defederate them, then we risk exposing all lemm.ee users and communities to massive amounts of spam, but if we do defederate them, we are cutting off small instances who are clearly already struggling. I really like the idea of federated networks and people being able to curate their own feed from whatever instances they enjoy, so I do not make any defederation decisions lightly. At the end of the day, I can only choose the lesser evil, which at the moment does seem to be defederation.

Going forward, I will be regularly checking for spambot instances. If I detect new ones, I will be defederating lemm.ee from them immediately. Less regularly, I will also be checking to see if any of the instances have taken steps to deal with the bots - if they have, then I am planning to federate with them again. If anybody is interested in getting a cleaned up instance federated again, feel free to contact me over DM (if you're currently defederated, you can contact me on Matrix: @sunaurus:matrix.org).

What is the criteria for defederation?

While I don’t want to give out the exact details (it would just help spam bots with evading defederation), I can tell you in broad strokes that I am focused on defederating small instances with unnaturally huge user growth. I am currently not planning to defederate any popular instances with large communities and active moderation.

What does defederation mean for me as a lemm.ee user?

• You will not be able to see any new posts or comments from defederated instances made on ANY instance.
  • You will still be able to see old ones that they made before defederation
• Users from defederated instances will not be able to post or comment at all in communities hosted on lemm.ee

Future: if lemm.ee gets hit by spam bots, then sign-ups will be (temporarily) closed

While it’s true that we so far have not had a problem with automated sign-ups at lemm.ee, it is for sure possible that the bots in the future will be improved to automate e-mail verification and captcha solving. I do have some additional measures in place already to protect us, but nothing is guaranteed.

If it does happen that lemm.ee sign-ups become a target for spam sign-ups, I am intending to completely close sign-ups until there are better tools to deal with bots. There are several such tools already proposed, and I am planning to start development on one of them next month, so hopefully any potential closing of sign-ups would not last very long!

I want to emphasize that even if we end up closing sign-ups, your communities on lemm.ee will still be able to grow. As always, users from any federated instance will be able to subscribe to your communities and interact in all the ways that a local lemm.ee user would be able to.

To conclude, I really hope that this news does not ruin the experience for any of our users.

It's honestly a really bad situation and I wish I wouldn't have to be writing this post right now, but the reality is that things like this happen from time to time. We just have to deal with it in the best ways that we can. If you have any feedback or thoughts about any of this, please leave a comment below!

Today, a bunch of new instances appeared in the top of the user count list. It appears that these instances are all being bombarded by bot sign-ups.

For now, it seems that the bots are especially targeting instances that have:

• Open sign-ups
• No captcha
• No e-mail verification

I have put together a spreadsheet of some of the most suspicious cases here.

If this is affecting you, I would highly recommend considering one of the following options:

1. Close sign-ups entirely
2. Only allow sign-ups with applications
3. Enable e-mail verification + captcha for sign-ups

Additionally, I would recommend pre-emptively banning as many bot accounts as possible, before they start posting spam!

Please comment below if you have any questions or anything useful to add.

Update: on lemm.ee, I have defederated the most suspicious spambot-infested instances.

To clarify: this means small instances with an unnaturally fast explosion in user counts over the past day and very little organic activity. I plan to federate again if any of these instances get cleaned up. I have heard that other instances are planning (or already doing) this as well.

It's not a decision I took lightly, but I think protecting users from spam is a very important task for admins. Full info here: https://lemm.ee/post/197715

If you're an admin of an instance that's defederated from lemm.ee but wish to DM me, you can find me on Matrix: @sunaurus:matrix.org