Hello,

I made a lot of adjustments to the infrastructure to Discuss.Online. Please let me know if you notice any strange behavior or federation issues.

Thanks a lot, jgrim

ATTENTION: There is a known issue with the database cluster for discuss.online. It's a managed cluster by #DigitalOcean. They are working on resolving the issue. I'll update you once the issue is resolved.

You might see strange errors like the inability to log in or view specific posts.

Update 03/01/2024 @ 9:40 pm ET: The database has been migrated to a new server. Please, let us know if you experience any issues.

We are thrilled to announce the upcoming release of Sublinks, a groundbreaking Link Aggregation Social Network, joining the Fediverse. This innovative platform is designed to revolutionize how we share and discover online. Our dedicated team of volunteer contributors has worked tirelessly, utilizing technologies like Java, Go, TypeScript, and HTML to bring this vision to life. Sublinks promises a user-friendly interface and robust features that cater to diverse online communities. Stay tuned for our launch date, and get ready to experience a new era of social link sharing!

Sublinks will have a fully compatible API with Lemmy so all current Lemmy apps will also work with Sublinks. In fact, discuss.online will switch to Sublinks to fully replace Lemmy once we reach our Parity Milestone.

For more information, visit GitHub - Sublinks and sublinks.org.

Stay tuned for more regular updates as we progress.

This is the first of the monthly financial reports. This one is short and sweet; I'll provide more details next time.

These costs include hosting for:

1. discuss.online (lemmy)
2. discuss.online (matrix)
3. wiki.discuss.online (open wiki)
4. utter.online (mastodon)
5. social.photo (pixelfed)
6. Sublinks.org (sublinks)
7. participating.online(wiki)

Expenses for November

• Hosting: $575.32
• Github: $436.16
• Email: $21.45
• Total: $1,032.93

Income for November

• OpenCollective: $65
• Patreon: $15.11
• Total: $80.11

Year-to-date

• Year-to-date total expenses: $4,087.63
• Year-to-date total income: $454.77
• Year-to-date balance: -$3632.86

Donations... if you can spare it

• Patreon
• OpenCollective

Lemmy doesn't provide any analytics. Give this post a like and/or comment if you use this instance. I'm trying to gauge usage by humans.

I'll let this census run for about a week to gauge usage.

Results might cause different changes to the services.

I'm doing this because the cost of running this instance is higher than most other instances due to reliability and disaster recovery implementations. I'm curious if I should back down many of these costs if we're not used fully. I don't see much activity directly in this instance.

I know that discuss.online wasn't listed on join-lemmy.org for about a month due to a bug with the Lemmy crawler. They just recently fixed it with their redesign. We are now listed but no longer featured.

tl;dr; is this instance dying? Let me know if you're here.

We are running the latest release candidate for Lemmy, version 0.19.0-rc.1. This release candidate includes many changes.

Every release comes with its own challenges; however, it resolves previous ones. We hope this release will resolve a longstanding issue with sync not working, causing a lot of manual work from the admins to keep things in sync.

Please let us know if you experience any weird behavior.

Some known issues so far are:

• HTML entities characters in posts bodies (Changing & to &)
• The spoiler tag crashes the site if used in the sidebar
• Mobile apps are borked again

Update 10/25/2023

• We are now running release candidate 3.

We are aware of an issue with some clients not being able to sync (receiving 502 errors, etc..). It does not appear to affect all clients and the web view is still working. We will update this post as we know more and when we have resolved it.

Updates:

09-Sept 11:19 PM UTC We attempted to roll back from Lemmy v0.18.4-beta.8 to v0.18.4 (as well as even earlier beta versions) but the service failed to start. We've reverted back to v0.18.4-beta.8 for now and will continue investigating the cause of the sync issues.

10-Sept 3:30 AM UTC And... we are back, where we started. If you hadn't noticed, we were down for almost 2.5 hours while we tried to rollback the database migrations introduced in the beta releases so we could go back to the stable v0.18.4. When that failed, we tried to restore from the backup we took just prior to the rollback attempt. When that failed, we used Digital Ocean's restore to point in time and created a new DB cluster and switched to it. So... we are back on v0.18.4-beta.8 again, with the expectation that 3rd party clients are still broken. Unfortunately, we are not sure what we can do about this as we believe it is something in Lemmy that is breaking it. The mitigation we have put in place to avoid this going forward is that we no longer use the "latest" tag when loading a docker image and now hard lock to a specific version. We will update that version in a more intentional manner going forward.

10-Sept 4:15 AM UTC After some additional digging through the Lemmy diff for beta.5 to beta.8 (and then looking through the PRs that were merged to create this diff), the issue is indeed with the Lemmy code and is even called out in one of the PRs,. What this means is that the 3rd party clients that are not able to access the site will need to update their code to fix things on their side (or else they will be broken when v0.19 is released and all other instances update to it).

16-Sept 11:23 PM UTC Both Sync and Mlem have now released new versions that resolve connectivity issues. If there are any remaining clients we suggest you reach out directly to them to ask them. We'll be unpinning this post in a few hours

27-Sept 10:30 PM UTC We have made the decision to update to the RC (release candidate) for v19 as the beta we were previously on was missing some much-missed admin tools that we need to run the site with. This update appears to have affected 3rd parties again (I have tested Mlem and Memmy and get errors about authentication, however they still seem to load?). There also appears to be an issue with the home page loading that we are currently looking into how to resolve. As before, we don't anticipate any way to roll back. To steal a line from my daughter's favorite movie: Keep Moving Forward

28-Sept 12:03 AM UTC We found an issue with the markdown parsing that was causing the homepage sidebar to throw errors. We've resolved the issue and will update the core developers of Lemmy.

03-Oct 1:28 AM UTC We have implemented a workaround that should have fixed all apps. Please let us know if you are still having issues with an app (we may not be able to add a workaround for it, but we would like to at least track the ones not working)

Please welcome @[email protected] to the discuss.online admin team.

I’ve known lazyguru for over ten years. He’s a close friend. His skills are like mine. He will be able to help contribute to all projects of the discuss.online ecosystem.

lazyguru will help with moderation, system administration, monitoring, and development of new features. Joe has already improved some monitoring and is working on some changes to the infrastructure to make it more scalable. He will soon be helping with SocialCare.cloud once the first beta is released.

Please, take some time to welcome him!

Thanks, Jason

tl;dr;

We use SendGrid to send our emails out. By default, SendGrid enables link tracking. Link tracking was causing issues with allowing users to verify their accounts. In short, I disabled link tracking, and links should work for all new emails in the future.

Contact support if you need your account activated. [email protected] from the email you used to sign up.

Issue

When new users sign up or when existing users reset their password, an email is sent to verify ownership of the account. These emails are sent from us through a service called SendGrid.

SendGrid, by default, will mask all the URLs to enable what they call link tracking. It's a marketing thing to track engagement with email. I have no use for it, but I didn't disable it before.

The issue is that sometimes this link doesn't resolve. An HTTPS error is shown to the user.

Resolution

The quickest and most logical resolution was to disable link tracking. I never use it. I did just that; however, any previous emails sent will still be broken.

I will continue to investigate the issue in hopes of resolving previous links.

If you are unable to verify your account, please email [email protected] from the email you used to sign up. We'll manually verify each account that hasn't been able to be activated.

The small outage from this morning was from a planned update to Lemmy 18.3. Let me know if you notice any issues!

Hello,

There is a unique problem of the fediverse. When an instance goes offline, its communities will never sync again.

Recently, vlemmy.com shutdown. Quite a few communities synced with discuss.online and other instances. Because vlemmy.com is not longer brokering communication, these communities will never be in sync again.

We have a several options:

1. Leave them there. Do nothing.
2. Leave them there but make a post that it's dead and hope people see it.
3. Purge the communities. Act like they never existed.
4. Build some elaborate system to work around vlemmy being gone. This would take a lot of work and collaboration with other instances.

Let me know what makes the most sense to you as users. Are any of you still using vlemmy communities? What about long-term planning? Maybe this isn't an issue now but what if lemmy.world vanished?

Please, let me know what you think. I'm torn on this one.

Thanks, Jason

I'm looking to get member feedback on the design of the site.

Lemmy comes with many default core themes. Some parts of these themes are broken with dark mode. For example, upload boxes might show white if your system is set to light mode.

I want to set a decent default theme. If unaware, you can change your theme in your account settings. There are 10 different themes currently.

There are 12 different theme options:

1. Browser Default
   • When this is set, it uses Darkly or Litely depending on browser settings
2. Browser Default Compact
   • When this is set, it uses Darkly Compact or Litely Compact, depending on browser settings
3. Darkly
4. Darly Red
5. Darkly Compact
6. Darkly Pure Black
7. Litely
8. Litely Red
9. Litely Compact
10. Vaporwave Dark
11. Vaporwave Light
12. i386

I currently have the default set to Browser Default Compact, which should set dark or light mode depending on whether your browser is dark or light.

I use Vaporwave Dark, personally.

What do you use? Which do you prefer?

I could make something new. I tried some new ones that broke with an update. I didn't fix them after the update, so I removed them. I could create new ones again.

Let me know what you think. This is a discussion/poll on what I should do as a default.

Thanks, Jason

Timeline and reasoning behind recent infra changes

Recently, you may have noticed some planned outages and site issues. I've decided to scale down the size and resilience of the infrastructure. I want to explain why this is. The tl;dr; is cost.

Reasons

• I started discuss.online about 4 weeks ago. I had hoped that the reaction to Reddit's API changes would create a huge rush to something new, for the people, by the people; however, people did not respond this way.
• I built my Lemmy instance like any other enterprise software I have worked on. I planned for reliability and performance. This, of course, costs money. I wanted to be known as the poster child for how Lemmy should operate.
• As I built out the services from a single server instance to what it became the cost went up dramatically. I justified this assuming that the rush of traffic would provide enough donors to supplement the cost for better performance and reliability.
• The traffic load on discuss.online is less that extraordinary. I've decided that I've way over engineered the resilience and scale. Some SubReddits that had originally planned to stay closed decided to re-open. I no longer needed to be large.
• The pricing of the server had gotten way out of control. More than the cost of some of the largest instances in Lemmy while running a fraction of the user base.

Previous infrastructure

• Load balancer (2 Nodes @ $24/month total)
• Two front-end servers (2 Nodes @ $84/month total)
• Backend Server (1 Node @ $84/month total)
• Pictures server (1 Node @ $14/month total)
• Database (2 Nodes @ $240/month total)
• Object Storage ($5/month + Usage see: https://docs.digitalocean.com/products/spaces/details/pricing/)
• Extra Volume Storage ($10/month)
• wiki.discuss.online web node ($7/month)
• wiki.discuss.online database node ($15/month) [Total cost for Lemmy Alone: $483 + Usage]

Additionally:

• I run a server for log management that clears all lots after 14 days. This helps with finding issues. This has not changed. ($21/month)
• Mastdon server & DB ($42/$15/+storage ~ $60 total/month)
• Matrix server & DB ($42/$30/+storage ~ $75 total/month)

Total Monthly server cost out of pocket: ~$640/month.

The wiki, Mastodon, Matrix, & log servers all remained the same. The changes are for Lemmy only and will be the focus going forward.

First attempt

As you can see it was quite large. I've decided to scale way down. I attempted this on 7/12. However, I had some issues with configuration and database migration. That plan was abandoned. This is what it looked like:

Planned infrastructure

• Single instance server (1 Node @ $63/month total)
  • Includes front-end, backend, & pictures server.
• Database server (1 Node @ $60/month total)
• Object Storage ($5/month + Usage)
• Extra Volumes ($20 / month total)

[Total new cost: ~$150 + Usage]

Second attempt

I had discovered that the issues from the first attempt were caused by Lemmy's integration with Postgres. So I decided to take a second attempt. This is the current state:

Current infrastructure

• Single instance server (1 Node @ $63/month total)
  • Includes front-end, backend, & pictures server.
• Database server (1 Node @ $60/month total)
• Object Storage ($5/month + Usage)
• Extra Volumes ($20 / month total)
• wiki.discuss.online web node ($7/month)
• wiki.discuss.online database node ($15/month)

[Total new cost for Lemmy alone: ~$170 + Usage]

New total monthly server cost out of pocket: ~$330

My current monthly bill is already more than that from previous infrastructure @ $336.

Going forward

Going forward I plan to monitor performance and try to balance the benefits of a snappy instance with the cost it takes to get there. I am fully invested in growing this community. I plan to continue to financially contribute and have zero expectations to have everything covered; however, community interest is very important. I'm not going to overspend for a very small set of users.

If the growth of the instance continues or rapidly changes I'll start to scale back up.

I'm learning how to run a Lemmy server. I'll adjust to keep it going.

Here are my current priorities for this instance:

1. Security
   • This has to be number one for every instance. Where you decide to store your data is your choice again. You must be able to trust that your data is safe and bad actors cannot get it.
2. Resilience & backups
   • Like before, it's your data and I'm keeping it useable for you. I plan to keep it that way by providing disaster recovery steps and tools.
3. Performance
   • Performance is important to me mostly because it helps ensure trust. A site that responds well mans the admin cares.
4. Features
   • Lemmy is still very new and needs a lot of help. I plan to contribute to the core of Lemmy along with creating 3rd party tools to help grow the community. I've already began working on https://socialcare.dev. I hope to help supplement some missing core features with this tool and allow others to gain from it in the process.
5. User engagement
   • User engagement would be #1; however, everything before this is what makes user engagement possible. People must be using this site for it to matter and for me to justify cost and time.

Conclusion

If you notice a huge drop in performance or more issues than normal please let me know ASAP. I'd rather spend a bit more for a better experience.

Thanks, Jason

There will be a short 5 to 10-minute outage as I scale down the environment. I was ambitious about the growth of this community during the Rexidus / Rexxit. The server sizes for discuss.online way outpace the daily load. Usage is typically well below 5% on all servers. Lemmy has become much more memory efficient, and I've made other adjustments to improve performance. Memory usage hangs around 10% for the backend and 30% for front-end servers.

The target is to reduce to better align with current donation payments. Server costs are over 10x the monthly donations right now.

Let me know if you have any issues. Please, remember to check https://status.discuss.online for updates during the outage.

Thanks, Jason

If you noticed some strange site behavior, it could have been caused by some load testing I performed.

A misconfiguration in rate limiting caused all users to share the same bucket. This has caused strange behaviors for some users.

I've stopped load testing and adjusted rate limits to prevent further disruptions.

Let me know if you still experience any issues.

You can always email: [email protected] if you have issues logging in.

Thanks, Jason

We're currently running the latest release candidate of Lemmy 18.2. It has many fixes; however, there are still some remaining issues.

One example is that some buttons do not appear correctly on the dark themes. I will keep my custom dark theme as the default theme for now. I think it just looks better, even with broken upload buttons. I will attempt to apply a fix to it later today.

If you see any other issues, please, let me know so I can forward them to the core development team-- if you don't wish to do so yourself.

Thanks a lot! -jgrim

We've updated Lemmy 18.1. There may still be some issues; however, I believe it resolves much more than it creates. Let me know if you see anything funky.

Need moderators

Hello all!

I've created several communities here on discuss.online to help seed the content for growth.

I'm looking for mods to help water them! Gardening analogies are starting to grow on me.

Please DM me if you are interested.

Lemmy isn't known to have the best moderation tools. I have a somewhat useful way to moderate through Grafana. I'll provide dashboards and the ability to create alerts for you.

Community list

Here is the list of communities I'm looking for moderators for:

• FlipperZero (Unofficial)
• Old People Internet
• To The Fediverse
• Buy it for life
• Apple
• BattleBit Remastered
• Neurodivergent
• ~~Photography~~
• ~~Linus Tech Tips (Unofficial)~~

Please reach out even if you are slightly interested.

For communities already with mods more can be added. Please, reach out to the listed mod to apply.

Create your own!

I have community creation open. Feel free to create and moderate a different community.

Did you know:

It's possible to create blogs on Lemmy.

1. Create a new community.
2. Check "Only moderators can post to this community."

This will allow you to create your blog posts while allowing others to comment.

Thanks, Jason

This is happening from 8:45 pm ET until 9:30 pm ET.

For status updates go to https://status.discuss.online.

The update will take from 5 minutes to 1 hour.

You may have intermittent access to the site and API.

There has been a release of Lemmy version 0.18.

There is a very long list of changes for the back end and front end. You can see them here: https://github.com/LemmyNet/lemmy/releases

I do plan to upgrade at some point. However, I'm considering waiting until Lemmy 0.18.1. In 0.18 they removed captcha support. This has been a very useful tool to keep bots from taking over the instance. If I upgraded to 0.18 now I'd have to turn on registration applications. I fear that the registration application would be too much friction to gain users.

Please, let me know what you think of this change. An announcement will be made once an upgrade is to take place.

You can view instance status and maintenance schedules here: https://status.discuss.online.

On June 19, 2023, I setup scaling for the environment. My experience doing this is with AWS; however, I host this service with DigitalOcean. The good news is I now understand how DigitalOcean Load Balancers work. The bad news is I had intermittent outages throughout the day.

Sorry, I'll do better.

• jgrim

I noticed some random slowness when using the web client. It looks like the web client's backend is significantly CPU hungry. I decided to segregate the front-end and back-end clients. I can already tell a significant difference.

Here is an update from the core developers of Limmy. It addresses a lot of open questions and concerns by the community. It's a great read.

The good news is there is a beta site running a newer version of the Lemmy front-end at beta.discuss.online.

The bad news is I took the site down for like one hour trying to work out the nginx configuration. Whoops.

Sorry for the outage!