Beehaw Support

this was made a few days ago when i was off but nobody announced it, so i guess i'm doing that now. the sidebar describes its purpose as follows:

Spirituality is a broad concept with room for many perspectives. In general, it includes a sense of connection to something bigger than ourselves, and it typically involves a search for meaning in life. As such, it is a universal human experience—something that touches us all. People may describe a spiritual experience as sacred or transcendent or simply a deep sense of aliveness and interconnectedness.

Some may find that their spiritual life is intricately linked to their association with a church, temple, mosque, or synagogue. Others may pray or find comfort in a personal relationship with God or a higher power. Still others seek meaning through their connections to nature or art. Like your sense of purpose, your personal definition of spirituality may change throughout your life, adapting to your own experiences and relationships

Improving Beehaw

BLUF: The operations team at Beehaw has worked to increase site performance and uptime. This includes proactive monitoring to prevent problems from escalating and planning for future likely events.

Problem: Emails only sent to approved users, not denials; denied users can't reapply with the same username

• Solution: Made it so denied users get emails and their usernames freed up to re-use

Details:

• Disabled Docker postfix container; Lemmy runs on a Linux host that can use postfix itself, without any overhead

• Modified various postfix components to accept localhost (same system) email traffic only

• Created two different scripts to:

  • Check the Lemmy database once in while, for denied users, send them an email and delete the user from the database
    • User can use the same username to register again!
  • Send out emails to those users (and also, make the other Lemmy emails look nicer)

• Sending so many emails from our provider caused the emails to end up in spam!! We had to change a bit of the outgoing flow

  • DKIM and SPF setup
  • Changed outgoing emails to relay through Mailgun instead of through our VPS

• Configure Lemmy containers to use the host postfix as mail transport

All is well?

Problem: NO file level backups, only full image snapshots

• Solution: Procured backup storage (Backblaze B2) and setup system backups, tested restoration (successfully)

Details:

• Requested Funds from Beehaw org to be spent on purchase of cloud based storage, B2 - approved (thank you for the donations)

• Installed and configured restic encrypted backups of key system files -> b2 'offsite'. This means, even the data from Beehaw that is saved there, is encrypted and no one else can read that information

• Verified scheduled backups are being run every day, to b2. Important information such as the Lemmy volumes, pictures, configurations for various services, and a database dump are included in such

• Verified restoration works! Had a small issue with the pictrs migration to object storage (b2). Restored the entire pictrs volume from restic b2 backup successfully. Backups work!

sorry for that downtime, but hey.. it worked

Problem: No metrics/monitoring; what do we focus on to fix?

• Solution: Configured external system monitoring via external SNMP, internal monitoring for services/scripts

Details:

• Using an existing self-hosted Network Monitoring Solution (thus, no cost), established monitoring of Beehaw.org systems via SNMP
• This gives us important metrics such as network bandwidth usage, Memory, and CPU usage tracking down to which process are using the most, parsing system event logs and tracking disk IO/Usage
• Host based monitoring that is configured to perform actions for known error occurrences and attempts to automatically resolve them. Such as Lemmy app; crashing; again
• Alerting for unexpected events or prolonged outages. Spams the crap out of @admin and @Lionir. They love me
• Database level tracking for 'expensive' queries to know where the time and effort is spent for Lemmy. Helps us to report these issues to the developers and get it fixed.

With this information we've determined the areas to focus on are database performance and storage concerns. We'll be moving our image storage to a CDN if possible to help with bandwidth and storage costs.

Peace of mind, and let the poor admins sleep!

Problem: Lemmy is really slow and more resources for it are REALLY expensive

• Solution: Based on metrics (see above), tuned and configured various applications to improve performance and uptime

Details:

• I know it doesn't seem like it, but really, uptime has been better with a few exceptions
• Modified NGINX (web server) to cache items and load balance between UI instances (currently running 2 lemmy-ui containers)
• Setup frontend varnish cache to decrease backend (Lemmy/DB) load. Save images and other content before hitting the webserver; saves on CPU resources and connections, but no savings to bandwidth cost
• Artificially restricting resource usage (memory, CPU) to prove that Lemmy can run on less hardware without a ton of problems. Need to reduce the cost of running Beehaw

THE DATABASE

This gets it's own section. Look, the largest issue with Lemmy performance is currently the database. We've spent a lot of time attempting to track down why and what it is, and then fixing what we reliably can. However, none of us are rust developers or database admins. We know where Lemmy spends its time in the DB but not why and really don't know how to fix it in the code. If you've complained about why is Lemmy/Beehaw so slow this is it; this is the reason.

So since I can't code rust, what do we do? Fix it where we can! Postgresql server setting tuning and changes. Changed the following items in postgresql to give better performance based on our load and hardware:

 huge_pages = on # requires sysctl.conf changes and a system reboot
 shared_buffers = 2GB
 max_connections = 150
 work_mem = 3MB
 maintenance_work_mem = 256MB
 temp_file_limit = 4GB
 min_wal_size = 1GB
 max_wal_size = 4GB
 effective_cache_size = 3GB
 random_page_cost = 1.2
 wal_buffers = 16MB
 bgwriter_delay = 100ms
 bgwriter_lru_maxpages = 150
 effective_io_concurrency = 200
 max_worker_processes = 4 
 max_parallel_workers_per_gather = 2
 max_parallel_maintenance_workers = 2
 max_parallel_workers = 6
 synchronous_commit = off  	
 shared_preload_libraries = 'pg_stat_statements'
 pg_stat_statements.track = all

Now I'm not saying all of these had an affect, or even a cumulative affect; just the values we've changed. Be sure to use your own system values and not copy the above. The three largest changes I'd say are key to do are synchronous_commit = off, huge_pages = on and work_mem = 3MB. This article may help you understand a few of those changes.

With these changes, the database seems to be working a damn sight better even under heavier loads. There are still a lot of inefficiencies that can be fixed with the Lemmy app for these queries. A user phiresky has made some huge improvements there and we're hoping to see those pulled into main Lemmy on the next full release.

Problem: Lemmy errors aren't helpful and sometimes don't even reach the user (UI)

• Solution: Make our own UI with ~~blackjack and hookers~~ propagation for backend Lemmy errors. Some of these fixes have been merged into Lemmy main codebase

Details

• Yeah, we did that. Including some other UI niceties. Main thing is, you need to pull in the lemmy-ui code make your changes locally, and then use that custom image as your UI for docker
• Made some changes to a custom lemmy-ui image such as handling a few JSON parsed error better, improving feedback given to the user
• Remove and/or move some elements around, change the CSS spacing
• Change the node server to listen to system signals sent to it, such as a graceful docker restart
• Other minor changes to assist caching, changed the container image to Debian based instead of Alpine (reducing crashes)

The end?

No, not by far. But I am about to hit the character limit for Lemmy posts. There have been many other changes and additions to Beehaw operations, these are but a few of the key changes. Sharing with the broader community so those of you also running Lemmy, can see if these changes help you too. Ask questions and I'll discuss and answer what I can; no secret sauce or passwords though; I'm not ChatGPT.

Shout out to @[email protected] , @[email protected] and @[email protected] for continuing to work with me to keep Beehaw running smoothly.

Thanks all you Beeple, for being here and putting up with our growing pains!

We have to move to object storage to have efficient image storage. We are currently at 70% of our disk storage.

We are sorry for the short notice and we don't know how long it will take though we believe it will take multiple hours.

Small post mortem :

1. The migration required us to be down - we thought we could run without images
2. It took much longer than initially anticipated
3. It broke at close to 2:30AM - 3 hours into the migration and I couldn't fix it so the server has been down doing nothing for hours now
4. We're booting back up from the backup made before the migration was attempted, we'll try another strategy.

There might've been some data loss in images, we're looking into it. In the meanwhile, if your profile picture or banner is broken, feel free to re-upload them.

Update :

Hi Beeple!

We're trying this again.

This time, Beehaw should remain available though no pictures will be able to be uploaded. The error will likely be weird because Lemmy will think it is possible but we will block the upload from happening.

We'll take a snapshot of the pictures before the migration in 60 minutes at 21:00 UTC - it should take around one hour, do the migration and testing on our own before shipping on Beehaw.

Once we've resolved all these kinks, Beehaw will momentarily go down and then back up with the migration complete without losing any old pictures.

I'm just curious if it is on the table at some point. I only see a small slice of beehaw when I'm logged in but the active participation feels like it is on a downward trend. Like, there appears to be ~700 on here right now. I know numbers aren't everything, but overall engagement is important. I'm on several instances with different accounts. I've been gravitating towards my .world account because it is so active. I get a grouchy or rude reply still from time to time, but it seems like most of the trolls have gone or been removed. That instance seems to be maturing fast and growing some personality all its own. The server seems constantly stressed, but Ruud is holding it together. The moderation seems much more in check now too. That's just my perspective. I'd like to see everyone come together again, but I am just one user.

I’ve started to see some posts from those instances showing up on the “all” page when sorting by new, so wanted to know if the admins refederated with those instances.

Hey all,

Moderation philosophy posts started out as an exercise by myself to put down some of my thoughts on running communities that I'd learned over the years. As they continued I started to more heavily involve the other admins in the writing and brainstorming. This most recent post involved a lot of moderator voices as well, which is super exciting! This is a community, and we want the voices at all levels to represent the community and how it's run.

This is probably the first of several posts on moderation philosophy, how we make decisions, and an exercise to bring additional transparency to how we operate.

our June 2023 financial update.

obligatory preface: we're 100%-user funded and everything you donate to us specifically goes to the website, or any outside labor we pay to do something for us.

because of the unique circumstances of this month i won't report an expected yearly expense like i did last time. that'll probably come next month, when our finances are closer to a useful baseline.

overall expenses this month: $566.98

this is a mighty looking expense, but only $371.98 of it is infrastructure (and even less of that is actually site hosting).

$312.54 for Digital Ocean hosting, which can be further subdivided into

• $241.47 for hosting the site itself
• $48.29 for backups
• $11.87 for site snapshots
• $10.91 for bandwidth overage (1091.10GB @ $0.01/GB)

$15.28 for Hive, an internal chat platform we're trying to set up (also being hosted on Digital Ocean, but distinct enough to break out from overall DO hosting)

• $13.89 for hosting Hive
• $1.39 for backups

~$39.16 for email functionality, which can be further subdivided into

• $35/mo for Mailgun (handles outbound emails, so approval/denial/notifications emails; also lets us not get marked as spam)
• ~$4.16/mo ($50/yr, already paid in full) for Fastmail (handles all inbound emails)

$5/mo for 1TB of backup storage with BackBlaze (redundant backup system that's standalone from Digital Ocean)

the remaining $195 of this month's expenses have gone to paying @[email protected] for his community icons. we do so at a rate of $5 per icon and he's done 39 of them for us (36 of which are live so far).

overall: we definitely think we're able to downsize infrastructure costs going forward. we're already investigating how best to do that (both in terms of host and overall cost)--there's no ETA for a few reasons, but this month should not be representative of many more subsequent months.

overall contributions this month: $3,870.44

we also have an incredible amount of support, so that really helps things as far as "being able to take time to get everything right". according to OpenCollective, we currently have approximately:

• 97 monthly contributions, totaling $549.58
• 9 yearly contributions, totaling $254.99
• 149 one-time donations, totaling $3,065.87

between monthly and yearly contributions, this means we are still more-or-less breaking even and sustainable overall with this month's costs. obviously, we would like to be substantially moreso though, through either lower costs, more donations, or a combination of both.

total end of month balance: $3,591.33

• yes yes, this is already out of date by a bunch. expect it to be like that, i use UTC for our reports lol.

expense runway, assuming no further donations

• assuming expenses like ours this month: we have about 6 months and 10 days of runway
• assuming just expenses like our infrastructure this month: we have 10 and a half months of runway

if you'd like to make the runway longer (and reward us for even having this site up today after yesterday's complete fiasco), now is a good time to donate :)

you know the drill i'm sure, let's not waste time. many of you have already noticed we've created the first batch of our six new communities, which are:

• AskBeehaw: a community which is exactly what it says it is, asking the Beehaw community stuff! this was our most popular community on the survey, and it's also inclusive of AMAs or stuff like out of the loop (which overlaps somewhat with Chat--we're fine with that).
• Tabletop Gaming: after some delay, the tabletop and boardgaming communities have their own section! this community is, as noted, inclusive of boardgames and similar things to them--we've already got a mod in the section looking out for that side of the community, so that's nice :)
• US News: we've converted !news into World News (see here for why in brief) to accommodate that popular demand, which means that US News now gets its own community! in conjunction with us the community there is being cultivated by @[email protected], who has quite a lot of news experience and has a pretty clear vision of what that community will hopefully bring to the community table. we hope you'll like that vision.

other three communities should drop about next week, so be on the lookout for those.

as noted in the other announcement we're also taking mod applications. we're taking them for both new and old communities, so please feel free to apply over there if you're interested. thanks

Jebora for Android refuses to work because it doesn't support v0.17.4 any more.

unless everyone else is getting those same errors and they just happen to perfectly correlate with when i send a link on discord

anyone have any ideas?

Title. I just get a "404: couldnt_find_community" error. This is the direct link to the sub: https://programming.dev/c/devops

So when using the Beehaw PWA created by Chrome or Firefox the nav buttons at the bottom of the screen are hidden, requiring swiping up from the bottom of the screen every time I want to hit the back button.

I'm also using another Lemmy PWA for another instance and that one works as expected, with the bottom buttons staying on the screen. I believe the Beehaw PWA is in fullscreen and the other isn't.

cross-posted from: https://lemmy.ca/post/942473

I have a fork of Jerboa to help with any login issues for those with accounts still on 0.17.x, until either the servers upgrade or until a newer version allows connecting to both 0.18 and pre 0.18 servers.

Later tomorrow I will include a version with the app name changed so that it can run alongside dessalines' version.

I just updated to the most recent version of Jerboa (the Android Lemmy client), and when I tried to log in to Beehaw, I got a message saying that Beehaw's server version is older than the minimum version now required by Jerboa.

Not complaining. Just FYI.

I've noticed that when browsing through threads, some users have the @ prefix, while others don't, despite both being members of this instance.

What's the reason for that? Is it something to do with the "Display name" option in the settings?

I can login to my beehaw account through an app but on my computer it just sits and spins forever. Gave it the whole weekend thinking it was just a capacity issue but I think the login might just be broken.

Can anyone else confirm?

Three questions:

• What does the language dropdown actually do?
• What does the "if you deselect Undetermined" warning banner mean?
• Are we able to tag our comments as being in whatever specific language across the board, by default (e.g., English)?

@support

Hey y'all! I got an email a week ago saying my account was approved, but I haven't been able to log in via Firefox on Debian or Android. The loading wheel just keeps spinning after I click the login button and doesn't stop. Does anybody here know it there's been issues with new accounts lately?

I have seen them around the Fediverse and on Reddit and if implemented properly it helps greatly.

I don't know about others but I appreciate a good TL/DR bot to help determine if I want to read the full article posted.

is this something that has been considered for communities like news? I know I would like to see this.

A working example seems to be: https://lemmy.world/u/tldrbot that used this under the hood: https://github.com/SleeplessOne1917/lemmy-bot

What's the story with this? I'm getting it every time I log into Beehaw over the last few days. And Beehaw keeps crashing without warning and going straight to that same warning. Should I worry?

This isn't happening on any of my other Fediverse accounts, by the way.