this post was submitted on 30 Jun 2023
61 points (96.9% liked)

Selfhosted

39205 readers
313 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

Hi, I'm new with self-hosting but managed to set up my own Lemmy and Mastodon instances on a VPS recently. However, I ran into an issue with disk space quite rapidly (which I had way too few, because I started with the cheapest, smallest package for my VPS).

Now I prepare a new setup, where I'll be able to dynamically scale disk space as needed, but this can get expensive quickly. Therefor my question: How much disk space do I typically need for private (1-3 user) instances of Lemmy and Mastodon? Are there settings, where I can limit the disk space utilization (at the cost of older stored content being overwritten)?

I would be fine with needing up to like 30-40 GB, but any more than that would be getting kinda expensive ....

top 23 comments
sorted by: hot top controversial new old
[–] ruud 18 points 1 year ago (1 children)

Do you need Mastodon, or can it be something similar? Mastodon does a lot of caching. For my single-user instance I run Akkoma, which is very leight-weight and has all (even more) features Mastodon has.

[–] ruud 21 points 1 year ago (3 children)

By the way this is Media disk usage for my Mastodon instance, but it's not really small ;-)

[–] void_wanderer 3 points 1 year ago (1 children)

So, how much do you spend, why are you doing it, and do you get any funding or paying this or of your pocket? (just trying to understand how the fediverse works)

[–] [email protected] 2 points 1 year ago

If this helps storage is cheap. See this for deals on high capacity drives https://shucks.top/

[–] bev 2 points 1 year ago

I appreciate the work you do. However, is this sustainable for the long term?

[–] [email protected] 1 points 1 year ago

Damn, 4GB of custom emojis?! Does your instance have a lot of them or is it just caching all emojis from every other instance?

[–] [email protected] 12 points 1 year ago (1 children)

That will depend on usage. If you subscribe to a lot of communities that are very active the database will grow fairly fast. If your users post a ton of large files then that will grow much much faster than the database. Your instance also caches image thumbnails so that can grow somewhat fast too.

Currently sitting at 750 MB for pictrs and 500MB for postgres after a week, so you may want to plan a decent size but 30-40GB should be fine for a while to get started. Add some monitoring for when it reaches 10, 20 and 30GB and keep an eye on it.

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago) (1 children)

Heya, sorry for the necropost, but would you mind sharing how you're doing on storage these days? I'm looking at spinning up a Lemmy instance of my own and I'm curious about the storage aspect on small instances

[–] [email protected] 2 points 3 months ago (1 children)
25G     pictrs
13G     postgres
38G     total

Seems fairly reasonable to me

[–] [email protected] 1 points 3 months ago
[–] [email protected] 9 points 1 year ago (1 children)

No expert here but usualy you use an external object storage (should be configurable in lemmy via pict-rs) which should be cheaper then VPS storage ^^

Maybe have a look into it.

[–] [email protected] 9 points 1 year ago

@Solvena my kbin instance is using 37 GB right now, and it is one of the oldest instances. I don't have many user thou.

[–] [email protected] 7 points 1 year ago

My Lemmy instance is using 3GB and it has 100 users.

[–] [email protected] 6 points 1 year ago* (last edited 1 year ago)

I can't help with Lemmy, but I've been running a single-user Mastodon instance for almost a year now.

Like you, I found that the media very quickly used up much more disk space than I anticipated. There are a few things you can do.

You can tune how long media is stored for: some of this is done in the admin interface, but really you need to set up cron jobs to regularly run various tootctl commands. This is the crontab I use:

SHELL=/bin/bash
PATH=/home/mastodon/.rbenv/shims:/home/mastodon/.rbenv/bin:/usr/local/bin:/usr/bin:/bin

RAILS_ENV=production
# Remove media attachments older than 8 days
11  19  *   *   *     cd /home/mastodon/live && time bin/tootctl media remove --days 8
# Remove link previews older than 28 days
22  5   *   *   *     cd /home/mastodon/live && time bin/tootctl preview_cards remove --days 28
# Remove files not linked to any post
 3  23  *   *   0     cd /home/mastodon/live && time bin/tootctl media remove-orphans
# Prune remote accounts that never interacted with a local user
44  1   *   *   *     cd /home/mastodon/live && time bin/tootctl accounts prune

You can of course choose even stricter settings but I found that no matter what I did, given that I am following approx 1,000 other Fediverse accounts it still used up more disk space than I was comfortable with.

So I offloaded most of the media storage onto an S3-compatible service. It's breaking the self-hosting ethos somewhat, but with Backblaze B2 I can happily store and serve several hundred GB of media files for just a couple of dollars a month. To me, that was a no-brainer.

[–] [email protected] 5 points 1 year ago

With Lemmy, it will depend more on the amount of media that your users upload than anything. With Mastodon, you also will have to consider the amount of data in the cache.

With Mastodon, I have (small) instances running for about an year or so which are using less than 100GB, and I have instances from power users that went on to take ~250GB in less than a month.

You can set up Mastodon to delete posts after a certain age and to clear the remote cache periodically. This would help mitigate both things.

[–] [email protected] 5 points 1 year ago

Will there be some sort of OS running on this VPS like Ubuntu, Debian etc.? The folder my Lemmy instance is stored in is 117 MB big. I have 1 user and follow 23 communities.

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago)

I have my lemmy instance currently using about 2 GB space, I'm going to set up Mastadon this weekend I hope. There are cheap smaller dedicated servers from kimsufi for like $10 with 1/2 TB HDD.

Also Hetzner cloud has compute and disk separated, so you can scale one or the other.

I've got a 500GBSSD box from Hetzner, but I'm also hosting other things there and dropped VMs from Linode I had previously and consolidated there.

You could use object storage, like s3 or wasabi, for lemmy picture/media storage, if that's eating up most space. "cloud" providers are bit bananas when you need more disk, at one point its cheaper to go down a dedicated route. Depends on your budget..

Edit: Others might chime in with a better answer, at the moment bot sure what your budget is, so its bit of assumptions from my side.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

My kbin instance's data (text data, that is) probably takes up less than 8 GB right now, and I've had it running about two weeks. Media storage (which I do through S3) is around 5 GB so far. Kbin does do media mirroring different than Lemmy though (I think), so YMMV. I think Lemmy mostly links to the original instance's media object as the source.

The main thing I found eating storage on my server was a lot of old Docker images (and volumes) from me trying to get everything up and running. If you are using Docker you could try doing a docker system prune --all to get rid of unused images/build caches (anything that isn't running currently).

[–] [email protected] 2 points 1 year ago (1 children)

@Solvena Hey, I consider hosting the mastodon intance for myself also. How much diskspace usage did you observed?

[–] Solvena 2 points 1 year ago

I had 10GB assigned for Lemmy and Mastodon (both single user instances) and the disk space filled up in a couple of days. I have implemented some of the tips given here now, so we will see.

[–] [email protected] 2 points 1 year ago

If you have storage at home, consider running pictrs and/or minio there and referencing it on the VPS through something like tailscale

[–] [email protected] 1 points 1 year ago

Following this ✌️

load more comments
view more: next ›