Our hardware failed this morning and we're stuck waiting on OVH to investigate the support ticket I filed. They claim their support is 24/7 but live chat / phone is just business hours, so worst case we might be down until tomorrow =(

https://status.lemmy.ca/ has an update, but not much to say really except "It's down until it's not"

Edit: I'm assuming we're down until I can call / chat with them first thing in the morning, so expect to be back up early tomorrow (depending on how long the parts replacement takes).

Edit 2: I called them when I woke up at 6:30 PST and hopefully they'll fix it soon now....

[–] Brewchin 15 points 2 days ago (1 children)

They claim their support is 24/7 but live chat / phone is just business hours

It's increasingly common for "24x7 support" to mean access to their KB/FAQ and the ability to raise a ticket, while the actual support is a skeleton crew working regular hours.

Not unlike "your call is very important to us..." 😅

[–] AnUnusualRelic 8 points 2 days ago

You can send us email at any time of day or night!

[–] [email protected] 31 points 2 days ago

Just changed all my maintenance messages to

It's down until it's not

[–] [email protected] 7 points 2 days ago

Thanks! I'll bookmark that.

[–] ininewcrow 16 points 2 days ago (2 children)

Weird to think what happens when you lose a site you visit on a regular basis. I can understand the whole instance going down but us users just get lost when things get turned off. There's no way to know what happened, why or when or if anything will return.

I had to resurrect my old lemmy.world account to figure out what happened. I guess we have to hang on to two or more unique instance accounts in case our main instance goes down.

Stay well guys and I totally appreciate the admins of lemmy.ca and all the work they do. I'll patiently wait and look forward to seeing it online again.

[–] [email protected] 7 points 2 days ago

At least it’s not like Reddit when the website goes down you can’t use it at all.

[–] [email protected] 9 points 2 days ago (2 children)

I keep trying to open the lemmy app on my phone out of habit.

[–] [email protected] 3 points 2 days ago

I did that five times before remembering I had an old account on this instance that I could switch to to post the question and see what communities I used to be subscribed to.

[–] ininewcrow 6 points 2 days ago

It feels shitty and I was doing the same thing all day. I had to resurrect my lemmy.world account just to get back online. It's made me realize that I should maintain both accounts for shut downs like this. And now I'm also having to do more in maintaining my Mastodon account too.

[–] [email protected] 16 points 2 days ago (1 children)

They posted on the admin chat that their hosting provider had a hardware failure and is working on providing a replacement server.

[–] Seigest 10 points 2 days ago

It may not get fixed untill tommrow since they are not getting any response from their service provider.

[–] [email protected] 15 points 2 days ago* (last edited 2 days ago) (1 children)

That's an error message from Cloudflare, their tunnel provider. The server is probably unreachable. Either they're doing some maintenance, or it's a hiccup or something went wrong. You need to wait for a moment. Or maybe a bit longer if they're facing any issues.

[–] [email protected] 3 points 2 days ago* (last edited 2 days ago) (1 children)

Wow, they're really offline for a long time now. Did OVH really mess up that hard? I guess hardware in a datacenter fails every day? Wouldn't they be prepared for that and just restore/migrate to a different machine, then?

[–] [email protected] 2 points 2 days ago (1 children)

Yeah that's usually how it's supposed to work, but sometimes it doesn't work properly for a variety of reasons.

[–] [email protected] 3 points 2 days ago* (last edited 2 days ago) (1 children)

Yeah, it's speculation at this point anyways. I mean I've only ever seen webservers with a RAID. So one harddrive failure shouldn't affect it too much. Then most people rent a VPS and that usually runs on some cluster. Even if a whole machine goes down, it's supposed to come up again on some other machine automatically within a few minutes. And then you should have tech support of a big hosting provider answer within some timespan. I hope someone will write a summary of the events and link it somewhere. Maybe a lot of things went wrong. Or it's some kind of error in the specific setup lemmy.ca did.

[–] [email protected] 3 points 2 days ago

It wasn't a hard drive failure but something more severe, since the remote console is non-responsive. I just called OVH to yell at them and press on the ticket, so hopefully should be resolved soon.

[–] [email protected] 6 points 2 days ago (2 children)

You can host a lemmy instance through a cloudflare tunnel??? :0

[–] [email protected] 12 points 2 days ago

Yes of course, as almost any other website.

[–] [email protected] 2 points 2 days ago

This very message is coming to you through a tunnel!