Meta

Hey! First, let me thank you for doing what you do. Maintaining a public service on your own even for a small group of people isn't easy and brings all kinds of responsibilities and stress.

However, this is a good opportunity to learn. Going from my over 20 years experience the best way to deal with such unfortunate events is going through a blameless postmortem, see https://sre.google/sre-book/postmortem-culture/

In a nutshell, it's important to be confident about the fact that something like that will happen again, find out what happened, why it's happened, and what you can do to prevent this happening in the future. The later could be some additional check, a peer review, or a process decision, e.g. "don't apply changes late in the night".

Thank you again and keep up the good work!

Hey, no worries. I accidentally deleted someone's VM while they were using it. I didn't even notice until a ticket poped up in another team's queue. It happens to all of us.

Throw a little DNS in the mix any every problem can become a jack-in-the-box ready to ruin your day any second. I'm guessing a lot of the members of this server have been there.

Thank you for being communicative!