this post was submitted on 26 Mar 2024
638 points (96.4% liked)

linuxmemes

21624 readers
416 users here now

Hint: :q!

Sister communities:

Community rules (click to expand)

1. Follow the site-wide rules

2. Be civil
  • Understand the difference between a joke and an insult.
  • Do not harrass or attack members of the community for any reason.
  • Leave remarks of "peasantry" to the PCMR community. If you dislike an OS/service/application, attack the thing you dislike, not the individuals who use it. Some people may not have a choice.
  • Bigotry will not be tolerated.
  • These rules are somewhat loosened when the subject is a public figure. Still, do not attack their person or incite harrassment.
    • 3. Post Linux-related content
  • Including Unix and BSD.
  • Non-Linux content is acceptable as long as it makes a reference to Linux. For example, the poorly made mockery of sudo in Windows.
  • No porn. Even if you watch it on a Linux machine.
    • 4. No recent reposts
  • Everybody uses Arch btw, can't quit Vim, and wants to interject for a moment. You can stop now.
    •  

    Please report posts and comments that break these rules!

    Important: never execute code or follow advice that you don't understand or can't verify, especially here. The word of the day is credibility. This is a meme community -- even the most helpful comments might just be shitposts that can damage your system. Be aware, be smart, don't fork-bomb your computer.

    founded 2 years ago
    MODERATORS
    poopsmith
    zephyr
    rtxn
    638
    What is the most difficult problem that you have fixed in linux? (lemmy.world)
    submitted 9 months ago by Waffelson to c/linuxmemes
    164 comments fedilink hide all child comments
     
    you are viewing a single comment's thread
    view the rest of the comments
    [–] [email protected] 9 points 9 months ago* (last edited 9 months ago) (1 children)

    More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run "rm -rf /* something*" in a root shell.

    Note the errant space that made that a fatal mistake. I don't remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.

    He had 1 active ssh connection, and couldn't start another one.

    It was a server that was "in production", was thousands of miles away from him, and which had no possibility for IPMI / remote hands.

    Everyone (but me) in the channel said that he was just SoL and should just give up.

    I stayed up most of the night helping him. I like challenges and I like helping people.

    This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.

    We recovered the bash binary by running something along the lines of

    bash_binary_contents="$( </proc/self/exe)"
printf "%s" > /tmp/bash

    (If you can access "lsof" then "sudo lsof | grep deleted" will show you any files that are open, but also "deleted". You may be surprised at how many there are!)

    But bash needed too many shared libraries to make that practical.

    Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.

    Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.

    We were able to restart openssh-server, log in again, and I don't remember if we were brave enough to test rebooting.

    But we fucking did it!

    I am certainly getting a lot of details wrong from memory. It's all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.

    I tried to find it once, and failed.

    [–] bitchkat 5 points 9 months ago

    I just told this story to a friend but I did the standard rm -rf * as root while in the / directory. And this was back in the day where we nfs mounted every other machine and root privileges propagated through NFS. I think it was on the 2nd or 3rd machine when I thought -- "this seems to be taking longer than I thought".