I woke up this morning to a text from my ISP, “There is an outage in your area, we are working to resolve the issue”

I laugh, this is what I live for! Almost all of my services are self hosted, I’m barely going to notice the difference!

Wrong.

When the internet went out, the power also went out for a few seconds. Four small computers host all of my services. Of those, one shutdown, and three rebooted. Of the three that ugly rebooted some services came back online, some didn’t.

30 minutes later, ISP sends out the text that service is back online.

2 hours later I’m still finding down services on my network.

Moral of the story: A UPS has moved to the top of the shopping list! Any suggestions??

  • CameronDev@programming.devEnglish
    26·
    8 months ago

    Did the services fail to come back due to the bad reboot, or would they have failed to come back on a clean reboot? I ugly reboot my stuff all the time, and unless the hardware fails, i can be pretty sure its all going to come back. Getting your stuff to survive reboot is probably a better spend of effort.

    • Padook@feddit.nlOPEnglish
      12·
      8 months ago

      I didn’t mean to imply that Services actually broke. Only that they didn’t come back after a reboot. A clean reboot may have caused some of the same issues because, I’m learning as I go. Some services are restarted by systemctl, some by cron, some…manual. This is certainly a wake up call that I need standardize and simplify the way the services are started.

      • CameronDev@programming.devEnglish
        11·
        8 months ago

        We’ve all.committed that sin before. Its better to rely on it surviving the reboot than to try prevent the reboot.

        Also worth looking into some form of uptime monitoring software. When something goes down, you want to know about it asap.

        And documenting your setup never hurts :D

    • fuckwit_mcbumcrumble@lemmy.worldEnglish
      11·
      8 months ago

      Yeah an unclean reboot shouldn’t break anything as long as it wasn’t doing anything when it went down. I’ve never had any issues when I have to crash a computer unless it was stuck doing an update.