One of our servers is a rotting carcass being kept alive by our collective prayers. It runs Windows 7 and custom software whose developer is dead and the source is missing, nothing has been updated for over a decade, and it has its own independent UPS because once it goes down, it has an extremely slim chance of recovering, and we’re afraid to test it. It controls the card entry system into the building, including the server room. Boss doesn’t want to replace it because we’d have to replace all of the terminals and controllers too, and it hasn’t catastrophically failed yet.
You’re right. It’s not a pet. It’s like one of the Saw movies: if it dies, we’re all fucked.
The question I often ask clients who think this way is "How much would it cost if it did fail? Let’s say this happened today. What would be the cost to replace it NOW and not only that but make sure people who are working can still do so with the interruption?
Now how much would it cost to schedule the interruption and manage the fall out in a way that is controllable?
For some, the catastrophic failure points to “hey I fixed the thing!” And the incentives for that kind of person are different from the person whose job is to mitigate risk.
It sounds like your boss is the former. In which case it’s going to be fun when it fails.
I learned this relatively quickly running my own server with the intention of my family also using it. Data on a separate drive, backed up regularly and automatically. System on it’s own drive, dd’d when it’s in it’s final state and backed up before I screw around any deeper than trying out a new container. I can bring my server back up in however long it takes to transfer data.
New guy at work ran this to try to fix permissions on his home folder, accidentally ran it on root (both would have been bad)
Several highly paid and experienced Linux admins finally just gave up and deleted the server and built a new one from the backups.
Why does he have rights to use sudo in the first place?
Which, honestly, is the better way to go. Treat your compute resources like cattle, not pets.
One of our servers is a rotting carcass being kept alive by our collective prayers. It runs Windows 7 and custom software whose developer is dead and the source is missing, nothing has been updated for over a decade, and it has its own independent UPS because once it goes down, it has an extremely slim chance of recovering, and we’re afraid to test it. It controls the card entry system into the building, including the server room. Boss doesn’t want to replace it because we’d have to replace all of the terminals and controllers too, and it hasn’t catastrophically failed yet.
You’re right. It’s not a pet. It’s like one of the Saw movies: if it dies, we’re all fucked.
The question I often ask clients who think this way is "How much would it cost if it did fail? Let’s say this happened today. What would be the cost to replace it NOW and not only that but make sure people who are working can still do so with the interruption?
Now how much would it cost to schedule the interruption and manage the fall out in a way that is controllable?
For some, the catastrophic failure points to “hey I fixed the thing!” And the incentives for that kind of person are different from the person whose job is to mitigate risk.
It sounds like your boss is the former. In which case it’s going to be fun when it fails.
So… the dead server controls who is even able to enter the building? Wow. That is one big juggernaut of a problem heading for you.
I learned this relatively quickly running my own server with the intention of my family also using it. Data on a separate drive, backed up regularly and automatically. System on it’s own drive, dd’d when it’s in it’s final state and backed up before I screw around any deeper than trying out a new container. I can bring my server back up in however long it takes to transfer data.