This morning I woke up to my phone using mobile data and my home assistant automations not working. Initially I thought it the power was out, but I could turn on the lights just fine. I checked my UniFi app and saw that the server was not connected to the network at all. This meant that the cable got unplugged, the switch isn’t working, or the server isn’t working. It said the switch was connected and another device was connected to the switch so that narrows it down to just 2 cases. So I opened my server closet in the basement and immediately noticed something was wrong. I couldn’t figure out what was wrong but I just felt like something was wrong. Everything was plugged in, the network switch lights were blinking like normal, my raspberry pi was running just fine, even the server indicator lights were on. My main server is an old gaming PC so it has a glass side panel so I looked inside and I could see the fan spinning, but I could not hear it. Usually I have it set to full speed and I can hear full speed very well. I tried rebooting the server with the power button and the fans didn’t go to full speed. As a last resort, I brought down a keyboard and monitor. As soon as I plugged in the monitor, I saw that there was a prompt to set the time on the BIOS! Picture of the prompt In my opinion, this was the stupidest reason for an outage.
Further investigations
I dug a little deeper and discovered that the BIOS had been reset during a power outage right before all of this happened. So far I have consulted the motherboard manual and found absolutely nothing about this. After a bit of research, I think it could have been that the CMOS battery has died. This is a really simple fix but I don’t have the replacement battery right now. This means that I will have the same exact issue after the next power outage unless I replace the battery.
Preventing this in the future
From what I can see, I just need to replace the CMOS battery. But this computer has been running for over 4 years, so what is stopping this from happening again around 2028? The most effective solution is going to be preventing power outages in the first place. This can be done using a battery backup or a standby generator. Standby generators will last longer during a power outage but are typically more expensive and harder to setup than a simple battery backup.
You must log in or register to comment.
The only real solution is to make this an extended maintenance task. The batteries are cheap so an alert every 4 years is likely sufficient to replace the battery before it dies. You could do it every 2 or 3 years instead at your discretion.
The only real solution is to make this an extended maintenance task.
This is the correct answer. No matter how reliable your power feed is you still need to reboot the server at some point for whatever reason and if CMOS battery is dead by then you’ll have the very same issue and you’ll need monitor and keyboard again. And even if you don’t mind about the RTC on board you’ll still lose the settings.
I wonder why manufacturers haven’t switched over to supercapacitors or something else than a coin cell battery, but perhaps there’s a valid reason for it. I think that supercaps can’t hold charge as long as a coin cell, but if your board is completely cold for a year or so maybe losing bios settings isn’t that big of a deal.
Well they still have a finite life and are less replaceable than a battery. Even if it quadrupled the lifespan (which is a reasonably generous estimate given OP’s 4-year duration and wikipedia telling me supercapacitors last 10-15 years), it would still eventually need to be replaced and that would generally require resoldering it.
I think a much better solution is 2 battery slots, one to be a backup battery, unused, and then when needed, an LED on the mobo can be turned on. Honestly OP could jury-rig up a similar system if he wanted to, although it’d be a bit ugly and anytime something is jury-rigged I don’t really think of it as reliable.
You could of course use some kind of socket or connetor for supercap, but as they last far less than I thought then I get why it doesn’t make sense. This thinkstation I’m writing with in my garage I got for free at old office is from 2011 and it’s still running original cmos battery. No idea if there’s any juice left on it, but at least it doesn’t complain anything at boot and once it refuses to boot it’ll become e-waste immediately (I do metal working, fix cars etc at the garage, so internals of this thing are far from clean, I think this is 3rd or 4th hardware for 10 years in here with only the SSD moved from setup to another).
Chec out this article
Man, some people have really thought of everything. I am so impressed.
grep bat /proc/driver/rtc❤️Always learning something new, thanks :)
What about all those fancy new laptops that are built like smartphone that can never be opened do they have CMOS batteries? Ive never thought if this till now.
If the main battery isn’t “meant to be replaced”, it will often act as the CMOS battery (e.g. MacBooks have been doing this since roughly 2008).
Huh cool
From what I can see, I just need to replace the CMOS battery. But this computer has been running for over 4 years, so what is stopping this from happening again around 2028?
Replace the CMOS battery again.
Why you are at it you might want to check the battery in your smoke alarm. I think your are support to change the smoke alarm battery every 5 years.
The battery in smoke detectors should be changed every year and you should ideally test the detectors using the little button monthly.
In Norway we have an official smoke detector day which is on December 1. where we’re supposed to change the batteries. Its because Christmas is the high season for house fires.
That depends on the type of battery. I have some of the old school ones, which are “test every 6 months, replace every year,” but also a fancy schmancy lithium ion-powered one that should be good for 10 years (so test every 6 months, but replace after 10 years).
Yea the lithium-ion batteries are pretty good, you can also get 10 year smoke detectors with a built in lithium battery. Also a psa, smoke detectors usually only last 10 years and need to be replaced after that. There’s often an expiry date on the detectors.
Just took over a house last year with smart detectors connected to a homebase, all Li-Ion. Can see the battery percentage in the app. No more December 1st checks for a long time!
It is 5 years in the US as it says it on the box. The battery can last 10 years but that is way to long from a safety perspective.
Lithium batteries last 5-10 years, regular 9v alkaline batteries last about 12-18 months. Some sources recommend changing the batteries every 6th month but i personally think that’s a bit excessive.
I only have a small 2 room apartment so i just got myself a cheap-ish optical smoke detector with a built in lithium battery. The battery should last 10 years which is how long smoke detectors last before they should be replaced.
Some BIOS manufacturers allow you to disable all halts on errors. As soon as you connect to the network your system should be able to sync its internal clock.
Here’s the thing, you now know you’ve to replace the battery every 3 years or so, but I’ve SONY batteries running for 5 years and they’re fine. Now, your motherboard’s manufacturer is a piece of shit, the board should’ve just ignored the issue and proceeded to the OS. All modern operating systems sync the UEFI time whenever they get time from NTP so no need to halt the boot process.
I just do an
uptime -pif I suspect there was a power outtageCan you replace the CMOS battery with a supercapacitor that is kept charged? This should not need to be replaced every 4 years, I think.
We use these in our products at work and I’ve never heard about them failing before the equipment is replaced anyways. They are soldered.
I won’t quote life expectancy (MTBF) but we have many customers with 15-20 year old PLC’s with the original capacitor.
I wonder… How about a hot swap cmos battery? Thinking you wire the cmos terminals to a few batteries in parallel Or 3 And add a voltage indicator for each of them
deleted by creator
I once worked on a product that you really did not want to have not coming back up. I was on it several years after the original engineers had designed an early model. Said engineers had not tested what happened when the CMOS battery died and triggered a reset of BIOS settings, brought it back to the hardware platform’s default state. When it did, the thing entered a non-bootable state. You could, with a serial port, access the BIOS and fiddle the settings back for one good boot…but the CMOS battery was non-removable, soldered to the motherboard. Our manufacturing process had not involved changing the default BIOS settings, just what was stored in CMOS. Oops.
IIRC our customer care guys just sent out new models for free to affected customers – the original hardware model wasn’t sold in large volume, and the cost of the actual hardware components wasn’t especially large relative to the cost of the product.
I had one sitting around on my desk, as it was sometimes handy to have a physically-accessible device to do work on. I rolled down to Radio Shack – yes, this was a few years back – got a removable CMOS battery case, stripped the non-removable battery out, soldered the battery case to the motherboard, and had the only instance of the device out there that could take a fresh CMOS battery.
