Benefits and Hindrances of Regular Server Reboots

Timur Fanshteyn picture Timur Fanshteyn · Jan 4, 2009 · Viewed 9.3k times · Source

In the ears of working in multiple teams, I've met multiple infrastructure managers that instituted a policy of weekly server reboots. As a developer, I was always against the policy - it seems that this is a hack to work around software bugs and hardware instabilities, instead of correcting them.

What are the people's opinions, positive and negative points regarding the policy?

Answer

derobert picture derobert · Jan 4, 2009

If you reboot your servers occasionally, you can be sure they will come back up. Though weekly sounds like a serious overkill, I have seen this problem on Linux machines with long uptimes.

Someone didn't bother to set up a critical service to start automatically on boot. Or the order of services coming up is wrong. Or someone upgraded libraries, added/removed software, etc. and the executable no longer works (it was started up with the old libraries, and continued using them; now it gets a dynamic linker error). Or it turns out service A depends on service B and service B depends on service A (oops).

At some point, when you least want to, you will take a reboot. The colo will drop the power on you; the server's power supplies will fail; someone will pull the cord/hit the reset button on the wrong server; etc. Now, when you can least afford downtime, your bloody server won't come back up.

Just like software, system configurations need testing. How often you need to do this testing depends on how your boxes are administered.