I'd like to give you a bit more details on the recent downtime. This is what happened in the last few days.
The server died last Wednesday after a reboot following a scheduled maintenance (patched with windows server
security updates). We started the troubleshooting but came to a conclusion we need to reinstall everything. We were simply unable to repair the OS. We have backups, everything is safe but the downtime was unfortunately inevitable.
For several days we were struggling with Windows 2008 R2 reinstall and security updates. The server got infected through SMB several times during the patching. We had to patch the server offline not to be vulnerable to attacks anymore during the update process but working through an IP console to reach the server in the datacenter did not make it any easier. If anyone have some experience with legacy operating systems you know how long it takes to patch a fresh install.
We managed to secure the server and finally update it again with he latest MS updates but then again the server died! Awesome, now we know that the issue was a MS patch since we basically reproduced the initial issue.
So what now, we could have kept on trying to patch the server, find the problematic MS update, fight the SMB attack, reinstall everything but it would have been a lot of extra time, effort and not to mention Windows Server 2008 product cycle will end soon too...
So at the end we decided to move on and buy a new Windows server (YAY for Windows Server 2019 Essentials!) which so far was a great decision! We had to test compatibility and set everything up but now we are at the stage when everything looks great and only some final touches left to be 100%.
Phwe, what a week.. During the last 10 years we never had a downtime like this and I hope there won't be any for the next 10 years either!
Special thanks to GM Peterigz, without him the recovery would have been a lot longer. He's a real wizard.
Thanks for reading and for all the patience! Happy hunting! =)
