A Friday 13th story
Are you superstitious? Do you believe in Murphy’s law? I usually don’t believe in such things, but sometimes you start to have doubts…
This site is served by my own virtual server hosted by Carrot Servers in Austria. The hardware nodes running their servers are housed in one of Germany’s largest hosting company’s datacenters. However, three of their datacenters recently suffered from a power loss because of a defunct uninterruptible power source and a series of other unfortunate problems – a little over 12,000 servers were without power for a few hours.
However, it seems my vServer’s hardware node was damaged by the power loss; one of the hard disk drives in the RAID array was replaced after the outage. Yesterday, the worst-case scenario came true: besides the hard disks, apparently the motherboard was also damaged causing the RAID controller to slowly corrupt the data on the drives. On Friday, 13th, my provider informed me data had been lost and the server will be swapped with a new machine.
Luckily, I did remember creating backup – and it happened to be a recent one (namely from Thursday, 12th). But as Murphy’s law states, things never work the way you expect them to work. In this case that meant that permissions were incorrect on the backup although I used rsync with the appropriate preserve-permissions option. I’ve thus been busy reinstalling the software, uploading the files and databases and fixing their permission. As of now, all data has been successfully restored (blessing in disguise) and I’m supervising the server’s functions.