At approx 22.30 on 07/05/2013 a RAID controller in a SAN participating in the storage area network failed causing corruption on 2 LUNS. Both LUNS host our shared web hosting servers.
The corruption in the LUNS was replicated to all parts of the SAN in all data centres.
The Vendor was able to manually rebuild the LUNS to remove 98% of the corruption.
Please accept our apologies for any inconvenience caused.
Update 18:00 08/05/2013 - All servers are back online, however WWW08 is being taken offline due to OS corruption and scheduled for rebuild.
Update 19:12 08/05/2013 - WWW08 Being rebuilt
Update 1.30 08/05/2013 - WWW08 has finished rebuilding and account restores are under way.
Update 9.06 09/05/2013 - WWW19 was rebooted due to the additional load. All services now returning to normal on WWW19
Update 10:35 09/05/2013 - WWW08 accounts still restoring. WWW10 and WWW13 are still running FSCK file system checks.
Update 12:09 09/05/2013 - WWW08 account restores are still progressing, FSCK file system checks are still running on WWW13 and WWW10.
Update 16:39 09/05/2013 - WWW08 account restores in progress, approx 50% complete. fsck complete on WWW10. WWW13 still running a fsck check.
Update 22:12 09/05/2013 - WWW08 Engineers are still working to restore all accounts on WWW08. FSCK scans on WWW10 and WWW13 are complete. Please log a suppot ticket if you are still experiencing problems.
Update 9:48 10/05/2013 - WWW08 is still in the process of restoring accounts, There is damage to one of the HDD which is slowing down the copy process. Engineers are monitoring. Apologies for the further delays.
Update 15:38 10/05/2013 - WWW08 rsync process restarted after an additional fsck scan. Account restores in progress.
Update 11:48 11/05/2013 - WWW08 rsync process if still running, bad sectors are causing the copy process to be very slow. Engineers are currently discussing building new accounts on other servers. If you would prefer your account is created so you can restore from your own backup on a new server, please contact support.
Update 17:23 11/05/2013 - WWW08 due to the length of time the rsync process is taking, we have decided to provision all accounts from www08 onto www11. This is being completed as we speak and you will receive login details. Which will also be available in MyAccount. This will allow you to configure email addresses while we are waiting on the rsync process.
Once the rsync process is completed we will restore each account as required. This will be much faster as the data will be coming from a new disk rather then a faulty disk.
12:08 12/05/2013 - All Accounts have been created on WWW11, please contact support if you have not received your new login details. We will handle account restores through the ticket system to ensure any data you upload now is not overwritten.
The SAN unit that caused the original issue has now been decommisioned and all servers moved over to our new Equallogic cluster.