At approx 22.30 on 07/05/2013 a RAID controller in a SAN participating in the storage area network failed causing corruption on 2 LUNS. Both LUNS host our shared web hosting servers.

The corruption in the LUNS was replicated to all parts of the SAN in all data centres.

The Vendor was able to manually rebuild the LUNS to remove 98% of the corruption.

Please accept our apologies for any inconvenience caused.

Update 18:00 08/05/2013 - All servers are back online, however WWW08 is being taken offline due to OS corruption and scheduled for rebuild.

Update 19:12 08/05/2013 - WWW08 Being rebuilt

Update 1.30 08/05/2013 - WWW08 has finished rebuilding and account restores are under way.

Update 9.06 09/05/2013 - WWW19 was rebooted due to the additional load. All services now returning to normal on WWW19

Update 10:35 09/05/2013 - WWW08 accounts still restoring. WWW10 and WWW13 are still running FSCK file system checks.

Update 12:09 09/05/2013 - WWW08 account restores are still progressing, FSCK file system checks are still running on WWW13 and WWW10.

Update 16:39 09/05/2013 - WWW08 account restores in progress, approx 50% complete. fsck complete on WWW10. WWW13 still running a fsck check.

Update 22:12 09/05/2013 - WWW08 Engineers are still working to restore all accounts on WWW08. FSCK scans on WWW10 and WWW13 are complete. Please log a suppot ticket if you are still experiencing problems.

Update 9:48 10/05/2013 - WWW08 is still in the process of restoring accounts, There is damage to one of the HDD which is slowing down the copy process. Engineers are monitoring. Apologies for the further delays.

Update 15:38 10/05/2013 - WWW08 rsync process restarted after an additional fsck scan. Account restores in progress.

Update 11:48 11/05/2013 - WWW08 rsync process if still running, bad sectors are causing the copy process to be very slow. Engineers are currently discussing building new accounts on other servers. If you would prefer your account is created so you can restore from your own backup on a new server, please contact support.

Update 17:23 11/05/2013 - WWW08 due to the length of time the rsync process is taking, we have decided to provision all accounts from www08 onto www11. This is being completed as we speak and you will receive login details. Which will also be available in MyAccount. This will allow you to configure email addresses while we are waiting on the rsync process. 

Once the rsync process is completed we will restore each account as required. This will be much faster as the data will be coming from a new disk rather then a faulty disk.

12:08 12/05/2013 - All Accounts have been created on WWW11, please contact support if you have not received your new login details. We will handle account restores through the ticket system to ensure any data you upload now is not overwritten.

The SAN unit that caused the original issue has now been decommisioned and all servers moved over to our new Equallogic cluster.

Background:

Server Centre will be approving CU6 for Microsoft Lync 2010 Phone Edition for production deployment during the maintenance window. This will allow the firmware update to be deployed to all Polycom handsets connected to our Hosted Lync Platform.

Microsoft Lync 2010 Phone Edition cumulative update 6 includes a number of fixes as detailed in KB2701671.

CU6 will take the version number to 4.0.7577.4100

This cumulative update has been tested and approved for production deployment by our Change Management team.

PLEASE NOTE: You do not need to do anything, the phones will pick up the new firmware and install as required. Should you wish to speed up the process you can reset the handset and lock it. This will force the handset to check for and apply the new firmware update.

Services Affected:

Polycom CX500, CX600 and CX3000 handsets

Impact:

The phone will apply the update and reboot. The phone will only install the update when the handset has been idle for at least 10 minutes. But users may notice the handsets restart once the update is installed.

Maintenance Window

Friday 6th July 2012 18:00

All further updates relating to this maintenance will be added to this post.

Our network operations engineers will closely monitor the work and will do everything possible to minimise any inconvenience to you. If you have any problems with your services after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at 0845 413 3800 or visit http://servercentre.net/support

While it is important for Server Centre to perform maintenance such as this to ensure the quality of our services, we do everything possible to minimise any inconvenience to our customers. We appreciate your patience and welcome any feedback.

Best regards,

Server Centre NOC

On the 11th June 2012, LINX (one of our upstream providers) will be performing maintenance on their network.

During this window, we will shut down our peering sessions at LINX and re-route our internet traffic via alternate providers. Customers should not experience any loss of service during this maintenance window and we are re-routing traffic merely as a precaution to ensure uninterrupted connectivity for our customers.

If you require any further information, please contact Technical support.

Best regards

Server Centre NOC

At 16:10 on the 31 May 2012 our monitoring system detected problems with our transit via LINX. After an initial investigation it was discovered that our peering sessions on the LINX Juniper LAN were all dropping.

At 16:15 we shutdown our peering session at LINX and traffic was re-routed via our other providers.

LINX are currently investigating this issue and we will provide further updates here as we receive them.

Server Centre would like to apologize for any inconvenience the problems at LINX have caused.

UPDATE from LINX @ 19:05

"Analysing our traffic captures and the log output from our devices definitely indicate an ethernet loop in a network external to LINX which connects to both LANs."

Both LANs are now remaining stable.

"On the Juniper LAN, we are unsure why traffic was impacted, as our mac filtering and mac learning limit configuration should protect the LAN against loops. We have raised the issue with Juniper and continue to investigate the reason for this."

UPDATE @ 20:45

We have received the all clear from LINX ops and have now re-enabled our peering sessions at LINX. If you experience any further problems please contact support.

Once again, we would like to apologize for any inconvenience caused.

Best regards

Server Centre NOC

Background:

Server Centre will be approving CU5 for Microsoft Lync 2010 Phone Edition for production deployment during the maintenance window. This will allow the firmware update to be deployed to all Polycom handsets connected to our Hosted Lync Platform.

Microsoft Lync 2010 Phone Edition cumulative update 5 includes a number of fixes as detailed in KB2672349.

CU5 will take the version number to 4.0.7577.4066

This cumulative update has been tested and approved for production deployment by our Change Management team.

PLEASE NOTE: You do not need to do anything, the phones will pick up the new firmware and install as required. Should you wish to speed up the process you can reset the handset and lock it. This will force the handset to check for and apply the new firmware update.

Services Affected:

Polycom CX500, CX600 and CX3000 handsets

Impact:

The phone will apply the update and reboot. The phone will only install the update when the handset has been idle for at least 10 minutes. But users may notice the handsets restart once the update is installed.

Maintenance Window

Friday 18th May 2012 18:00

All further updates relating to this maintenance will be added to this post.

Our network operations engineers will closely monitor the work and will do everything possible to minimise any inconvenience to you. If you have any problems with your services after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at 0845 413 3800 or visit http://servercentre.net/support

While it is important for Server Centre to perform maintenance such as this to ensure the quality of our services, we do everything possible to minimise any inconvenience to our customers. We appreciate your patience and welcome any feedback.

Best regards,

Server Centre NOC

Derby/DC1&2 Service Affecting Planned Maintenance Friday 4th May

Background:

In order to increase network port capacity in Derby we plan to replace one of our two core Juniper MPLS/Internet routers on the morning of Friday 4th May. In particular, the link between our two data centres in Debry & Wakefield will be affected during this maintenence Window.

Services & Locations Affected

MPLS services for customers * directly connected to the Core router in DC-2 (Derby)

Impact:

Customers will experience loss of service of up to 30 minutes (estimated) while engineers move services to the new router. However services will be "at risk" for the full maintenance window.

Services & Locations Affected

Internet services for customers * directly connected to the Core router in DC-2 (Derby)

Impact:

Customers will experience loss of service of up to 30 minutes (estimated) while engineers move services to the new router. However services will be "at risk" for the full maintenance window.

Services & Locations Affected

Internet & MPLS services for customers in DC-1 & DC-2 (Derby) and SIPLink.

Impact:

Customers may experience some minor disruption during routing convergence. However services will be "at risk" for the full maintenance window.

Services & Locations Affected

Internet & MPLS services for all customers at all locations

Impact:

Services will be "at risk" for the full maintenance window as some network resiliency will be lost during the period where the new router is installed.

*The customers directly affected by this maintenance will be emailed by 30th March

Maintenance Window

Friday 20th April 01:00 – 04:00

Preparation works will start 00:00, with service affecting work beginning at 01.00. We expect the work to take no longer than 30 minutes (and hopefully less) however services will be "at risk" for the full maintenance window.

Our network operations engineers will closely monitor the work and will do everything possible to minimise any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at 0845 413 3800

While it is important for Server Centre to perform maintenance such as this to ensure the quality of our network, we do everything possible to minimise any inconvenience to our customers. We appreciate your patience and welcome any feedback.

Best regards

Server Centre NOC

We have received reports from a number of customers with services with Virgin Media that they are having intermittent connectivity issues to our network. We believe that this is an issue within Virgin's network and we recommend that you raise a fault with them.

We suggest that you use WINMTR to gather end to end hop statistics which will help identify where the network issues are occurring.

Best regards

Server Centre NOC

Background:

Server Centre will be applying SP2 across all Exchange Servers in the work window detailed below. This update has been tested and will not impact any production services, a small portion of clients may get Outlook disconnect and then re-connect within 60 seconds during the work window as live connections are moved to another cluster member.

Services & Locations Affected:

Hosted Exchange 2010 Mailbox's, BES/BIS BlackBerry Email, Outlook Web Access and Lync integration with Exchange.

Impact:

This work will not be service affecting however all the above services will be deemed 'At Risk' while the work is completed.

Maintenance Window

Saturday 14h Jan 2012 01.00 – 05.00

All further updates relating to this maintenance will be added to this post.

Our network operations engineers will closely monitor the work and will do everything possible to minimise any inconvenience to you. If you have any problems with your services after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at 0845 4133800 or visit http://servercentre.net/support

While it is important for Server Centre to perform maintenance such as this to ensure the quality of our services, we do everything possible to minimise any inconvenience to our customers. We appreciate your patience and welcome any feedback.

Best regards,

Server Centre NOC