We are currently investigating an issue with the power at our Wakefield Datacentre (DC3), which may be affecting some customers.  We believe that all services have now been restored.  Further updated will be posted shortly.

Update 15:14
During monthly routine power load testing, a sequence of failures to the generator back-up system resulted in a 30 second power outage to the site.  This test has been carried out successfully every month since the site opened to ensure power back-up systems perform correctly in the event of a power failure.

The root cause of the issue is being investigated and generator and UPS engineers are currently on-site to carry out a full investigation.  Once this investigation is complete a full report will be made available.  This is the first such occurrence Node4 have experienced since the business started.  We take this issue very seriously and apologise for any inconvenience.  Node4 technical staff are also available at site if any technical assistance is needed.

Update 15:53
Engineers in DC3 have been checking that all equipment is live and back in service.  If you are still experiencing issues with your equipment, please make sure you log a ticket with the helpdesk and an engineer will be available to assist.

We have an outstanding issue with one of our voice interconnects.  During the power issue equipment belonging to one of our voice interconnect providers failed.  We have an engineer on site investigating this and hope to restore full service shortly.  This is impacting a small percentage of customers who are experiencing issues with inbound calls, and intermittent issues with outbound calls.

Update: 17:00
Generator engineers have completed diagnostics tests on both gensets and not found any issues. We are awaiting feedback from the UPS engineers. Power and cooling systems are operating and we have additional engineers onsite to deal with remote hands requests from customers.

Update: 18:00
Normal service for voice customers was restored at 17:40. Further information will be included in the Outage Report that will be made available once the investigations have been completed.  Apologies again for any inconvenience caused

Update 19:50
Panel board and generator engineers will be onsite at approximately 8:30pm to begin further diagnostic works. These works are classed as non-intrusive and low risk, however, given the nature of the problem, we will issue a general “At Risk” notification when this work is undertaken. This website will be updated before we begin this process.

Update 21:35
We are in the process of completing panel board tests and off-load generator tests. On successful completion of these, we plan to perform a full on-load test at 10pm. We will update this website after that test has been completed

Update 22:30
We have completed the first onload test and are continuing with our diagnostics.

Update 01:10
We have identified an issue with a relay on the mains / generator changeover control and replaced this part. Subsequent onload tests and diagnostics have been successful and we are confident that the board is operating back under normal condition.

Engineers will be back on site tomorrow morning to complete further diagnostic tests on the generators but we are satisfied that any further risk has been mitigated. We will continue to update this site with any further information and, of course we will issue a full incident report as soon as we can. Customer’s affected by today’s incident should contactsupport@servercentre.net or any ongoing technical problems or your account manager for any other queries.

Server Centre would like to apologise again for the disruption to service that this has caused.

Best regards

Server Centre Technical Support

Background:

Engineers plan to perform maintenance on our “MPLS Access” Layer-2 Edge Ethernet Switches at DC-2 Derby and DC-3 Wakefield Data Centres. The purpose of this planned maintenance is to perform a software upgrade on the switches in order to enable new features.

This maintenance will be performed in two phases.

Impact:

Layer-2 MPLS Ethernet Interconnects, between our DC’s or DC to a POP, terminating on a “MPLS Access” Switch at Derby and/or Wakefield Data Centres will be impacted during this maintenance work. We expect two service outages, with each being up to 5 minutes.  However these services will be deemed “at risk” until the maintenance work is completed. Colocation customers directly affected by this work have been notified via email (07/01/11).

Maintenance Window #1

Thursday 20th January 06.00 – 06.30

Maintenance Window #2

Friday 28th January 06.00 – 06.30

Our network operations engineers closely monitor the work and will do everything possible to minimise any inconvenience to you.  If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at 0845 413 3800 or visit http://servercentre.net/support

While it is important for Server Centre to perform maintenance such as this to ensure the quality of our network, we do everything possible to minimize any inconvenience to our customers.  We appreciate your patience and welcome any feedback.

Regards,

Server Centre Technical Support