News
Oct
21
Unplanned Outage – DC3 Wakefield
Posted by adam.heavens on 21 October 2011 02:24 PM

We are currently investigating an issue with the power at our Wakefield Datacentre (DC3), which may be affecting some customers.  We believe that all services have now been restored.  Further updated will be posted shortly.

Update 15:14
During monthly routine power load testing, a sequence of failures to the generator back-up system resulted in a 30 second power outage to the site.  This test has been carried out successfully every month since the site opened to ensure power back-up systems perform correctly in the event of a power failure.

The root cause of the issue is being investigated and generator and UPS engineers are currently on-site to carry out a full investigation.  Once this investigation is complete a full report will be made available.  This is the first such occurrence Node4 have experienced since the business started.  We take this issue very seriously and apologise for any inconvenience.  Node4 technical staff are also available at site if any technical assistance is needed.

Update 15:53
Engineers in DC3 have been checking that all equipment is live and back in service.  If you are still experiencing issues with your equipment, please make sure you log a ticket with the helpdesk and an engineer will be available to assist.

We have an outstanding issue with one of our voice interconnects.  During the power issue equipment belonging to one of our voice interconnect providers failed.  We have an engineer on site investigating this and hope to restore full service shortly.  This is impacting a small percentage of customers who are experiencing issues with inbound calls, and intermittent issues with outbound calls.

Update: 17:00
Generator engineers have completed diagnostics tests on both gensets and not found any issues. We are awaiting feedback from the UPS engineers. Power and cooling systems are operating and we have additional engineers onsite to deal with remote hands requests from customers.

Update: 18:00
Normal service for voice customers was restored at 17:40. Further information will be included in the Outage Report that will be made available once the investigations have been completed.  Apologies again for any inconvenience caused

Update 19:50
Panel board and generator engineers will be onsite at approximately 8:30pm to begin further diagnostic works. These works are classed as non-intrusive and low risk, however, given the nature of the problem, we will issue a general “At Risk” notification when this work is undertaken. This website will be updated before we begin this process.

Update 21:35
We are in the process of completing panel board tests and off-load generator tests. On successful completion of these, we plan to perform a full on-load test at 10pm. We will update this website after that test has been completed

Update 22:30
We have completed the first onload test and are continuing with our diagnostics.

Update 01:10
We have identified an issue with a relay on the mains / generator changeover control and replaced this part. Subsequent onload tests and diagnostics have been successful and we are confident that the board is operating back under normal condition.

Engineers will be back on site tomorrow morning to complete further diagnostic tests on the generators but we are satisfied that any further risk has been mitigated. We will continue to update this site with any further information and, of course we will issue a full incident report as soon as we can. Customer’s affected by today’s incident should contactsupport@servercentre.net or any ongoing technical problems or your account manager for any other queries.

Server Centre would like to apologise again for the disruption to service that this has caused.

Best regards

Server Centre Technical Support


Read more »