RSS Feed
News
Jan
17
Connectivity Issues for Virgin Media Customers
Posted by Adam on 17 January 2012 04:54 PM

We have received reports from a number of customers with services with Virgin Media that they are having intermittent connectivity issues to our network. We believe that this is an issue within Virgin’s network and we recommend that you raise a fault with them.

We suggest that you use WINMTR to gather end to end hop statistics which will help identify where the network issues are occurring.

Best regards

Technical Support


Read more »



Oct
21
Voice Services
Posted by adam.heavens on 21 October 2011 02:26 PM

We are currently experiencing an outage on our Voice platform.  We are investigating and will provide a further update shortly.

Update 09.50
Service has now been restored although we are continuing to investigate. Further updates will be provided.

Update 10.27
We are continuing to investigate the cause of this system failure.  Calls were affected between 09:25 and 09:47 today.  Normal service has been resumed.  An outage report and further details will be provided as soon as possible.

We would like to apologise for the inconvenience caused by this issue


Read more »



Oct
21
Unplanned Outage – DC3 Wakefield
Posted by adam.heavens on 21 October 2011 02:24 PM

We are currently investigating an issue with the power at our Wakefield Datacentre (DC3), which may be affecting some customers.  We believe that all services have now been restored.  Further updated will be posted shortly.

Update 15:14
During monthly routine power load testing, a sequence of failures to the generator back-up system resulted in a 30 second power outage to the site.  This test has been carried out successfully every month since the site opened to ensure power back-up systems perform correctly in the event of a power failure.

The root cause of the issue is being investigated and generator and UPS engineers are currently on-site to carry out a full investigation.  Once this investigation is complete a full report will be made available.  This is the first such occurrence Node4 have experienced since the business started.  We take this issue very seriously and apologise for any inconvenience.  Node4 technical staff are also available at site if any technical assistance is needed.

Update 15:53
Engineers in DC3 have been checking that all equipment is live and back in service.  If you are still experiencing issues with your equipment, please make sure you log a ticket with the helpdesk and an engineer will be available to assist.

We have an outstanding issue with one of our voice interconnects.  During the power issue equipment belonging to one of our voice interconnect providers failed.  We have an engineer on site investigating this and hope to restore full service shortly.  This is impacting a small percentage of customers who are experiencing issues with inbound calls, and intermittent issues with outbound calls.

Update: 17:00
Generator engineers have completed diagnostics tests on both gensets and not found any issues. We are awaiting feedback from the UPS engineers. Power and cooling systems are operating and we have additional engineers onsite to deal with remote hands requests from customers.

Update: 18:00
Normal service for voice customers was restored at 17:40. Further information will be included in the Outage Report that will be made available once the investigations have been completed.  Apologies again for any inconvenience caused

Update 19:50
Panel board and generator engineers will be onsite at approximately 8:30pm to begin further diagnostic works. These works are classed as non-intrusive and low risk, however, given the nature of the problem, we will issue a general “At Risk” notification when this work is undertaken. This website will be updated before we begin this process.

Update 21:35
We are in the process of completing panel board tests and off-load generator tests. On successful completion of these, we plan to perform a full on-load test at 10pm. We will update this website after that test has been completed

Update 22:30
We have completed the first onload test and are continuing with our diagnostics.

Update 01:10
We have identified an issue with a relay on the mains / generator changeover control and replaced this part. Subsequent onload tests and diagnostics have been successful and we are confident that the board is operating back under normal condition.

Engineers will be back on site tomorrow morning to complete further diagnostic tests on the generators but we are satisfied that any further risk has been mitigated. We will continue to update this site with any further information and, of course we will issue a full incident report as soon as we can. Customer’s affected by today’s incident should contactsupport@servercentre.net or any ongoing technical problems or your account manager for any other queries.

Server Centre would like to apologise again for the disruption to service that this has caused.

Best regards

Server Centre Technical Support


Read more »



Aug
1
Connectivity Issues – 01/08/2011 between 10.30 -11.30
Posted by Adam Heavens on 01 August 2011 11:45 AM

Between 10.30 and 11.30 today customers may have experienced issues with Internet Transit and MPLS Services.

The symptoms were slow internet access, web page timeouts and general poor connectivity.

We traced the source of the problem to a 10Gbs Ethernet link between two of our POP’s. This link has now been shutdown and traffic has been re-routed via alternative circuits.

Further information will follow later today, and an incident report will be available from your account manager.

Please accept our apologises for any inconvenience this disruption may have caused.

Best regards

Server Centre Technical Support


Read more »



Apr
20
Network Issues Affecting UDP Traffic
Posted by Thomas Brown on 20 April 2011 11:18 AM

We are currently experiencing issues relating to UDP traffic on our network.  Our Network Engineers have restored normal service for majority of our customers, and are currently working on a complete resolution to the issue

We apologise for any inconvenience.  Further updates will be posted here shortly.


Read more »



Feb
9
Network Issues Affecting Multiple Services 09/02/11 06:48
Posted by Thomas Brown on 09 February 2011 09:36 AM

Update 10/02/11 12:00:

To confirm, the last service affecting outage occurred at 10.26 on 09/02/11. The 10Gbs circuit has been backup since 18.30 on 09/02/11, and therefore resiliency was restored at that point.

A post incident report is now available to our customers; this can be requested from your account manager or through a support ticket.

Update 10/02/11 06:56:

We received official confirmation of the “all clear” at 21.00 on 09/02/11

Update 09/02/11 18:56:

Work to repair the broken fibre (traced back to Olympic Village) has been on-going all afternoon. The circuit was restored at 18:30 although we are still to receive official confirmation that the circuit has been fully repaired. We have request an incident report from the provider, however it appears to be a case of the fibre cable being damaged by building work. We will produce an official report relating to this incident, this will be available from midday 10/2/11.

Update 09/02/11 10:55:

ETA on the fix of the fibre break is midday.

Update 09/02/11 10:35:

At 03:16 today the 10Gbs circuit between Derby and Global switch London failed. This was raised with the circuit provider and an underground fibre break has been identified.

This failure did not have any significant impact as traffic was automatically re-routed via Telehouse and Manchester, however we have now experienced a number of link flaps on the 10Gbs circuit between Telehouse and Manchester. Unfortunately we can’t permit the circuit provider to carry out any intrusive testing while the other circuit is down. The first link flaps were experienced between 06.48 and 07.00, we’ve seen flaps again at 10.11 to 10.13 and 10.25 to 10.26. We are currently waiting for an ETA on the fix for the fibre break.

Background:

Between 06.48 and 07.02 we experienced BGP routing instabilities within the core network.

Impact:

Internet Transit and MPLS services will have experienced intermittent connectivity during this time.

Senior Engineers are investigating the cause. Further updates will follow.

Regards,


Read more »