Is CDC Service down?

December 6, 2024 10:17 GMT
Monitoring - At 09:46 (GMT) on December 6, 2024, Custodian’s monitoring systems detected a fault in one of our core routers, specifically affecting a line card. As a result, this segment of our network infrastructure reloaded, potentially causing a brief disruption for customers relying on one of the redundant links we provide.

We believe this to be an isolated incident, but our team will continue closely monitoring the affected device to ensure stability.

As always, we are committed to minimizing any customer impact.

This incident is being tracked under [CDC-ID: #251211].

December 6, 2024 14:36 GMT
Resolved - Line card remained stable.

October 9, 2024 09:33 BST
Investigating - We have experienced a double fail/restore/fail event on a peering transport link with COLT at 09:21 and 09:24. We have now taken the link out of service to prevent additional disruption while the issue is investigated by COLT.

October 9, 2024 10:06 BST
Monitoring - The physical links have remained stable, but we are holding the service down until we receive confirmation from COLT that the underlying cause has been resolved on their side.

October 9, 2024 11:30 BST
Resolved - COLT have assured us that this is now resolved. The outage was caused by a "physical fibre disruption" on a multi-client core link at Crawley, an error by one of their engineers.

We have raised sessions once again now and will be monitoring.

August 29, 2024 17:17 BST
Investigating - At 16:43 Custodian's monitoring systems detected a connection loss between us and Redcentric - Byfleet.

Services via BYF are currently hard down due to an unknown fault at Redcentric Byfleet.

The provider has been contacted to investigate this issue and provide an update as soon as possible on the outage.

We are continuing to monitor this event and push for further updates.

This case is being tracked under [CDC-ID: # 250231]


August 29, 2024 18:06 BST
Investigating - CustodianDC have observed services start to return.
We are currently running through our own checks to ensure all services have restored back to normal.

At present, we are waiting for a further update on the status of this event and an RFO

Customers that were impacted by this event should start to see services restore if they have not already done so.

August 29, 2024 18:27 BST
Resolved - Redcentric have confirmed the outage was caused by a configuration change on their network.
Services restored when they performed a roll back.

CustodianDC are awaiting an official RFO for the case.
In the mean time we will continue to monitor the stability of these services.

Customers affected by this event should have now see services restored fully.

August 29, 2024 06:34 BST
Investigating - At 05:33 Custodian's monitoring systems detected a dark fibre loss between our Dartford site and Goswell Road.

Protected and resilient services automatically rerouted and are operating via other dark fibre links. Unprotected services on this leg are now hard down.

The fibre provider has been engaged to begin fault-finding.

Updates will follow as investigations with the provider progress. We will strive to keep customer impact to a minimum as always.

This is being tracked under [CDC-ID: # 250216]


August 29, 2024 08:56 BST
Investigating - Fibre engineers continue to investigate this fault. Direct optical and unprotected services between DA2 and GOS will be impacted until this is resolved. Protected/resilient services automatically restored at the point of failure and are therefore unaffected.

August 29, 2024 09:43 BST
Identified - Fibre engineers have identified the break location and are now enroute with an assessment and repair crew.

August 29, 2024 11:03 BST
Identified - Additional engineers from the provider are due to site and the fault location soon.

ETA given by provider of 12:00hrs

August 29, 2024 12:31 BST
Identified - Engineers are on-site at the suspected fault location since 12:12 and are continuing their investigations.

August 29, 2024 13:17 BST
Resolved - The physical break has been found and was re-spliced at 12:55

All impacted services have restored - and resilient services now have full protection once more.