Is CDC Service down?

March 9, 2024 11:17 GMT
Investigating - At 10:55 Custodian's monitoring systems detected a dark fibre loss between Maidstone and Telehouse North.
Services quickly restored back to their normal level after the event and we are investigating the fault with the provider.

Direct services from Maidstone to THN should be considered at risk for the time being.

Services traversing GOS,SOV,MER,GS2,THN. And services direct services from Maidstone to GOS are not impacted by this event.

The provider has been contacted to find out if there are to be any further outages between the two sites and what the root cause of this outage was. Updates will follow as investigations with the provider progress. We will strive to keep customer impact to a minimum as always. This is being tracked under [CDC-ID: # 248547]


March 9, 2024 12:03 GMT
Monitoring - CustodianDC engineers have identified the path of interruption and reviewed all logs from this event. We have confirmed services have returned to their original state.

Services were impacted by an unexpected dark fibre fault between Maidstone and Telehouse, this caused an interruption to IP Transit, backhauls and wave lengths via this direction. Services that are protected migrated over to our alternate routes until the dark fibre fault resolved.

The dark fibre service and subsequent services travailing via this path have been stable since.

We are in contact with our third party provider for this incident and are awaiting confirmation of the cause of this interruption.

While we await confirmation from our third party services are to be considered at risk.

March 9, 2024 18:28 GMT
Resolved - Our third party provider has confirmed they also saw an interruption along the same path that our dark fibre takes, consequently they had already engaged their team to investigate the service interruption.
Our provider has confirmed that they do not anticipate any further unexpected interruptions to the dark fibre service.

We are awaiting for investigations to conclude and for an RFO to be issued by our fibre provider. There has been no additional impact since the initial incident.

February 13, 2024 12:56 GMT
Scheduled - Custodian staff will be performing cabling alterations in our racks at THN and GS2. This is an at risk window only, we do not anticipate causing any interruptions to service.

March 1, 2024 19:00 GMT
Active - Scheduled maintenance is starting.

March 2, 2024 03:00 GMT
Completed - Scheduled maintenance is complete.

January 3, 2024 15:04 GMT
Investigating - We are currently investigating an issue with one of our core routers at THN. This is being investigated and further updates will be provided as we have more info. This will affect IP Transit as traffic is re-routed.

January 3, 2024 15:14 GMT
Investigating - We experienced an unplanned line card reload on one of our core routers in THN.

We are currently investigating the cause and further updates will be provided as we have more info.

Services are fully restored.

Hardware Event Start: 14:52:36 GMT
Hardware Event Recovery: 14:53:19 GMT

Due to global BGP convergence, some IP destinations may have seen an impact for longer than the hardware fault window shown above.

January 3, 2024 16:28 GMT
Monitoring - Services on the line-card in question returned to service immediately after the automated fast-reset was triggered.

The issue appears it was isolated to one network processor (NP) on this particular line-card.

The card and NP have both had significant clean up-time and shown excellent long-term stability, so at present we will be monitoring it for any re-occurrence.

In order to avoid taking unnecessarily disruptive planned maintenance actions.

January 4, 2024 09:09 GMT
Resolved - All services remain fully operational.

December 21, 2023 14:13 GMT
Investigating - We are currently investigating an issue with one of our power feeds at our THN PoP.
All services are still running from the redundant systems. All services traversing THN should be considered at-risk.

December 21, 2023 16:10 GMT
Investigating - Investigations into the issue are still on-going.

December 21, 2023 17:46 GMT
Investigating - We have a CustodianDC engineer on site working alongside THN engineers to investigate the cause of the outage before working towards restoring power to the feed.

December 21, 2023 18:41 GMT
Identified - Our CustodianDC Engineer has located the cause of the outage and the faulty device has successfully been replaced. We are now monitoring the situation to ensure all THN services are stable.

December 22, 2023 07:02 GMT
Resolved - The troubled power feed has been restored and monitored overnight.
All feeds fully operation and no further issues expected.