Network status will be displayed on this page

Incidents

Below is a list of ongoing incidents and their details.

Updates:

Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: Service Restored

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
- The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage
- Virgin Media confirm the first splicing team arrived on site assessing the damage. Additional splicers requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860c
- Further out of hours resources arrived on site
- ERS splicers and Network Engineer determine if there are any spare fibres that can be utilised to restore services
- The Virgin Media Network Engineer and Splicers move the impacted services onto spare fibres
- Virgin Media have asked to re test services as they are seeing affected links back online, however, initial checks from CityFibre TSC still show our services down
- CityFibre Major Incident team liaise with Virgin Media regarding impacted services. Virgin Media confirm they continue to investigate
- CityFibre MI team escalated within Virgin Media for an updated plan to restore services
- Virgin Media splicer reversed the splice to restore service at 05:26


Resolution Notes:
The Virgin Media on site Splicer identified a spare fibre and moved the services onto the spare fibre at 04:05. At 04:07 Virgin Media was notified of impact. The Splicer was instructed to reverse the Splicing change made at 04:05, this was completed by 05:30.

At 05:26, CityFibre monitoring shows services restored and CityFibre NOC confirmed the ENNI is now Up.

CityFibre and Virgin Media completed a period of monitoring to ensure on site works were complete and no further impact to service was seen.

Resolved: 25/06/22 05:26


We aim to have the full Major Incident Report following a Major Incident review made available within 10 working days of the incident being resolved.

Should you require a copy of this report, please contact Service.Management@cityfibre.com
25/06/2022 07:26
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 7

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
- The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage
- Virgin Media confirm the first splicing team arrived on site assessing the damage. Additional splicers requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860c
- Further out of hours resources arrived on site
- ERS splicers and Network Engineer determine if there are any spare fibres that can be utilised to restore services
- The Virgin Media Network Engineer and Splicers move the impacted services onto spare fibres
- Virgin Media have asked to re test services as they are seeing affected links back online, however, initial checks from CityFibre TSC still show our services down
- CityFibre Major Incident team liaise with Virgin Media regarding impacted services. Virgin Media confirm they continue to investigate
- CityFibre MI team escalated within Virgin Media for an updated plan to restore services


Current Action Plan:
Virgin Media Network Engineers on site continue work on restoring service for the remaining services.

Restoring service remains the priority.

Next Update: 25/06/22 07:30 (or sooner if available)
25/06/2022 03:08
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 6

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
- The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage
- Virgin Media confirm the first splicing team arrived on site assessing the damage. Additional splicers requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860c
- Further out of hours resources arrived on site
- ERS splicers and Network Engineer determine if there are any spare fibres that can be utilised to restore services
- The Virgin Media Network Engineer and Splicers move the impacted services onto spare fibres
- Virgin Media have asked to re test services as they are seeing affected links back online, however, initial checks from CityFibre TSC still show our services down
- CityFibre Major Incident team liaise with Virgin Media regarding impacted services. Virgin Media confirm they continue to investigate
- CityFibre MI team escalated within Virgin Media for an updated plan to restore services


Current Action Plan:
CityFibre TSC, NOC and MI team have confirmed the ENNI remains down.

CityFibre MI team has escalated within Virgin Media for an updated plan to restore services. Virgin Media Duty Manager confirm the MSO is ongoing and investigations continue as a number of their customers remain impacted.

Next Update: 25/06/22 03:00
25/06/2022 00:28
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 4

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
- The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage
- Virgin Media confirm the first splicing team arrived on site assessing the damage. Additional splicers requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860c
- Further out of hours resources arrived on site
- ERS splicers and Network Engineer to determine if there are any spare fibres that can be utilised to restore services
- The Virgin Media Network Engineer and Splicers continue working to move the impacted services onto spare fibres


Current Action Plan:
Virgin Media have asked to re test services as they are seeing affected links back online, however, initial checks from CityFibre TSC still show our services down.

CityFibre Major Incident team is liaising with Virgin Media regarding impacted services. Virgin Media confirm they continue to investigate.

Next Update: 25/06/22 00:30
24/06/2022 23:04
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 4

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
-The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage
- Virgin Media confirm the first splicing team arrived on site assessing the damage. Additional splicers requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860c
- Further out of hours resources arrived on site
- ERS splicers and Network Engineer to determine if there are any spare fibres that can be utilised to restore services


Current Action Plan:
The Virgin Media Network Engineer has confirmed that there are enough spare fibres to restore all services.

The Virgin Media Network Engineer and Splicers continue working to move the impacted services onto spare fibres.
Services are expected to start to restore imminently, the suppliers are monitoring for service restoration.

Next Update: 23:00
24/06/2022 21:43
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 3

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
-The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage
- Virgin Media confirm the first splicing team arrived on site assessing the damage. Additional splicers requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860


Current Action Plan:
Further out of hours resources have arrived on site to continue repair works for the evening. The current plan is for the ERS splicers and Network Engineer to determine if there are any spare fibres that can be utilised to restore services. Should no spares be available, work will begin on pulling in fibre/coax straps and splicing them into place, the ETA will be confirmed on arrival of the replacement fibre.


Next Update: 22:30
24/06/2022 19:28
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 2

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing
-The supplier Virgin Media have confirmed the fibre break. Identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints
- Additional resources requested with an ETA for splicers of 16:00. An ERS Cable Gang and Civils crew have also been requested to attend site
- The Network Engineers are assessing the ducting for potential additional damage.


Current Action Plan:
Virgin Media confirm the first splicing team are currently on site assessing the damage. Additional splicers are being requested, in addition to a Civils and Cable crew with 2 500m lengths of 288f and 150m of 860.

CityFibre continue to liaise with the supplier and will provide updates as investigations progress.

Next Update: 19:30
24/06/2022 17:07
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Update Number: 1

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked
- Supplier Virgin Media NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted
- Virgin Media Network Engineers despatched and are completing OTDR testing
- Virgin Media engineers have identified a fault and identified further impact in the area nearby to their datacentre
- Additional fibre engineer investigating the fibre break and completing OTDR testing


Current Action Plan:
The supplier Virgin Media have confirmed the fibre break. Following OTDR testing, an event was identified at 2.1km on one leg of a 17.5km span which indicates an event on a 288f cable between two joints. Virgin Media have requested additional ERS resources. The ETA for splicing resources is expected to be 16:00.

An ERS Cable Gang and Civils crew have also been requested to attend site - ETA to be confirmed.

The Network Engineers are assessing the ducting for potential additional damage.


Next Update: 17:15
24/06/2022 15:21
Description: Major Incident - INC0104173 - Multiple Off-Net Leased Lines down

Current Impact: TLOS for up to 100 services

Incident Start Time: 2022-06-24 09:20

Major Incident Raised: 2022-06-24 14:06

Major Incident Manager: Charlotte Brown

Location/City: National

Completed Actions:
- P1 Incident raised
- Raised ticket to supplier to investigate
- Escalation path requested into supplier from CityFibre
- Supplier investigating as a Major Incident
- MI Proposed
- MI Process invoked


Current Action Plan:
The Supplier is currently investigating as a Major Incident. Their NOC team attempted to restore services by moving ENNIs onto backup links, which was only partially successful for a number of customers, unfortunately CityFibre remain impacted. Network Engineers have been despatched and are completing OTDR testing.

The engineers have identified a fault and identified further impact in the area nearby to their datacentre. This is likely why their attempts to failover all services has not been successful.

They are currently continuing their investigations and developing a plan to fix. ODTR testing is still taking place as part of this. Additionally, they have requested that their fibre engineers investigate the fibre break identified from their ODTRs.

Next Update: 15:30
24/06/2022 14:18
We are aware of an ongoing incident affecting a number of services. Our Engineers are conducting preliminary investigations to determine the comprehensive Service Impact Analysis.

We apologise for the inconvenience caused. Our main priority is to deliver the levels of service that our customers deserve and as such we have invoked our Major Incident Process. CityFibre are working to affect a full restoration as soon as possible.

We will continue to issue regular updates until all services have been restored. Further details will be issued within 15 minutes.
24th June 2022 14:06:34

Updates:

Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Service Restored

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Completed Actions:
• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President
• Technical call completed
• Relevant recommended captures applied to CityFibre core
• Core link between Reading - Coventry costed out of the network
• Juniper engaged to assist with investigations
• Ciena engaged to assist with investigations
• Services remain stable following period of monitoring

Resolution Notes:
Following a 5 day period of extensive monitoring, no further impact has been experienced since approx. 12:00BST on the 22nd June 2022 when the MPLS path between Reading - Coventry was costed out of the network. CityFibre continued enhanced monitoring and engagement as we build confidence that our stability is restored.

Investigations into the root cause continue via City Fibre's Problem Management process. CityFibre NOC and Network Engineer teams continue to liaise with Vendors Nokia and Ciena regarding root cause investigations.

Resolved: 22/06/22 12:00

We aim to have the full Major Incident Report following a Major Incident review made available within 10 working days of the incident being resolved.

Should you require a copy of this report, please contact Service.Management@cityfibre.com
27/06/2022 16:37
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 9

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President
• Technical call completed
• Relevant recommended captures applied to CityFibre core
• Core link between Reading - Coventry costed out of the network
• Juniper engaged to assist with investigations
• Ciena engaged to assist with investigations

Current Action Plan:

CityFibre NOC Engineers have continued to monitor the network and no further impacts have been experienced. Service has remained stable since approx. 12:00BST on the 22nd June 2022.

The relevant captures remain in place on recommendation from the cross-party vendors to assist us with our investigations and in the event of a further failure use the logs to determine any potential software bugs across the LSP's.

Nokia believe there may be a software big on their element of the network, however, the continue to investigate and will provide a status update week commencing the 27th June 2022.

The MPLS path between Reading - Coventry remains costed out of the network and this will be the condition until we have a plan of attack to mitigate further impact to service.

The incident remains on extensive monitoring and the Major Incident Team remain engaged to assist in the event of a reoccurrence.

Next Update: 17:30 27/06/2022
25/06/2022 21:53
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 9

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President
• Technical call completed
• Relevant recommended captures applied to CityFibre core
• Core link between Reading - Coventry costed out of the network
• Juniper engaged to assist with investigations
• Ciena engaged to assist with investigations

Current Action Plan:

CityFibre NOC Engineers have continued to monitor the network and no further impacts have been experienced. Service has remained stable since approx. 12:00BST on the 22nd June 2022.

The relevant captures remain in place on recommendation from the cross-party vendors to assist us with our investigations and in the event of a further failure use the logs to determine any potential software bugs across the LSP's.

Nokia believe there may be a software big on their element of the network, however, the continue to investigate and will provide a status update week commencing the 27th June 2022.

The MPLS path between Reading - Coventry remains costed out of the network and this will be the condition until we have a plan of attack to mitigate further impact to service.

The incident remains on extensive monitoring and the Major Incident Team remain engaged to assist in the event of a reoccurrence.

Next Update: 17:30 27/06/2022
24/06/2022 16:30
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 8

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President
• Technical call completed
• Relevant recommended captures applied to CityFibre core
• Core link between Reading - Coventry costed out of the network
• Juniper engaged to assist with investigations
• Ciena engaged to assist with investigations

Current Action Plan:

CityFibre NOC Engineers have continued to monitor the network and no further impacts have been experienced. We have engaged with a number of our customer who have confirmed service stability.

At this juncture we have been unable to determine to underlying root cause of the incident, however, we have all the relevant captures in place on recommendation from the cross-party vendors to assist us with our investigations and in the event of a further failure use the logs to determine any potential software bugs across the LSP's. CityFibre, in conjunction with our vendors, continue to investigate this issue as priority, however, there is potentially several contributing factors.

The MPLS path between Reading - Coventry was removed from the network as part of our investigations and this has helped stabilise services since around 12:00 BST on 22nd June 2022. The network will remain in this situation until we are confident that we have an action plan to mitigate further impacted to service.

Our Technical Teams continue to pursue the live TAC cases with the vendors to (i) repair this path and (ii) determine root cause and remediate the behaviour observed on certain MPLS paths. A further Major Incident Meeting has been scheduled for the 24th of June and the Major Incident Team will release a further statement at 17:30 on the 24th of June 2022.

The incident remains on extensive monitoring and the Major Incident Team remain engaged to assist in the event of a reoccurrence.

Next Update: 17:30 24/06/2022
23/06/2022 17:13
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 7

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President
• Technical call completed
• Relevant recommended captures applied to CityFibre core
• Core link between Reading - Coventry costed out of the network
• Juniper engaged to assist with investigations
• Ciena engaged to assist with investigations

Current Action Plan:

CityFibre, in conjunction with our vendors, have continued to investigate this issue with priority. The current assessment is that there are several contributing factors. CityFibre have removed an MPLS path between Reading - Coventry during the course of investigations and this has helped stabilise services since around 12:00 BST on 22nd June 2022. We are still monitoring the effectiveness on this action.

Our Technical Teams continue to pursue the live TAC cases with the vendors to (i) repair this path and (ii) determine root cause and remediate the behaviour observed on certain MPLS paths. We expect to release a further statement at 17:30

Next Update: 17:30
23/06/2022 11:29
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 6

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President
• Technical call completed
• Relevant recommended captures applied to CityFibre core

Current Action Plan:

CityFibre in conjunction with the vendor have completed their Technical conference call and have redefined the debug setup in the event of further LSP flaps. Following the meeting and recommendation from the vendor CityFibre have implemented additional captures across elements of our core network to capture more detail in the event of a reoccurrence. The technical session covered the detail below, however, at this juncture the underlying root cause has not been identified therefore we have no recovery plan.

• Reviewed the historical debug captures - no issues identified
• On vendor recommendation CityFibre have applied further debug captures on more nodes
• CityFibre network has been more stable today
• Amended the way the debugs are being stored to remove limitations
• Behaviours seen on another element of the network and an incident has been raised with Juniper. Feedback will flow into Nokia when we have the feedback.

At this juncture we have exhausted all avenues of investigation and CityFibre NOC continues to monitor the network for further LSP flaps. Nokia remain on point to support us from a technical perspective and review the analysis as we experience issues across the network.

CityFibre NOC continues to monitor the situation and the MI Team will reconvene tomorrow to review the status of the network overnight.

Next Update: 11:00 23/06/2022
22/06/2022 17:35
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 5

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President

Current Action Plan:

A conference call has taken place between CityFibre and Nokia at senior level at we have agreed an approach to assist with investigations as outlined below:

• Technical Call to be convened with both teams to look at end to end (ETE) path for LSP Flaps
• CityFibre to investigate a Nokia ETE path and pass to the NOC to monitor for LSP Flaps to rule out multi-vendor associations
• Daily management bridge with status updates arranged for 17:00
• Review the current software and devise a plan to upgrade to the latest version
• CityFibre NOC to send any further drops to Nokia

CityFibre are confident that we now have the right level of support and focus from a Technical & Escalation perspective and updates will follow following todays call.

Next Update: 18:00
22/06/2022 12:03
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 5

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia
• Escalation placed into Nokia Vice President

Current Action Plan:

Investigations have continued between CityFibre and Nokia, however, at this juncture they have been unable to establish what is causing the flaps across the network. CityFibre have provided further analysis to assist them with their investigations and the team are analysing the information today.

In additional to the historical information the vendor are now investigating avenues out with a live event that may be causing the trigger as outlined below:

• Analyse all the information provided from CityFibre
• Review the current version of software to determine if there are any software bugs that could be triggering the event
• Review historical changes with their vendor to establish if any changes have taken place that could be causing the impacts
• Review if any third party product is triggering the event

In addition to vendor CityFibre Senior Network Engineers remain engaged to review if there is any action we can take to mitigate impact when the incident conditions present themselves to alleviate impact to our customers.

Escalations into Nokia continues at a Senior Level and a Technical Call has been arranged for 10:00 to review the current status and provide an update on progress.

Next Update: 12:00
22/06/2022 08:59
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 4

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations
• Senior level escalations placed into Nokia
• Escalation acknowledged by Nokia

Current Action Plan:

CityFibre have continued to escalate the incident with the vendor, however at this juncture we have no further information in terms of the recovery plan from the vendor. Escalation into the vendor is now at Vice President level and we have asked them for a Technical Session to devise a plan of action to alleviate impacts across the CityFibre network on the 22nd June 2022.

We apologise for the inconvenience caused. Our main priority is to deliver the levels of service that our customers deserve. CityFibre are working to affect a full restoration as soon as possible.

Next Update: 10:00 22/06/2022
21/06/2022 23:01
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 3

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis
• Nokia have acknowledged the outputs and have commenced their investigations

Current Action Plan:

CityFibre have held a further conference call with Nokia who have confirmed they have received the outputs from the outages this morning and are analysing the logs and anticipate to provide feedback by 22:00.

Our Major Incident Team are escalating into Nokia for further support at a senior level and escalations will continue.

We will provide a further update from the outputs from Nokia.

Next Update: 22:30
21/06/2022 15:49
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 2

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis

Current Action Plan:

CityFibre NOC Engineers continued to monitor the network and identified drops between lon5-1 to lee1957-2 between 05:03 - 05:24. The tech support files, debug and supporting evidence requested by Nokia has been sent to Nokia to allow them to conduct further investigations at 07:00.

Nokia advised that it could take up to 24 hours for their teams to review the logs and determine the plan of action to mitigate impact and we plan reviewing this with the vendor on a call at 15:00.

Further updates will follow when Nokia confirm their position.

• CityFibre NOC to monitor and feedback to vendor
• CityFibre Technical Support Centre to continue to triage incidents on report
• CityFibre TSC to link any associated incidents
• CityFibre Major Incident (MI) Team – Nokia to reconvene at 15:00
• Nokia to review the logs

Next Update: 15:30
21/06/2022 10:10
Description: Major Incident - INC0102827 - Multiple (Off-Net and On-Net) leased lines experiencing intermittent loss of service

Current Impact: Intermittent losses of service.

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Update Number: 1

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis

Current Action Plan:

To allow us to gather the evidence to pass to the vendor to assist with investigations the MI Team and vendor will reconvene on the 21st June at 14:30 and an update will follow.


• CityFibre NOC to monitor and feedback to vendor
• CityFibre Technical Support Centre to continue to triage incidents on report
• CityFibre TSC to link any associated incidents
• CityFibre Major Incident (MI) Team – Nokia to reconvene on the 21st of June

Next Update: 15:30 21/06/2022
20/06/2022 16:12
Description: Major Incident - Multiple (Off-Net and On-Net) leased lines experiencing intermittent drops across the internet

Current Impact: Intermittent losses of service

Incident Start Time: 2022-06-10 16:29:57

Major Incident Raised: 2022-06-20 15:22:31

Major Incident Manager: Stephen Martin

Completed Actions:

• P2 incident raised
• CityFibre Network Operations Centre (NOC) engaged to conduct preliminary investigations
• Escalated to Senior Network Engineers within CityFibre NOC
• Network Engineering Team engaged
• Vendor engaged
• Escalations placed into the vendor
• Bridge call convened between Nokia – CityFibre
• Action plan agreed to assist with investigations/root cause analysis

Current Status:

A number of CityFibre customers have been experiencing intermittent loss of service from the 10th of June 2022. CityFibre in conjunction with our vendor (Nokia) have been gathering evidence of the drops from the impacted customers to assist with investigations. At this juncture the vendor has been unable to determine the cause of the incident and investigations continue, however, they have confirmed the incident conditions are not linked to any hardware.

CityFibre NOC Engineers continue to run a technical support file/debug every hour to assist with investigations on recommendation from the vendor to provide further evidence of the drops in order for them to take remediation to mitigate impact.

Identifying the cause of the incident has been problematic and CityFibre are working with the vendor to establish the correct evidence of the drops. Escalations have been placed into the vendor at a Senior Level and a further call has been scheduled to review the status and output from further drops at 14:30 on the 21st of June 2022.

The Major Incident Team continues to assess the issue, however, at the impact is intermittent root cause determination could be challenging.

Current Action Plan:

• CityFibre NOC to monitor and feedback to vendor
• CityFibre Technical Support Centre to continue to triage incidents on report
• CityFibre TSC to link any associated incidents
• CityFibre Major Incident (MI) Team – Nokia to reconvene on the 21st of June

Next Update: 15:30 20/06/2022
20/06/2022 16:01
We are aware of an ongoing incident affecting a number of services. Our Engineers are conducting preliminary investigations to determine the comprehensive Service Impact Analysis.

We apologise for the inconvenience caused. Our main priority is to deliver the levels of service that our customers deserve and as such we have invoked our Major Incident Process. CityFibre are working to affect a full restoration as soon as possible.

We will continue to issue regular updates until all services have been restored. Further details will be issued within 15 minutes.
20th June 2022 15:22:31

Change status will be coming soon