News
|
Jun 11 |
[Core Network] Service Alert 22nd May 2018 - Update 1
Posted by David Croft on 11 June 2018 08:33 AM |
|
This is an incident notification regarding an outage on our core network. Date: Tuesday, 22nd May 2018 Start time: 14:46 BST End time: 16:44 BST (Final clear) Services affected: Intermittent partial and total outage across our IP network Report: A failure of a network device in our core IP network led to a cascade of additional failures across the network. Controlled shutdowns of portions of our network were necessary to bring it back to a stable state in order to fully restore service. Both the failure and the controlled shutdowns caused packet loss and temporary routing failures to services hosted on or delivered through our network. Root Cause Analysis: A memory exhaustion on a edge transit router caused it to restart its BGP process, and the consequent withdrawal and announcement of all routes from BGP caused other devices on the network to suffer similar failures in a cascading fashion. Next Steps: We are bringing forward our upcoming planned network upgrades, which will now take place in July. Emergency maintenance sessions will be announced once lab testing has been completed. In the meantime we will continue to observe the change freeze on our network to prevent a further incident. Regards, David Croft Read more » | |
|
May 22 |
[Core Network] Service Alert 22nd May 2018
Posted by David Croft on 22 May 2018 05:35 PM |
|
This is an incident notification regarding an outage on our core network. Date: Tuesday, 22nd May 2018 Start time: 14:46 BST End time: 16:44 BST (Final clear) Services affected: Intermittent partial and total outage across our network Report: A failure of a network device in our core network led to a cascade of additional failures across the network. Controlled shutdowns of portions of our network were necessary to bring it back to a stable state in order to fully restore service. Both the failure and the controlled shutdowns caused packet loss and temporary routing failures to services hosted on or delivered through our network. Our network is built with a high level of resilience, and we and our vendor are investigating as a matter of urgency how this failure was able to have such a wide-reaching impact and the steps we will need to take to prevent it recurring. Regards, Read more » | |
|
Apr 25 |
[ADSL/FTTC] Service Alert 10th April 2018 - Update 3
Posted by David Croft on 25 April 2018 10:37 AM |
|
This is an incident notification regarding ADSL/FTTC services. Date: Tuesday, 10th April 2018 Start time: 11:00 BST End time: 11:58 BST Services affected: ADSL/FTTC circuits provided by TTB (excluding BT and Cloud Ethernet FTTC). Report: An incident occurred on the TTB network that caused a large number of ADSL/FTTC customers to disconnect and be mostly unable to reconnect. This affected all customers across all ISPs using the TTB broadband network. TTB have informed us that the outage was due to a faulty card at Telecity Manchester which caused a backplane failure. The faulty card was replaced and traffic was re-established. The carrier has postponed all planned internal network changes in order to investigate why the card failed and how this was able to cause such a widespread outage in order to prevent a reoccurrence. This is the final update on this service alert. Regards, David Croft Read more » | |
|
Apr 10 |
[ADSL/FTTC] Service Alert 10th April 2018 - Update 2
Posted by David Croft on 10 April 2018 12:04 PM |
|
This is an incident notification regarding ADSL/FTTC services. Date: Tuesday, 10th April 2018 Start time: 11:00 BST End time: 11:58 BST Services affected: ADSL/FTTC circuits provided by TTB (excluding BT and Cloud Ethernet FTTC). Report: We are aware of an issue on the TTB network that has cause a large number of ADSL/FTTC customers to disconnect. TTB informed us that a core device within their network had failed and customers should be reconnecting automatically via alternate routes. At this time we can see that the majority of our customers have reconnected. If your device has not reconnected automatically, we recommend power-cycling it. If it still does not reconnect, please open a ticket with noc at comtec.com or by phone to 0845 899 1423. Regards, David Croft Read more » | |
|
Apr 10 |
[ADSL/FTTC] Service Alert 10th April 2018 - Update 1
Posted by David Croft on 10 April 2018 11:45 AM |
|
This is an incident notification regarding ADSL/FTTC services. Date: Tuesday, 10th April 2018 Start time: 11:00 BST Services affected: ADSL/FTTC circuits provided by TTB (excluding BT and Cloud Ethernet FTTC). Report: We are aware of an issue on the TTB network that has cause a large number of ADSL/FTTC customers to disconnect. TTB have informed us that a core device within their network has failed and customers should be reconnecting automatically via alternate routes. If your device does not reconnect automatically, we recommend power-cycling it. If it still does not reconnect, please open a ticket with noc at comtec.com or by phone to 0845 899 1423. Regards, David Croft Read more » | |
|
Apr 10 |
[ADSL/FTTC] Service Alert 10th April 2018
Posted by David Croft on 10 April 2018 11:23 AM |
|
This is an incident notification regarding ADSL/FTTC services. Date: Tuesday, 10th April 2018 Start time: 11:00 BST Services affected: ADSL/FTTC circuits provided by TTB (excluding BT and Cloud Ethernet FTTC). Report: We are aware of an issue on the TTB network that has cause a large number of ADSL/FTTC customers to disconnect. At this time they are unable to reconnect. This event is impacting ISPs nationwide and TTB is working to resolve it as fast as possible. Regards, David Croft Read more » | |