|
Apr 26 |
[Comtec Announce] [IP Voice Services] Service Alert : SAA0023 26/04/12 13:02
Posted by Michael Cullen on 26 April 2012 03:26 PM |
|
Please be advised of the following Service Alert on the IP Voice Services platform: Date: 13:02 to 13:09 on Thursday, 26/04/2012. Description The IPVS Engineering team have identified an issue on the Call Recorder causing the web GUI to become unresponsive. It is understood that stuck user sessions in the web GUI were accumulating and causing unexpectedly high memory usage, in consultation with the vendor the decision was taken to restart the Web Server element of the system at 13:00. Impact This caused the web GUI for the Call Recorder to be offline from 13:02-13:09, during this period calls would have been recorded as usual. Only clients who use the network call recording will have been affected. The IPVS Engineering team are continuing to monitor the memory usage of the Web Server, if any further actions are required we will update you accordingly. Regards, Comtec Network Operation Centre For urgent operational issues please always contact noc at comtec.com or 0845 899 1423 and not any named individual. Read more » | |
|
Apr 18 |
[Comtec Announce] 21CN Service Outage - 18/04/12 18:30 - Update #16
Posted by Nick Claxson on 18 April 2012 06:31 PM |
|
I can confirm that all circuits were restored at 10:15 this morning and that service has been stable since that time. The total outage duration was 23 hours, 7 minutes. Our management team are now in discussions with BT management regarding the scale and duration of this disaster, and what can be done on both sides to mitigate the risk of a future occurrence. We are also told that a project team has already been assembled within BT Openreach to analyse this outage with a goal of improving routing diversity. We will provide a final update with the outcome of these discussions in the next few days. I can confirm that no customer with redundant connections into our network was affected. While, in this instance, BT missed their repair deadline due to the severity of the third-party damage, we do advise customers who rely on network connectivity for business operations to consider the cost/benefit of adding resilience to such connectivity. We have a range of products that fill this need, from guaranteed diversely-routed fibre, to copper EFM or ADSL backup with automatic failover. Please talk to your account manager if you wish to discuss these further. Regards, Comtec Network Operation Centre For urgent operational issues please always contact noc at comtec.com or Read more » | |
|
Apr 18 |
[Comtec Announce] 21CN Service Outage - 17/04/12 11:08 - Update #15
Posted by Michael Cullen on 18 April 2012 10:24 AM |
|
Service was restored on our customer data circuits at 10:15. We do not have confirmation or an "all-clear" from BT, however, so we cannot rule out further risk until the whole work is complete. A further update will follow when we have more information from BT. Read more » | |
|
Apr 18 |
[Comtec Announce] 21CN Service Outage - 17/04/12 11:08 - Update #14
Posted by Michael Cullen on 18 April 2012 09:35 AM |
|
There has been no further information from BT. The BT estimated resolve time is still 11:00. During this time we will remain in contact with BT. The next update is due at 11:00 although any earlier information will be provided upon receipt.
Comtec Network Operation Centre For urgent operational issues please always contact noc at comtec.com or Read more » | |
|
Apr 18 |
[Comtec Announce] 21CN Service Outage - 18/04/12 08:00 - Update
Posted by Nick Claxson on 18 April 2012 08:23 AM |
|
BT have now revised their estimated time to repair, stating that works should be complete by 11:00 today as opposed to the 18:00 estimation previously submitted. Again, we will remain in contact with BT and advise upon any further changes to this latest status update. The next update is due at 09:00 although any earlier information will be provided upon receipt. Regards, Comtec Network Operation Centre Read more » | |
|
Apr 18 |
[Comtec Announce] 21CN Service Outage - 17/04/12 11:08 - Update
Posted by Michael Cullen on 18 April 2012 12:12 AM |
|
Apologise for the lateness of this update. As you may imagine, BT faults management are very busy today with this major service outage. We have gathered the following information from BT: Around 11am this morning, a tunnelling machine operated by a third party dug through the 250-odd fibres that link Leytonstone, Poplar and Deptford exchanges together. Poplar in particular services the London Docklands area, in which a majority of the UK's internet infrastructure is located. New replacement fibre has been laid into the ground and splicing of the old and new fibre has now commenced in a prioritised order. Emergency service circuits will be reconnected first and then customer circuits should begin to be spliced from around 01:00. We have also had confirmation on the estimated time to repair. BT are still saying that the that the estimated repair time is 24 hours from 18:00 this evening, however they also have said that individual circuits will start to work as and when the relevant fibres are connected. BT expect all splicing to be completed by morning, though they cannot guarantee this due to the nature of the work required. Customers with redundant connections into our network should note that they are also "at-risk" to further fibre break until this work is completed. We will remain in contact with BT throughout and update accordingly. The next update will be at 08:00, or earlier if there are further developments. Regards, Comtec Network Operation Centre Read more » | |