News
|
Jul 7 |
[IP Voice Services] Service Alert 30th June 2014 - Update 5
Posted by David Croft on 07 July 2014 09:37 AM |
|
This is an incident notification regarding IP Voice Services. Date: Monday, 30th June 2014 Start time: 18:00 BST (UTC+0100) End time: 18:49 BST (UTC+0100) Services affected: IP Voice Services routing to/from PSTN. Report: The engineering team was investigating an issue affecting all off-net call routing. On-net calls remained unaffected. Root Cause: The carrier SBC cluster was unable to process calls due to the license expiring. The engineering team should have received notification from the carrier SBC network management application from 4 weeks prior to the license expiry date. Engineering and TAC use this application daily and in this instance the system did not provide the notification banner as seen previously when licenses were due to expire. The carrier SBC network management system should also have provided an email alert which was not generated. Symptoms: Unable to route off-net calls. Once off-net call routing was restored, subsequent issues of excessive post dial delay and intermittent delays with media establishing on answer were experienced. Resolution: The vendor issued and installed a perpetual license to provide a permanent fix to prevent the license expiry and associated notification failure being an issue in the future. Timeline: 18:05 - Network Monitoring system alerted the Engineering team that the carrier SBC cluster was not processing calls. 18:07 - Engineering identified that the SBCs had stopped processing calls due to the license they were using expiring. 18:08 - The Engineering team contacted the license vendor to request immediate replacements. 18:46 - Replacement license were provided and installed by the Vendor. At this point off-net service was restored. However excessive PDD (Post Dial Delay) and delays with media establishing were still being observed. The Engineering team worked in conjunction with the Vendor to resolve the ongoing problems. It was identified that the heartbeat between the two devices was not establishing correctly causing the secondary to periodically try and come into service. 19:47 - Both of the SBCs were shut down so that they could be restarted cleanly in turn to resolve the heartbeat problems. At this point off-net calls were unavailable. 19:51 - One of the SBCs was returned to service and it was confirmed that call processing to and from off-net destinations was restored and there were no longer any issues being experienced. 20:00 - The second SBC was restored to service and heartbeat re-established cleanly with no affect on service. 20:25 - Service was rolled from one SBC to the other to confirm stability. During this process there was no effect on call processing; all existing calls were sustained and new calls established normally. The Engineering Team and the Vendor closely monitored the service until 22:00 to confirm all service was restored and no residual issues remained. We apologise for the inconvenience this has caused. This is the final update. Comtec NOC was tracking this issue under ticket [#OEE-382-69090]. Best regards, David Croft | |
Comments (0)