The operator presented the findings of its internal investigation into why its network could not handle emergency calls for several hours.
Stéphane Richard, Chairman and CEO of Orange, assigned the Group’s Internal Audit department to carry out a full investigation into this crisis to identify the root causes and to provide recommendations.
On 2 June, between 16:45 and midnight, Orange’s voice services as well as access to certain emergency services were severely disrupted nationally. During this period, about 11% of calls to emergency services were not routed properly – about 11,800 calls.
This was not due to a cyberattack, but a software malfunction which affected the interconnection between mobile voice and Voice over IP (VoIP) services on the one hand and those hosted on the PSTN switched network (including most emergency numbers).
The incident occurred after a network update, initiated in early May, to increase capacity. The interconnection architecture is based on a platform of call servers, which turns out to have had a software bug which caused the malfunction and servcie disruption.
The bug was activated by the application of standard reconnection commands – and here's the really scary bit, despite their redundancy over six different sites. Orange says the software problem has been identified and fixed by the supplier of the system concerned.
The alert process
Orange’s technical teams identified the software malfunction immediately but were unable to restore the service due to the complexity of the malfunction, the variety of technologies involved and the specific network architectures of the emergency services – it took about 100 experts several hours to restore service.
Orange acknnowledged that despite the swift activation of technical teams, the managerial crisis committee was late communicating with all stakeholders about the crisis.
On 3 June, in agreement with the Inter-ministerial Crisis Committee, Orange set-up a dedicated structure aimed at dealing with residual local situations reported by the Prefectures (French regional authorities) on a case-by-case basis.
Despite prompt action by technical teams, clearly faster dissemination of information to the various stakeholders is essential, such as public authorities, emergency services and the media.
The investigation by the Group’s Internal Audit department also proposed several concrete recommendations:
• Strengthen end-to-end supervision of critical services and emergency numbers. -
• Reduce the maximum time for triggering a crisis committee from two hours to 30 minutes in the event of any disruption affecting calls to emergency services or other critical services at a national level.
• Support State services in accelerating the migration of public service call centers and businesses from PSTN to IP technology in order to strengthen the resilience of this equipment.
• Set-up a dedicated number, available 24 hours a day, 7 days a week, for all relevant stakeholders (State services, hospitals, emergency ambulance services, etc.) in the event of a problem regarding emergency numbers.
• Provide for the use, in consultation with each actor concerned, of a mechanism for the mass distribution of information by SMS in the event of a breakdown affecting the emergency services. - Pursue investigations with an analysis of the event in comparison with similar incidents in other European countries such as Germany, the United Kingdom, Belgium and the US.
As Chairman of the GSMA, Stéphane Richard will also propose, at a global level, the creation of a team responsible for listing and analyzing sensitive network malfunctions in order to better share feedback between operators.
Orange will continue its investigations alongside State services to learn as much as possible from this crisis. In particular, the Group will contribute to the investigation led by France’s National Agency for the Security of Information Systems (ANSSI).