China Mobile achieves Level 4 AN in network operation center with intelligent agents

Who: China Mobile and Huawei

What: Automated operations and maintenance and reached level 4 of autonomous networks in its network operating centers (NOCs)

How: Developed automated and intelligent agents and co-pilots for operations and maintenance, supported by investment in accurate telco generative AI (GenAI) and AI models and the use of TM Forum’s frameworks and other assets.

Results:

• The equivalent of 5,500 full-time manual roles replaced by the end of 2024
• Over 30% reduction in backend operations and maintenance (O&M) manpower
• More than 5% savings in frontline installation and maintenance manpower
• 30% reduction in fault and customer complaint mean time to repair (MTTR) on average
• Autonomous network (AN) level of the NOC rose from 3.2 to 4, according to self-assessment using TM Forum AN Levels assessment (ANL)

The size of China Mobile’s network holds up a useful magnifying glass to the impact network faults have on both customers and a company’s bottom line, as well as how AI agents can help address them.

With 578 million 5G customers and just over 1 billion total mobile subscribers, China Mobile runs the world’s largest 5G network by number of subscribers. It also lays claim to operating the largest 400 Gigabits per second optical transport network (OTN) and SRv6 backbone worldwide, as well as the biggest telco-based, single-cluster intelligent computing center.

The extent of its network management needs has given it good reason to be a pioneer in autonomous networks: By 2023 it had reached AN level 3.2 across network-based high-value scenarios. Its next target was to reach level 4, which represents a shift from human-defined automation processes to true autonomous decision-making.

Attaining Level 4, however, depends on end-to-end automation of the network operations center (NOC) in high-value scenarios, and particularly those of fault management and customer complaint handling. China Mobile’s aim was to come as close as possible to eliminating manual intervention in operations and maintenance (O&M) by reducing the number of site visits by engineers and technicians to one per ticket.

In doing so the company saw an opportunity to ensure rapid service activation. “The complete IP backhaul activation process consisted of 54 steps, 41 of which required manual involvement, resulting in prolonged response times, increased error probability and elevated operational expenses,” says Yao Yuan, Project Manager of Autonomous Networks, China Mobile.

Just as importantly it wanted to be able to identify and address faults before they impacted customers. When it came to IP backhaul fault monitoring and resolution alone, in 2023 China Mobile processed 30,000 fault tickets each month in Guangdong, with backend O&M experts spending an average of 45 minutes on each one. And since those backend O&M experts needed to interact with personnel on-site, each ticket typically took between 30 to 100 minutes to resolve.

Automating network operations

The company’s answer was to develop an end-to-end, highly autonomous NOC – also known as a dark NOC – where the need for human oversight of critical functions is eliminated. It did so in association with Huawei, using a range of TM Forum assets and its success won the company the TM Forum Excellence Award for Autonomous Networks.

China Mobile tested network automation solutions in two provinces before national rollout: Guangdong, which is China’s most populated province with more than 126 million inhabitants, and Zhejiang province, which has a population of approximately 65 million. In China Mobile’s Guangdong subsidiary, the focus was on dealing with IP backhaul fault management and home broadband customer complaint handling. Teams in the Zhejiang subsidiary, meanwhile, prioritized RAN fault management and core network fault management.

The company developed two different types of AI agents: role-based copilots and scenario-based agents. These use natural language and intelligent knowledge Q&A to work on fault management and customer complaint handling across radio access, core networks, IP backhaul and home broadband.

Agents live or die by the accuracy of the AI that supports them. Keen to limit inaccuracies and GenAI hallucinations, while at the same time automating the use and upkeep of knowledge graphs, China Mobile together with Huawei built a large AI model with more than 10 billion parameters and specialized algorithms. It also developed autonomous closed-loop operation toolchains that draw on collaboration between large models, which offer the benefit of generalization, and small AI models, which bring specialization.

Other considerations included variations in business models and data patterns between China Mobile’s provincial subsidiaries, which required teams to adapt and train telecoms GenAI / AI models for big data analysis using local data. When it came to the next step of scaling solutions nationally, China Mobile “mapped out a comprehensive view of its operation and maintenance scenarios and established an evaluation system”, according to Yuan. Focusing on a high-value scenario, it “selected provincial subsidiaries and collaborated with equipment, network and OSS vendors to implement a bottom-up deployment practice. This approach allows for rapid replication and promotion from the city level to the provincial level and nationwide,” explains Yuan.

China Mobile also needs to gauge its success. For this, it uses TM Forum’s framework of key business value indicators (KBI), key effectiveness indicators (KEI) and key capability indicators (KCI) as a basis for measuring quality, efficiency and effectiveness, as shown in the graphic below.

Finding an answer

China Mobile says developing ‘intelligent Q&A’ and ‘intelligent analysis’ for real-time queries has helped it improve:

• Q&A accuracy: China Mobile achieved 90% accuracy for answers to questions and reduced to 3% the occurrence of large model hallucinations by integrating the model with RAG (Retrieval-Augmented Generation),
• Intent Query: By leveraging NL2SQL (natural language to SQL), NL2API (natural language to API), the network management system can handle freestyle queries across 1,000+ scenarios.
• Knowledge Q&A: Answers to operational and maintenance-related questions achieve an average accuracy rate of 97%, which is 20% above the industry average, according to China Mobile. This was achieved by combining high-precision dense models with high-efficiency sparse models.
• Interactive fault resolution guidance: Feedback from resolving faults is used to improve future troubleshooting guidance.
• No-Code development: People can use natural language to feed expertise into the system, which the model autonomously converts into workflows.
• Model collaboration: Combines the generalization strengths of large models and the specialization of small AI models to enable autonomous closed-loop operations toolchains.

Both Guangdong Mobile and Zhejiang Mobile report improvements to fault management and reduced MTTR (Mean Time to Repair), which in turn has enhanced customer experience. In addition, the company reduced operational manpower by the equivalent of a total of 5,500, people per year. Around 75% of the manpower reductions were made in network operations, with customer operations accounting for the other 25% and represented approximately 10% of the total workforce in these areas.

The role of TM Forum assets in developing a dark NOC

China Mobile made extensive use of TM Forum assets throughout. Indeed, “without TM Forum assets, the entire process would have been significantly longer, if not impossible,” according to Yao Yuan.

For example, China Mobile deployed GB921 Business Process Framework (eTOM) to define the scope of the work in partnership with Huawei. It then drew on GB1022 ODA Functional Framework for a clear set of functions for defining business requirements, as well as principles for decoupling and integration that simplified management and operations.

The AN Journey methodology defined in IG1218F AN Framework, has helped China Mobile move through assessment, gap analysis, solution design, and implementation, as it builds out a dark NOC.

During the implementation stage, TM Forum’s process framework (eTOM) served as a common language for China Mobile, Huawei, and other parties, unifying the understanding of business requirements and processes.

China Mobile and Huawei have also fed what they learnt from building a dark NOC back into TM Forum standards. This includes contributing an Agent and Co-pilot design to ‘IG1345 Embracing Generative AI in Telecom’, as well as:

• upgrading E2E fault management use case for IG1228 How to use ODA (TMFS022)
• providing new fault management solution to IG1373 Self-Healing Domain
• GB1059 ANLET series with AN Level assessment questionnaire for IP Fault Management and Home Broadband Complaints.
• AI/MLOps.

Key results

Guangdong Mobile

IP backhaul faults:

• 30% of all back-end staff and 5% of field staff managing IP backhaul faults are agents
• 25% reduction in fault MTTR
• 15% decrease in fault tickets
• Real-time self-service replaces 30-minute interactions

Home broadband complaint handling:

• 25% of backend and 6% of field roles are performed by agents
• 50% reduction in fault MTTR
• 30% fall in rate of customer complaints

Zhejiang Mobile

During 2023, Zhejiang rolled out RAN fault management in Hangzhou, the capital city of Zhejiang province. This initial deployment resulted in more than 100 maintenance engineers using the RAN fault management system daily, closing more than 8,000 fault tickets per month across approximately 20,000 sites and over 100,000 cells in the city. Since then, Zhejiang Mobile has achieved the following results across the province:

RAN fault management in the city of Hangzhou:

• 23% of RAN operations performed by digital employees
• 27% reduction in fault MTTR, using precise fault-to-ticket mapping for on-site visits
• 80% reduction in time spent in the back end on troubleshooting

Core network fault / complaint handling in the province of Zhejiang

• 40% of core network back-end operations performed by digital employees
• 87% reduction in core network fault MTTR
• 65% reduction in end-to-end complaint MTTR
• 75% complaint pre-emption rate, up from a previous rate of 5%

Guangdong Mobile

In Guandong, network automation has delivered improvements in:

• IP backhaul fault management – agents diagnosed more than 14,000 work tickets per month from Q4 2023 to the end of 2024.
• Home broadband complaint handling. More than 9,000 installation and maintenance engineers are now equipped with copilots that are involved in resolving 3,000 to 4,000 enquiries daily.

After over a year of deployment and operation in the field, China Mobile has started replicating its network automation solutions nationwide.

“With L4 network autonomy as the goal, Huawei and China Mobile are co-creating the future. Focusing on high-value scenarios of network maintenance, experience optimization, and service operation, we are jointly innovating more Level 4 best practices. Furthermore, we are working on L4 solution packages and target architecture, systematically accelerating China Mobile’s AN journey to L4 autonomy,” according to Sam Wang, the General Manager of ADN Solution, Huawei.