AICon 深圳 2024: New Frontiers in AI Infrastructure and Engineering

A Gathering of Minds in Shenzhen

On August 21st and 22nd, Shenzhen plays host to the AICon Global AI Development and Application Conference. Over those two days, engineers, researchers, and tech leads from across the industry will converge to talk about what comes after the current wave of large language models. The vibe isn't just about bigger models; it's about making AI dependable in real business environments. The full schedule is now live—ten themed tracks, a hands-on lab, and nearly sixty sessions.

The conference spans a lot of ground: agent engineering, AI infrastructure, native AI development, embodied intelligence, and more. But threading through all of it is a shift in focus. We've spent years pushing model capabilities. Now the question is how to build systems that actually work in production—reliable agents, solid engineering practices, and stable performance under messy conditions.

Spotlight on Communication Bottlenecks

One engineer who knows this challenge intimately is Li Bochao from Huawei's 2012 Lab. He'll be speaking in the AI Infra, Inference Engineering, and Heterogeneous Computing track. His talk, titled "Pangu Training and Inference Communication Optimization: Performance Exploration on Ascend," zeroes in on a silent killer in large-scale AI: communication overhead.

When you're training a MoE (Mixture of Experts) model, AllToAll communication can eat up more than 30% of total end-to-end time. That's been a persistent bottleneck for years. And now there's a new wrinkle. In ultra-long context scenarios—think 1 million tokens—the delay from Host to Device KV Cache transfer has become a major drag on TTFT, the time to first token. If you don't optimize for the specific hardware, you're leaving performance on the table.

Huawei's Ascend: A Different Beast

Li's talk dives into two specific optimizations on the Ascend platform. First, he'll cover MoE AllToAll optimization. By adapting to the Ascend 950's network topology and using its dedicated CCU communication accelerator, his team managed to boost AllToAll performance in the Pangu model's EP communication domain by over 10%.

Second, he'll discuss KV Cache offloading in long-sequence inference. Using Omni Cache's efficient H2D and D2H communication, they achieved a 10% improvement in TTFT. The bigger picture? They want to hide communication time entirely, so it never becomes a visible bottleneck in training or inference.

Who Is Li Bochao?

Li is a systems engineering expert in Huawei's 2012 Lab. He earned his PhD from Sun Yat-sen University and has spent years working on communication protocols—covering data center networks, wide area networks, and even terminal wireless networks. Currently, he's focused on optimizing Pangu's training and inference communication performance, particularly on the Ascend 950. He's led the effort to tune communication operators for that specific chip.

Key Challenges in Communication

During his session, Li will walk through the key challenges. The first is the AllToAll bottleneck in MoE models. In these models, tokens are routed across multiple experts, which requires heavy information exchange. That's where AllToAll comes in, and it's a known pain point.

The second challenge is H2D communication in ultra-long context inference. When you have a million tokens, the KV cache—the key-value pairs that speed up attention—becomes huge. Storing it on the host and transferring it to the device can cause serious latency. It's a new bottleneck that didn't exist in shorter contexts.

Optimizing for Ascend: Three Angles

Li's approach isn't one-size-fits-all. He'll share three angles of optimization:

• Topology affinity: The DeepEP solution doesn't work on Ascend 910A3, so they had to adapt to the Ascend 950's specific network layout.
• Compute affinity: He'll explain how to get the most out of the Ascend 950's dedicated communication acceleration engine, CCU.
• Model affinity: They built custom communication operators to implement flexible collective communication semantics, tailored to Pangu's needs.

These aren't generic tweaks. They're deep, hardware-specific optimizations that sacrifice universality for raw performance. The same strategies would fail on other platforms, like Ascend 910A2/A3 or NVIDIA H20. That's a trade-off you need to be aware of.

Hardware and Software, Hand in Hand

To speed up H2D communication, Li will talk about both hardware and software. On the hardware side, the Ascend 950 provides each NPU with a dedicated H2D pathway. That's a significant departure from shared buses. On the software side, Omni Cache is designed to offload KV cache efficiently, minimizing the transfer time.

He'll also touch on future directions: how to reduce unoverlapped communication time, and how to fuse operators and use multi-stream parallelism to hide communication behind computation.

What You'll Gain

If you're deploying models on Ascend hardware, this session is packed with practical insights. You'll learn about the Ascend 950's hardware features and how to align your training and inference deployment with them. You'll see real-world examples from Pangu's optimization—both in communication operators and parallel strategies—that you can apply to your own models.

But even if you're not on Ascend, the underlying principles matter. Communication is a universal bottleneck. Understanding how to think about topology, compute, and model affinity is valuable no matter the platform.

A Broader Program

Of course, Li's talk is just one piece of the puzzle. The conference has ten tracks, including AI Infra, Inference Engineering, and Heterogeneous Computing; the co-evolution of super individuals and swarm intelligence; key technologies and industry practices for robot AGI; agent security; large model efficiency engineering; and high-value commercial agent applications. More than 50 experts from companies like Alibaba, Tencent, Huawei, Kuaishou, and Google Cloud will be there, along with top academics.

It's a packed agenda. But the focus is clear: moving AI from capability to reliability. The future isn't just about what models can do—it's about what systems can do. And that's a conversation worth having in Shenzhen.