The Case for Domain-Specific Networks

Modern parallel computers are dichotomized into capacity or capability systems. Capacity systems cater to a wide range of weak scaling workloads, using distributed parallel systems with message passing while capability systems focus on strong scaling workloads across a significant fraction of the machine's processing units. The interconnection network differs under these regimes, with commodity Ethernet or Infiniband solutions typically deployed for capacity systems, while capabilityclass systems often necessitate tightly-coupled, fine-grained communication. Systems built for AI training and inference embody traits from both classes: tight coupling and strong scaling for model parallelism, and weak scaling for data parallelism in a distributed system. Handling 100-billion-parameter large-language models and trillion-token data sets presents computational challenges for current supercomputing infrastructure. This paper discusses the crucial role of the interconnection network in these large-scale systems, advocating for flexible, low-latency interconnects that can deliver high throughput at large scales with tens of thousands of endpoints. This work also emphasizes the importance of reliability and resilience in enduring long-running training workloads and demanding inference requirements of domain-specific workloads.