llm-d: Building a Cross-Workload, Cross-Modal, Cross-Hardware, and Cross-Platform Distributed Inference Control Plane

Talk overview

This talk introduces llm-d's distributed inference control plane, including intelligent routing, endpoint selection, inference pools, model servers, prefix- and load-aware scheduling, hierarchical KV-cache offloading, Prefill–Decode disaggregation, expert parallelism, flow control, and elastic scaling across Kubernetes, Slurm, Ray, and bare metal.