Turbo: Efficiently Serving Long-Context LLMs with In-Network Aggregation

Talk overview

This workshop presents Turbo, which offloads query multicast and partial-attention aggregation to programmable switches for context-parallel LLM inference. Online aggregation, rolling pipeline state, and load-aware aggregation trees reduce network traffic and decoding latency while preserving output quality.