From Hearing You to Seeing You — Real-Time Multimodality Brings AI into the Real World

Talk overview

Over the past two years, large language models have transformed AI from a tool that follows commands into a conversational partner. But an AI that can only listen will always receive the world through the user's descriptions. It cannot tell whether you are reading, cycling, helping a child with homework, or walking around your home. To become the companion that understands you best, AI must share your context and respond to it: vision, sound, space, movement, facial expressions, and the surrounding environment. This is the direction Agora's Physical AI team is pursuing: moving from real-time conversational AI to real-time multimodal AI. This talk will preview Agora's latest development in real-time multimodality: the R2 development kit. More than a conventional hardware development kit for a pocket robot, R2 is an open platform that enables AI to see, understand, and enter the real world. It integrates a camera, a screen, motion with multiple degrees of freedom, stepper motors, a multi-microphone array, and a range of sensors. Dedicated on-device computing supports local perception capabilities from facial recognition to sound source localization. R2 is also part of a broader system. It fully integrates with Agora's real-time conversational AI engine, transmitting audio, video, and data over three parallel channels to coordinate the “brain” — a cloud-based LLM or agent — with the “cerebellum” — on-device vision and motion control — in real time. Combined with the Botstation agent platform and the MCP tool ecosystem, developers can turn an idea into an interactive multimodal robot prototype within days. We will share the thinking behind this journey: Why does AI need to enter the real world? Why build an open development foundation rather than a closed robot product? How can we address developers' real challenges with noise, computing power, mass production, and privacy? And as real-time multimodality becomes new infrastructure for AI, how will it change education, cycling, security, office work, and other applications? This talk is both a product launch and an exploration of AI's next destination. We believe AI's future is by your side, beyond the cloud.