Vision Agents is an open-source project developed by Stream that enables the creation of intelligent, low-latency video experiences powered by custom models, infrastructure, and use cases. It provides a set of building blocks for developers to build voice and vision agents quickly with any model or video provider, utilizing Stream's edge network for ultra-low latency. The project supports real-time video AI, low-latency video interactions, and native APIs from providers like OpenAI and Gemini. Vision Agents also features a pluggable processor pipeline for YOLO, Roboflow, or custom PyTorch/ONNX models, turn detection with VAD, diarization, and smart turn-taking, and phone integration via Twilio or Telnyx with bidirectional audio streaming. Additionally, it includes tools for MCP and function calling, allowing agents to validate data and take real-world actions during live conversations. The project has a wide range of applications, including real-time coaching, video understanding, and custom video models for security monitoring and moderation. Vision Agents is designed to be production-ready, with built-in HTTP server, Prometheus metrics, horizontal scaling, and Kubernetes deployment support.
Solves
Vision Agents solves the problem of creating intelligent, low-latency video experiences that can understand and respond to user input in real-time. It addresses the challenges of building voice and vision agents that can process video and audio inputs, detect turns, and respond accordingly, all while maintaining low latency and high performance. By providing a set of building blocks and a flexible architecture, Vision Agents enables developers to create custom video experiences that can be used in a variety of applications, from real-time coaching and video understanding to security monitoring and moderation.