Estuary

Project Lead
September 2023 - Present

Estuary is an open-source multimodal framework for building off-cloud low-latency real-time socially interactive agents. It is designed to be a flexible and extensible platform that can be used to create a wide variety of agents, from embodied conversational agents to virtual assistants. Leveraging an edge computing architecture and integration with both industry-grade and open-source AI models, researchers and developers are afforded the maximum flexibility of which AI models they would like to use without being constrained by hardware architecture incompatibilities or the limitations of cloud computing. Furthermore, as an open-source project, Estuary is designed to be a community-driven project that can be easily extended as well as run completely off-cloud, ensuring user privacy and data security. Finally, integration with Unity gives developers a familiar environment to build their agents in, and the ability to deploy their agents to a wide variety of platforms.

If you are interested in using or contributing to Estuary, please feel free to check out our website, GitHub, or Discord community server linked at the bottom. Please give us a star on GitHub if you'd like to support us!

Conferences

• Case Study surveying the current needs of researchers and developers and how well Estuary addresses those research gaps accepted at CHI 2025
• Technical Demonstration accepted and presented at IVA 2024

Accomplishments

• Leading a multidisciplinary team of researchers and developers to develop an open-source multimodal framework for building off-cloud low-latency real-time socially interactive agents
• Developed for, designed a study, IRB approved, recruited domain experts, conducted live user study, conducted qualitative analysis, and first author (the whole shebang) on a case study paper accepted at CHI 2025
• Developed an embodied, semantically aware, conversational agent on the Apple Vision Pro using Estuary that can intelligently interact with dynamic environments and users in realtime
• First author and demonstrated Estuary at the ACM Intelligent Virtual Agents 2024 conference

Key Tools Used

• Unity + Polyspatial + AR Foundation
• PyTorch
• LangChain
• HuggingFace
• Ollama
• NVivo