PawPals Connect

MIT Reality Hack Hackathon Team Lead
January 2025

Paw Pals Connect is a multiplayer social AR app for the Apple Vision Pro that features off-cloud LLM-driven virtual dogs designed to connect you with fellow virtual pet owners! The vision is rather than "solving loneliness" by giving people a virtual pet, Paw Pals are supposed to learn more about you overtime and create happy coincidences in real life so that you can organically connect with fellow humans! We leveraged Estuary, an open-source multimodal framework for building low-latency real-time socially interactive agents to bring our PawPals to life. I developed this project with a team of 4 other multidisciplinary students at the 2025 MIT Reality Hack Hackathon over the span of two days. Our work was warmly received by the judges and mentors with many remarking on the novelty of our concept and the high degree of technical sophistication in our design, going one step further to prioritize privacy and data security of conversational data by leveraging Estuary's completely off-cloud capabilities.

If you are interested in using or contributing to Estuary , please feel free to check out our project website where you can find our GitHub repo, documentation, and Discord Server! Please give us a star on GitHub if you'd like to support us!

Awards

• Best Use of Apple Vision Pro Award at MIT Reality Hack 2025

Accomplishments

• Concepted and led this project with a completely new team formed at the hackathon
• Set up the project architecture and development environment for the team
• Developed an embodied, semantically aware, conversational agent on the Apple Vision Pro using Estuary that can intelligently interact with dynamic environments and users in realtime
• Trained the team on how to develop for Apple Vision Pro and troubleshooted technical issues in a high stress high radio noise environment

Key Tools Used

• Estuary
• Unity + Polyspatial + AR Foundation
• PyTorch
• LangChain
• HuggingFace
• Ollama