Immersive Synthetic Memories

Student Assistant
Mobile & Environmental Media Lab
September 2024 - December 2024

Was there ever a time you wished you could relive a memory, but you couldn't quite remember it? Maybe you dug through your phone; perhaps the good 'ol family photo album... but still nothing. In these cases, generative AI may be the perfect tool to help recreate those special moments. But, to take things even further, what if you can completely immerse yourself into that memory to relive it once more as if you were seeing it from the first person? That's where Immersive Synthetic Memories comes in. This is a project born out of a class I helped teach at the USC School of Cinematic Arts under Professor Scott Fisher. By combining a mixture of commercial and open-source AI technologies with VR headsets, we developed a novel pipeline to generate 180 degrees stereoscopic immersive videos, complete with AI-generated spatial audio.

The Process

We partnered with Barcelona-based company Domestic Data Streamers (DDS) to develop the process of creating these memories. Previously, DDS specialized in generating 2D synthetic memories using commercially available tools, whereas our lab specialized in XR applications. By combining our expertise, we built a novel generative AI pipeline and came up with a process that involves a 15 to 20 minute interview with the guest, whilst a team of prompters generate the synthetic memory in real-time. The guest is then invited to don a VR headset and experience the memory. The results were astounding, with many guests reporting the experience as "uncannily accurate" and "emotionally moving" as you can see in the comparison photo to the left! More details about our process to come in our upcoming paper!

Accomplishments

• Assisted teaching and provided technical support for the development of the Immersive Synthetic Memories generative AI pipeline
• Facilitated and presented Immersive Synthetic Memories at the 2024 Flux Festival

Tools I Worked With

• ComfyUI
• Stable Diffusion
• Unity
• immerGallery