Creating a 3D Sound System for Paired Volumetric Video

Intel Sports Mathematics, 2017-18

Liaison(s): Cody Gabriel, Peter Sankhagowit, Steven Xing
Advisor(s): Weiqing Gu
Students(s): Daniel Johnson (PM), Daniel Gorelik, Ross Mawhorter, Kyle Suver

Intel’s TrueView technology can reconstruct three-dimensional video at any location in a stadium using multiple video recordings. However, this system does not extend to audio. This project involves developing an approach to reconstruct game audio at any location based on microphone recordings, to combine with the existing volumetric video system. After exploring existing algorithms for sound source separation and localization, we are developing a prototype system called auVVio that extends these techniques to interactively reconstruct game sounds in 3D.