Interactive Sound Source Localization using Natural User Interfaces and Mobile Devices

In this paper we present a system for sound source localization and directional audio based on two consumer NUIs, namely the Kinect extension module for the Microsoft XBOX 360 game console providing a video camera, a depth camera and a microphone array together with a Google Android mobile phone equipped with a touch sensitive display and an accelerometer. Audio is captured by the microphone array from a Microsoft Kinect which is connected to a personal computer. A beamforming algorithm enables the microphone array to capture audio from a certain direction and suppresses audio from other directions. Directional audio can be computed for several directions at the same time which makes it possible to locate sound sources. The directed audio signal is then streamed to a Google Android mobile phone wirelessly and played back on the device. By moving a finger over the touchscreen the user can control the direction of the perceived audio signal. To help the user in determining the desired beam angle a video stream from the Kinect video camera is also displayed on the mobile device. This enables the user to directly point to an object captured by the Kinect and listen to it. We give an overview over the Microsoft Kinect and its audio capabilities and present the software architecture of the system.