The Case for Virtual Director Technology – Enabling Individual Immersive Media Experiences via Live Content Selection and Editing

An emergence of applications based on live audio-visual content streams could be observed in recent years. While the technological infrastructure in terms of bandwidth and device capabilities has advanced, media formats and related consumption paradigms have not changed as fundamentally. Meanwhile, a considerable amount of research has addressed automatic personalization of multimedia content for the sake of enabling immersive multimedia experiences, however, mostly considering pre-recorded and not live content. This paper states the case for more research to be conducted on what we refer to as Virtual Director technology as one key enabling technology for the hyper-personalization of live content delivery. A Virtual Director is software that automatically selects, frames, mixes and cuts from a number of AV content streams. It aims to automate the complex and challenging tasks that a broadcast director and team undertake during a live event broadcast. Virtual Director software can be applied in a range of use-cases, taking the individual's needs into account. There is unlimited scope regarding which factors such components could reason about in decision making. While such Virtual Director software has been developed as research prototypes, manifold challenges remain open to unlock its full potential. This paper presents recent technological achievements and reects the potential of the approach in two selected application domains, interactive live event broadcast and group videoconferencing.