Natural interaction in virtual TV sets through the synergistic operation of low cost sensors

A virtual TV set combines images from the real world with a virtual environment in order to obtain images that give the impression of the real elements, such as actors or physical objects, being present in a computer-generated scene. Thus, the audience has the feeling of the talents being present in a place where they are not. One of the most relevant aspects to obtain a good sense of presence on stage is the capability of the actors to interact, in real time, with the virtual world. To make this possible, it is necessary to track the body of the presenters and detect their gestures so that they can modify the synthetic environment in a natural way. In this paper, a study that analyzes the feasibility of the synergistic use of several sensors to improve the interaction of the actors with the scene, mainly focusing on natural gesture detection, is presented. A new workflow, based on the learning of natural gestures by the system through artificial intelligence techniques in order to use them during live broadcasts, is proposed. Using this approach in the pre-production process, the actors are able to create their own custom paradigm of interaction with the virtual environment, thus increasing the naturalism of their behavior during live broadcasts and reducing the training time needed for new productions.

keywords: Natural user interaction Virtual TV set Microsoft Kinect V2