Deep Learning for VR/AR: Body Tracking with Intel® RealSense™

Skeletal tracking first became viable as a product around 2011, since then, many different additional approaches to the problem have been explored, but almost all of them suffer from high compute usage – which absorbs a lot of GPU or CPU cycles, as well as being designed for general pose estimation. By focusing on VR/AR, it allows us to use the position of the HMD to train with a much smaller machine learning model. Philip will share some of his work on this convolutional neural network with integrated head and controller information approach, as well as talking about some of his prior work on hand tracking in real time using only a CPU.

Intel® RealSense™ depth cameras allow all these solutions to work in multiple lighting situations, and the depth information collected enables higher frame rates, without requiring any additional CPU utilization. By using multiple cameras, this can then become a markerless motion capture solution, with additional volumetric and surface data available for a wide variety of use cases, like gaming or social VR. As an example, Philip will talk about an application he created that uses Unity integration for a real-time interactive VR ball-pit simulator that doesn’t require any additional tracking devices or sensors.

Philip Krejov received a PhD in Computer Vision and a BEng (Hons) degree in Electronic Engineering from the University of Surrey, United Kingdom. On graduation he was awarded the prize for best final year dissertation. Philip is a member of the CTO office at the RealSense group at Intel, conducting research in the field of Computer Vision with the focus of improving HCI. More specifically, he is developing tooling and architectures for capturing, sanitizing and learning from visual data. He has presented on many occasions, including a press conference held at the Royal Society, London. Philip has also published several international papers regarding hand pose estimation and novel methods for human computer interaction.