Using deep neural networks for accurate hand-tracking on Oculus Quest

What it is:

Researchers and engineers from Facebook Reality Labs and Oculus have developed what is, as of today, the only fully articulated hand-tracking system for VR that relies entirely on monochrome cameras. The system does not use active depth-sensing technology or any additional equipment (such as instrumented gloves). We will deploy this technology as a software update for Oculus Quest, the cable-free, stand-alone VR headset that is now available to consumers.

By using Quest’s four cameras in conjunction with new techniques in deep learning and model-based tracking, we achieve a larger interaction volume for hand-tracking than depth-based solutions do, and we do it at a fraction of the size, weight, power, and cost. Processing is done entirely on-device, and the system is optimized to support gestures for interaction, such as pointing and pinch to select.

How it works:

Deep neural networks are used to predict the location of a person’s hands as well as landmarks, such as joints of the hands. These landmarks are then used to reconstruct a 26 degree-of-freedom pose of the person’s hands and fingers. The result is a 3D model that includes the configuration and surface geometry of the hand. APIs will enable developers to use these 3D models to enable new interaction mechanics in their apps or to drive a user interface.

We use a novel tracking architecture that produces accurate, low-jitter estimates of hand pose robustly across a wide range of environments, and an efficient, quantized neural network framework that enables real-time hand-tracking on a mobile processor, without compromising resources dedicated to user applications.

Why it matters:

Precise hand-tracking will unlock a range of new experiences as well as reduce friction for current experiences on Quest. People could be able to pause a movie in VR with just a gesture, for example, and express themselves more naturally in social games. In enterprise applications, an instructor could lead a VR-based training class without having to maintain a fleet of paired, charged controllers.

More broadly, hand-tracking will make VR feel more natural and intuitive, and help developers create new ways for people to interact in virtual worlds. Facebook Reality Labs and Oculus will look for ways to build on this core technology to enhance other AR/VR experiences into the future.