Publications

Publication

October 14, 2021

GRAPHICS

COMPUTER VISION

Ego4D: Around the World in 3,000 Hours of Egocentric Video

We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,025 hours of dailylife activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 855 unique camera wearers from 74 worldwide locations and 9 different countries. …

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Ramakrishnan, Fiona Ryan,Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, …

October 14, 2021

Read the Paper

Publication

COMPUTER VISION

LiRA: Learning Visual Speech Representations from Audio through Self-supervision

The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audio- visual self-supervised learning. Recent works have focused on each of these modalities separately, while others have attempted to model both simultaneously in a cross-modal fashion. How- ever, comparatively little attention has been given to leveraging one modality as a training objective to learn from the other. In this work, we propose Learning visual speech Representations from Audio via self-supervision (LiRA). Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech. We find that this pre-trained model can be leveraged towards word-level and sentence-level lip-reading through feature extraction and fine-tuning experiments. We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild (LRW) dataset and achieves state-of-the-art performance on Lip Reading Sentences 2 (LRS2) using only a fraction of the total labelled data.

Maja Pantic, Stavros Petridis, Bjoern Schuller, Pingchuan Ma, Rodrigo Schonburg Carrillo De Mira

Publication

October 01, 2021

COMPUTER VISION

CONVERSATIONAL AI

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn …

Hung Le, Chinnadhurai Sankar, Seungwhan Moon, Ahmad Beiram, Alborz Geramifard, Satwik Kottur

October 01, 2021

Read the Paper

Publication

September 27, 2021

RANKING & RECOMMENDATIONS

Transformers4Rec: Bridging the Gap between NLP and Sequential / Session-Based Recommendation

Much of the recent progress in sequential and session-based recommendation has been driven by improvements in model architecture and pretraining techniques originating in the field of Natural Language Processing. …

Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, Even Oldridge

September 27, 2021

Read the Paper

Publication

August 29, 2021

NLP

Many-Speakers Single Channel Speech Separation with Optimal Permutation Training

Single channel speech separation has experienced great progress in the last few years …

Shaked Dovrat,Eliya Nachmani, Lior Wolf

August 29, 2021

Read the Paper

Publication

August 09, 2021

COMPUTER VISION

GRAPHICS

Deep Relightable Appearance Models for Animatable Faces

We present a method for building high-fidelity animatable 3D face models that can be posed and rendered with novel lighting environments in real-time. …

Sai Bi, Stephen Lombardi,Shunsuke Saito, Tomas Simon, Shih-en Wei, Kevyn McPhail, Ravi Ramamoorthi, Yaser Sheikh, Jason Saragih

August 09, 2021

Read the Paper

Publication

August 09, 2021

GRAPHICS

COMPUTER VISION

Control Strategies for Physically Simulated Characters Performing Two-player Competitive Sports

In two-player competitive sports, such as boxing and fencing, athletes often demonstrate efficient and tactical movements during a competition …

Jungdam Won, Deepak Gopinath, Jessica Hodgins

August 09, 2021

Read the Paper

Publication

August 01, 2021

GRAPHICS

COMPUTER VISION

Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards?

Leaderboards are widely used in NLP and push the field forward. While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items …

Pedro Rodriguez, Joe Barrow, Alexander Hoyle, John P. Lalor, Robin Jia, Jordan Boyd-Graber

August 01, 2021

Read the Paper

Publication

August 01, 2021

GRAPHICS

COMPUTER VISION

Phase Transitions, Distance Functions, and Implicit Neural Representations

Representing surfaces as zero level sets of neural networks recently emerged as a powerful modeling paradigm, named Implicit Neural Representations (INRs) …