RESEARCH

COMPUTER VISION

EGO-TOPO: Environment Affordances from Egocentric Video

June 14, 2020

Abstract

First-person video naturally brings the use of a physical environment to the forefront, since it shows the camera wearer interacting fluidly in a space based on his intentions. However, current methods largely separate the observed actions from the persistent space itself. We introduce a model for environment affordances that is learned directly from egocentric video. The main idea is to gain a human-centric model of a physical space (such as a kitchen) that captures (1) the primary spatial zones of interaction and (2) the likely activities they support. Our approach decomposes a space into a topological map derived from first-person activity, organizing an ego-video into a series of visits to the different zones. Further, we show how to link zones across multiple related environments (e.g., from videos of multiple kitchens) to obtain a consolidated representation of environment functionality. On EPIC-Kitchens and EGTEA+, we demonstrate our approach for learning scene affordances and anticipating future actions in long-form video. Project page: http://vision.cs.utexas.edu/projects/ego-topo/

Download the Paper

AUTHORS

Written by

Tushar Nagarajan

Yanghao Li

Christoph Feichtenhofer

Kristen Grauman

Publisher

Conference on Computer Vision and Pattern Recognition (CVPR)

Research Topics

Computer Vision

Recent Publications

January 09, 2021

COMPUTER VISION

Tarsier: Evolving Noise Injection in Super-Resolution GANs

Super-resolution aims at increasing the resolution and level of detail within an image.…

Baptiste Roziere, Nathanaël Carraz Rakotonirina, Vlad Hosu, Andry Rasoanaivo, Hanhe Lin, Camille Couprie, Olivier Teytaud

January 09, 2021

December 10, 2020

COMPUTER VISION

Better Set Representations For Relational Reasoning

Incorporating relational reasoning into neural networks has greatly expanded their capabilities and scope.…

Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser-Nam Lim, Austin Benson

December 10, 2020

December 06, 2020

COMPUTER VISION

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

The combination of deep reinforcement learning and search at both training and test time is a powerful paradigm that has led to a number of successes in single-agent…

Noam Brown, Anton Bakhtin, Adam Lerer, Qucheng Gong

December 06, 2020

December 06, 2020

COMPUTER VISION

Riemannian Continuous Normalizing Flows

Normalizing flows have shown great promise for modelling flexible probability distributions in a computationally tractable way. However, whilst data is often naturally described on…

Emile Mathieu, Maximilian Nickel

December 06, 2020

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.