Publications

Publication

April 27, 2020

The Early Phase of Neural Network Training | Facebook AI Research

Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent…

Jonathan Frankle, David J. Schwab, Ari Morcos

April 27, 2020

Download the Paper

Publication

April 26, 2020

SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum | Facebook AI Research

Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local…

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat

April 26, 2020

Read the Paper

Publication

April 25, 2020

Permutation Equivariant Models for Compositional Generalization in Language | Facebook AI Research

Humans understand novel sentences by composing meanings and roles of core language components. In contrast, neural network models for natural language modeling fail when such compositional generalization is required. The main contribution of…

Jonathan Gordon, David Lopez-Paz, Marco Baroni, Diane Bouchacourt

April 25, 2020

Read the Paper

Publication

April 25, 2020

And the bit goes down: Revisiting the quantization of neural networks | Facebook AI Research

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather…

Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, Hervé Jégou

April 25, 2020

Read the Paper

Publication

April 20, 2020

Direction of Arrival Estimation in Highly Reverberant Environments Using Soft Time-Frequency Mask | Facebook AI Research

A recent approach to improving the robustness of sound localization in reverberant environments is based on pre-selection of time-frequency pixels that are dominated by direct sound. This approach is equivalent to applying a binary…

Vladimir Tourbabin, Jacob Donley, Boaz Rafaely, Ravish Mehra

April 20, 2020

Download the Paper

Publication

April 09, 2020

Environment-aware reconfigurable noise suppression | Facebook AI Research

The paper proposes an efficient, robust, and reconfigurable technique to suppress various types of noises for any sampling rate. The theoretical analyses, subjective and objective test results show that the proposed noise suppression (NS)…

Jun Yang, Joshua Bingham

April 09, 2020

Download the Paper

Publication

March 02, 2020

SYSTEMS RESEARCH

Federated Optimization in Heterogenous Networks | Facebook AI Research

Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network…

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith

March 02, 2020

Download the Paper

Publication

February 21, 2020

Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection | Facebook AI Research

Recommendation is a prevalent application of machine learning that affects many users; therefore, it is important for recommender models to be accurate and interpretable. In this work, we propose a method to both interpret and augment the…

Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, Yan Liu

February 21, 2020

Read the Paper

Publication

February 21, 2020

CoPhy: Counterfactual Learning of Physical Dynamics | Facebook AI Research

Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We develop the CoPhy benchmark…

Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, Christian Wolf

February 21, 2020

Download the Paper

Publication

February 17, 2020

The Architectural Implications of Facebook’s DNN-based Personalized Recommendation | Facebook AI Research

The widespread application of deep learning has changed the landscape of computation in data centers. In particular, personalized recommendation for content ranking is now largely accomplished using deep neural networks. However, despite their…

Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Mark Hempstead, Bill Jia, Hsien-Hsin S. Lee, Andrey Malevich, Dheevatsa Mudigere, Mikhail Smelyanskiy, Liang Xiong, Xuan Zhang

February 17, 2020

Read the Paper

Publication

February 14, 2020

Neural Machine Translation with Byte-Level Subwords | Facebook AI Research

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily…

Changhan Wang, Kyunghyun Cho, Jiatao Gu

February 14, 2020

Read the Paper

Publication

February 14, 2020

Fair Resource Allocation in Federated Learning | Facebook AI Research

Federated learning involves training statistical models in massive, heterogeneous networks. Naively minimizing an aggregate loss function in such a network may disproportionately advantage or disadvantage some of the devices. In this work, we…