Alexei Baevski

Alexei works on natural language processing and speech problems at Facebook AI Research, with a special interest in self-supervised learning across different domains, as well as contributing and maintaining open-source toolkits such as fairseq. Alexei holds a Bachelor of Science degree from the University of Toronto and has previously held various roles in the fintech industry and on the search team at Facebook London.

Alexei's Publications

June 03, 2019

RESEARCH

COMPUTER VISION

Pay less attention with Lightweight and Dynamic Convolutions

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight…

Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, Michael Auli,

June 03, 2019

May 06, 2019

RESEARCH

NLP

Adaptive Input Representations for Neural Language Modeling

We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output…

Alexei Baevski, Michael Auli,

May 06, 2019

November 07, 2019

RESEARCH

NLP

Cloze-driven Pretraining of Self-attention Networks

We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each…

Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli,

November 07, 2019