Alexei Baevski

Alexei works on natural language processing and speech problems at Facebook AI Research, with a special interest in self-supervised learning across different domains, as well as contributing and maintaining open-source toolkits such as fairseq. Alexei holds a Bachelor of Science degree from the University of Toronto and has previously held various roles in the fintech industry and on the search team at Facebook London.

Alexei's Publications

May 03, 2020

RESEARCH

COMPUTER VISION

Pay less attention with Lightweight and Dynamic Convolutions

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight…

Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, Michael Auli,

May 03, 2020

May 03, 2020

RESEARCH

NLP

Adaptive Input Representations for Neural Language Modeling

We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output…

Alexei Baevski, Michael Auli,

May 03, 2020

May 03, 2020

RESEARCH

NLP

Cloze-driven Pretraining of Self-attention Networks

We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each…

Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli,

May 03, 2020