Alexei Baevski
Alexei works on natural language processing and speech problems at Facebook AI Research, with a special interest in self-supervised learning across different domains, as well as contributing and maintaining open-source toolkits such as fairseq. Alexei holds a Bachelor of Science degree from the University of Toronto and has previously held various roles in the fintech industry and on the search team at Facebook London.
Alexei's Work
Alexei's Publications
December 23, 2020
RESEARCH
COMPUTER VISION
Pay less attention with Lightweight and Dynamic Convolutions
Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight…
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, Michael Auli,
December 23, 2020
December 23, 2020
RESEARCH
NLP
Adaptive Input Representations for Neural Language Modeling
We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output…
December 23, 2020
December 23, 2020
RESEARCH
NLP
Cloze-driven Pretraining of Self-attention Networks
We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each…
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli,
December 23, 2020
