Alexei works on natural language processing and speech problems at Facebook AI Research, with a special interest in self-supervised learning across different domains, as well as contributing and maintaining open-source toolkits such as fairseq. Alexei holds a Bachelor of Science degree from the University of Toronto and has previously held various roles in the fintech industry and on the search team at Facebook London.
December 23, 2020
Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight…
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, Michael Auli,
December 23, 2020
December 23, 2020
We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output…
December 23, 2020
December 23, 2020
We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each…
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli,
December 23, 2020