The wav2vec model is trained by predicting speech units for masked parts of speech audio. It learns basic units that are 25ms long to enable learning of high-level contextualized representations. This enables us to build speech recognition systems that can outperform the best semi-supervised methods, even with 100 times less labeled training data. Our model learns a set of speech units, which are shorter than phonemes, to describe the speech audio sequence. Since this set is finite, the units encourage the model to focus on the most important factors to represent the speech audio.

Wav2vec learns from recorded speech audio and unpaired text, lessening the need for transcriptions. The self-supervised model segments the voice recording into speech units that loosely correspond to individual sounds. Training a generative adversarial network consisting of a generator and a discriminator network helps the model learn to recognize the words in the audio recording. Reducing our dependence on annotated data by using self-supervised learning is an important part of expanding access to speech recognition tools to many more people around the world.

Alexei Baevski, Steffen Scheider, Wei-Ning Hsu, Alexis Conneau, Henry Zhou, Ronan Collobert, Abdelrahman Mohamed, Michael Auli