November 08, 2021
Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 12.2% in unsupervised code translation, and 5.3% in natural language code search. Incidentally, we found that our pre-trained model is able to deobfuscate fully obfuscated source files, and to suggest descriptive variable names.
Publisher
Neurips
December 06, 2021
Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Shubham Muttepawar, Edward Wang (AI Infra), Sara Zhang, David Adkins, Orion Reblitz-Richardson
December 06, 2021
December 06, 2021
Yinglong Xia
December 06, 2021
December 06, 2021
Hongyu Gong, Yun Tang, Juan Miguel Pino, Xian Li
December 06, 2021
December 06, 2021
Weizhe Hua, Yichi Zhang, Chuan Guo, Zhiru Zhang, Edward Suh
December 06, 2021