SYSTEMS RESEARCH
RANKING & RECOMMENDATIONS
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
February 27, 2021
Abstract
The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models. Meanwhile, when training state-of-the-art personal recommendation models, which consume the highest number of compute cycles at our large-scale data centers, the use of GPUs came with various challenges due to having both compute-intensive and memory-intensive components.
GPU performance and efficiency of these recommendation models are largely affected by model architecture configurations such as dense and sparse features, MLP dimensions. Furthermore, these models often contain large embedding tables that do not fit into limited GPU memory. The goal of this paper is to explain the intricacies of using GPUs for training recommendation models, factors affecting hardware efficiency at scale, and learnings from a new scale-up GPU server design, Zion.
AUTHORS
Publisher
High Performance Computer Architecture (HPCA 2021)
Related Publications
SYSTEMS RESEARCH
Rethinking floating point for deep learning | Facebook AI Research
Reducing hardware overhead of neural networks for faster or lower power inference and training is an active area of research. Uniform quantization using integer multiply-add has been thoroughly investigated, which requires learning many…
Jeff Johnson
SYSTEMS RESEARCH
NLP
Fast Convolutional Nets With fbfft: A GPU Performance Evaluation | Facebook AI Research
We examine the performance profile of Convolutional Neural Network training on the current generation of NVIDIA Graphics Processing Units. We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA’s cuFFT…
Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, Yann LeCun
SYSTEMS RESEARCH
Federated Optimization in Heterogenous Networks | Facebook AI Research
Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network…
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith
RANKING & RECOMMENDATIONS
SYSTEMS RESEARCH
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models…
Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, Kim Hazelwood
