SYSTEMS RESEARCH

RANKING & RECOMMENDATIONS

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

February 27, 2021

Abstract

The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models. Meanwhile, when training state-of-the-art personal recommendation models, which consume the highest number of compute cycles at our large-scale data centers, the use of GPUs came with various challenges due to having both compute-intensive and memory-intensive components.

GPU performance and efficiency of these recommendation models are largely affected by model architecture configurations such as dense and sparse features, MLP dimensions. Furthermore, these models often contain large embedding tables that do not fit into limited GPU memory. The goal of this paper is to explain the intricacies of using GPUs for training recommendation models, factors affecting hardware efficiency at scale, and learnings from a new scale-up GPU server design, Zion.

Download the Paper

AUTHORS

Written by

Bilge Acun

Matthew Murphy

Xiaodong Wang

Jade Nie

Carole-Jean Wu

Kim Hazelwood

Publisher

High Performance Computer Architecture (HPCA 2021)

Research Topics

Ranking and Recommendations

Systems Research

Related Publications

June 22, 2015

SYSTEMS RESEARCH

NLP

Fast Convolutional Nets With fbfft: A GPU Performance Evaluation | Facebook AI Research

Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, Yann LeCun

June 22, 2015

March 02, 2020

SYSTEMS RESEARCH

Federated Optimization in Heterogenous Networks | Facebook AI Research

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith

March 02, 2020

September 01, 2020

SYSTEMS RESEARCH

ResiliNet: Failure-Resilient Inference in Distributed Neural Networks

Ashkan Yousefpour, Brian Q. Nguyen, Siddartha Devic, Guanhua Wang, Aboudy Kreidieh, Hans Lobel, Alexandre M. Bayen, Jason P. Jue

September 01, 2020

February 27, 2021

SYSTEMS RESEARCH

RANKING & RECOMMENDATIONS

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, Kim Hazelwood

February 27, 2021

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.