ML APPLICATIONS

COMPUTER VISION

Graph Structure of Neural Networks

August 14, 2020

Abstract

Neural networks are often represented as graphs of connections between neurons. However, despite their wide use, there is currently little understanding of the relationship between the graph structure of the neural network and its predictive performance. Here we systematically investigate how does the graph structure of neural networks affect their predictive performance. To this end, we develop a novel graph-based representation of neural networks called relational graph, where layers of neural network computation correspond to rounds of message exchange along the graph structure. Using this representation we show that: (1) a “sweet spot” of relational graphs leads to neural networks with significantly improved predictive performance; (2) neural network’s performance is approximately a smooth function of the clustering coefficient and average path length of its relational graph; (3) our findings are consistent across many different tasks and datasets; (4) the sweet spot can be identified efficiently; (5) top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks. Our work opens new directions for the design of neural architectures and the understanding on neural networks in general.

Download the Paper

AUTHORS

Written by

Jiaxuan You

Jure Leskovec

Kaiming He

Saining Xie

Publisher

International Conference on Machine Learning (ICML)

Related Publications

July 03, 2019

SPEECH & AUDIO

NLP

Linguistic generalization and compositionality in modern artificial neural networks | Facebook AI Research

In the last decade, deep artificial neural networks have achieved astounding performance in many natural language processing tasks. Given the high productivity of language, these models must possess effective generalization abilities. It is…

Marco Baroni

July 03, 2019

May 06, 2019

HUMAN & MACHINE INTELLIGENCE

Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies | Facebook AI Research

In this work we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks. The agent is split into a low-level and a high-level policy. The low-level policy only accesses internal,…

Kenneth Marino, Abhinav Gupta, Rob Fergus, Arthur Szlam

May 06, 2019

April 24, 2017

HUMAN & MACHINE INTELLIGENCE

COMPUTER VISION

Episodic Exploration for Deep Deterministic Policies for StarCraft Micro-Management | Facebook AI Research

We consider scenarios from the real-time strategy game StarCraft as benchmarks for reinforcement learning algorithms. We focus on micromanagement, that is, the short-term, low-level control of team members during a battle. We propose several…

Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, Soumith Chintala

April 24, 2017

December 03, 2018

HUMAN & MACHINE INTELLIGENCE

SPEECH & AUDIO

Forward Modeling for Partial Observation Strategy Games | Facebook AI Research

We formulate the problem of defogging as state estimation and future state prediction from previous, partial observations in the context of real-time strategy games. We propose to employ encoder-decoder neural networks for this task, and…

Gabriel Synnaeve, Zeming Lin, Jonas Gehring, Dan Gant, Vegard Mella, Vasil Khalidov, Nicolas Carion, Nicolas Usunier

December 03, 2018

Help Us Pioneer The Future of AI

We share our open source frameworks, tools, libraries, and models for everything from research exploration to large-scale production deployment.