REINFORCEMENT LEARNING

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

March 01, 2021

Abstract

Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL algorithms can be fairly complex due to the separate dynamics modeling and the subsequent planning algorithm, and as a result, they often possess tens of hyperparameters and architectural choices. For this reason, MBRL typically requires significant human expertise before it can be applied to new problems and domains. To alleviate this problem, we propose to use automatic hyperparameter optimization (HPO). We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts. In addition, we show that tuning of several MBRL hyperparameters dynamically, i.e. during the training itself, further improves the performance compared to using static hyperparameters which are kept fixed for the whole training. Finally, our experiments provide valuable insights into the effects of several hyperparameters, such as plan horizon or learning rate and their influence on the stability of training and resulting rewards.

Download the Paper

AUTHORS

Written by

Baohe Zhang

Raghu Rajan

Luis Pineda

Nathan Lambert

André Biedenkapp

Kurtland Chua

Frank Hutter

Roberto Calandra

Publisher

AISTATS

Research Topics

Reinforcement Learning

Related Publications

December 15, 2021

ROBOTICS

REINFORCEMENT LEARNING

Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning

Roberto Calandra, Nathan Owen Lambert, Albert Wilcox, Howard Zhang, Kristofer S. J. Pister

December 15, 2021

December 05, 2021

REINFORCEMENT LEARNING

Local Differential Privacy for Regret Minimization in Reinforcement Learning

Evrard Garcelon, Vianney Perchet, Ciara Pike-Burke, Matteo Pirotta

December 05, 2021

December 05, 2021

REINFORCEMENT LEARNING

Hierarchical Skills for Efficient Exploration

Jonas Gehring, Gabriel Synnaeve, andreas krause, Nicolas Usunier

December 05, 2021

November 12, 2021

THEORY

REINFORCEMENT LEARNING

Bandits with Knapsacks beyond the Worst-Case Analysis

Karthik Abinav Sankararaman, Aleksandrs Slivkins

November 12, 2021