Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabi

Bram Grooten; Jelle Wemmenhove; Maurice Poot; Jim Portegies

Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabi

Bram Grooten
, Jelle Wemmenhove
, Maurice Poot
, Jim Portegies

Eindhoven University of Technology

Research output: Contribution to conference › Paper › Academic

Abstract

In pursuit of enhanced multi-agent collaboration, we analyze several on-policy deep reinforcement learning algorithms in the recently published Hanabi benchmark. Our research suggests a perhaps counter-intuitive finding, where Proximal Policy Optimization (PPO) is outperformed by Vanilla Policy Gradient over multiple random seeds in a simplified environment of the multi-agent cooperative card game. In our analysis of this behavior we look into Hanabi-specific metrics and hypothesize a reason for PPO’s plateau. In addition, we provide proofs for the maximum length of a perfect game (71 turns) and any game (89 turns). Our code can be found at: https://github.com/bramgrooten/DeepRL-for-Hanabi.

Original language	English
Publication status	Published - 2022
Externally published	Yes
Event	Adaptive and Learning Agents Workshop, ALA 2022 at AAMAS 2022 - Auckland, New Zealand Duration: 9 May 2022 → 10 May 2022

Conference

Conference	Adaptive and Learning Agents Workshop, ALA 2022 at AAMAS 2022
Country/Territory	New Zealand
City	Auckland
Period	9/05/22 → 10/05/22

Bibliographical note

Publisher Copyright:
© 2022 ALA 2022 - Adaptive and Learning Agents Workshop at AAMAS 2022. All rights reserved.

Keywords

Deep reinforcement learning
Hanabi
multi-agent collaboration
PPO
Vanilla Policy Gradient

Cite this

@conference{cbf6b76c79bd4645bb09f8d303467ffd,

title = "Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabi",

abstract = "In pursuit of enhanced multi-agent collaboration, we analyze several on-policy deep reinforcement learning algorithms in the recently published Hanabi benchmark. Our research suggests a perhaps counter-intuitive finding, where Proximal Policy Optimization (PPO) is outperformed by Vanilla Policy Gradient over multiple random seeds in a simplified environment of the multi-agent cooperative card game. In our analysis of this behavior we look into Hanabi-specific metrics and hypothesize a reason for PPO{\textquoteright}s plateau. In addition, we provide proofs for the maximum length of a perfect game (71 turns) and any game (89 turns). Our code can be found at: https://github.com/bramgrooten/DeepRL-for-Hanabi.",

keywords = "Deep reinforcement learning, Hanabi, multi-agent collaboration, PPO, Vanilla Policy Gradient",

author = "Bram Grooten and Jelle Wemmenhove and Maurice Poot and Jim Portegies",

note = "Publisher Copyright: {\textcopyright} 2022 ALA 2022 - Adaptive and Learning Agents Workshop at AAMAS 2022. All rights reserved.; Adaptive and Learning Agents Workshop, ALA 2022 at AAMAS 2022 ; Conference date: 09-05-2022 Through 10-05-2022",

year = "2022",

language = "English",

}

TY - CONF

T1 - Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabi

AU - Grooten, Bram

AU - Wemmenhove, Jelle

AU - Poot, Maurice

AU - Portegies, Jim

PY - 2022

Y1 - 2022

N2 - In pursuit of enhanced multi-agent collaboration, we analyze several on-policy deep reinforcement learning algorithms in the recently published Hanabi benchmark. Our research suggests a perhaps counter-intuitive finding, where Proximal Policy Optimization (PPO) is outperformed by Vanilla Policy Gradient over multiple random seeds in a simplified environment of the multi-agent cooperative card game. In our analysis of this behavior we look into Hanabi-specific metrics and hypothesize a reason for PPO’s plateau. In addition, we provide proofs for the maximum length of a perfect game (71 turns) and any game (89 turns). Our code can be found at: https://github.com/bramgrooten/DeepRL-for-Hanabi.

AB - In pursuit of enhanced multi-agent collaboration, we analyze several on-policy deep reinforcement learning algorithms in the recently published Hanabi benchmark. Our research suggests a perhaps counter-intuitive finding, where Proximal Policy Optimization (PPO) is outperformed by Vanilla Policy Gradient over multiple random seeds in a simplified environment of the multi-agent cooperative card game. In our analysis of this behavior we look into Hanabi-specific metrics and hypothesize a reason for PPO’s plateau. In addition, we provide proofs for the maximum length of a perfect game (71 turns) and any game (89 turns). Our code can be found at: https://github.com/bramgrooten/DeepRL-for-Hanabi.

KW - Deep reinforcement learning

KW - Hanabi

KW - multi-agent collaboration

KW - PPO

KW - Vanilla Policy Gradient

UR - https://www.scopus.com/pages/publications/85154057587

M3 - Paper

AN - SCOPUS:85154057587

T2 - Adaptive and Learning Agents Workshop, ALA 2022 at AAMAS 2022

Y2 - 9 May 2022 through 10 May 2022

ER -

Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabi

Abstract

Conference

Bibliographical note

Keywords

Other files and links

Fingerprint

Cite this