The QV Family Compared to Other Reinforcement Learning Algorithms

M.A. Wiering; H.P. van Hasselt

The QV Family Compared to Other Reinforcement Learning Algorithms

M.A. Wiering, H.P. van Hasselt

Dept of Information and Computing Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

This paper describes several new online model-free reinforcement learning (RL) algorithms. We designed three new reinforcement algorithms, namely: QV2, QVMAX, and QVMAX2, that are all based on the QV-learning algorithm, but in contrary to QV-learning, QVMAX and QVMAX2 are off-policy RL algorithms and QV2 is a new on-policy RL algorithm. We experimentally compare these algorithms to a large number of different RL algorithms, namely: Q-learning, Sarsa, R-learning, Actor-Critic, QV-learning, and ACLA. We show experiments on five maze problems of varying complexity. Furthermore, we show experimental results on the cart pole balancing problem. The results show that for different problems, there can be large performance differences between the different algorithms, and that there is not a single RL algorithm that always performs best, although on average QV-learning scores highest.

Original language	Undefined/Unknown
Title of host publication	Proceedings of the IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009)
Publisher	IEEE
Pages	101-108
Number of pages	8
Publication status	Published - 30 Mar 2009

Bibliographical note

IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009)

Access to Document

http://people.cs.uu.nl/hado/papers/The_QV_Family_Compared_to_Other_Reinforcement_Learning_Algorithms.pdf

Cite this

@inproceedings{58a43d70974a44ad852a3a133281af99,

title = "The QV Family Compared to Other Reinforcement Learning Algorithms",

abstract = "This paper describes several new online model-free reinforcement learning (RL) algorithms. We designed three new reinforcement algorithms, namely: QV2, QVMAX, and QVMAX2, that are all based on the QV-learning algorithm, but in contrary to QV-learning, QVMAX and QVMAX2 are off-policy RL algorithms and QV2 is a new on-policy RL algorithm. We experimentally compare these algorithms to a large number of different RL algorithms, namely: Q-learning, Sarsa, R-learning, Actor-Critic, QV-learning, and ACLA. We show experiments on five maze problems of varying complexity. Furthermore, we show experimental results on the cart pole balancing problem. The results show that for different problems, there can be large performance differences between the different algorithms, and that there is not a single RL algorithm that always performs best, although on average QV-learning scores highest.",

author = "M.A. Wiering and {van Hasselt}, H.P.",

note = "IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009)",

year = "2009",

month = mar,

day = "30",

language = "Undefined/Unknown",

pages = "101--108",

booktitle = "Proceedings of the IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009)",

publisher = "IEEE",

address = "United States",

}

TY - GEN

T1 - The QV Family Compared to Other Reinforcement Learning Algorithms

AU - Wiering, M.A.

AU - van Hasselt, H.P.

N1 - IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009)

PY - 2009/3/30

Y1 - 2009/3/30

N2 - This paper describes several new online model-free reinforcement learning (RL) algorithms. We designed three new reinforcement algorithms, namely: QV2, QVMAX, and QVMAX2, that are all based on the QV-learning algorithm, but in contrary to QV-learning, QVMAX and QVMAX2 are off-policy RL algorithms and QV2 is a new on-policy RL algorithm. We experimentally compare these algorithms to a large number of different RL algorithms, namely: Q-learning, Sarsa, R-learning, Actor-Critic, QV-learning, and ACLA. We show experiments on five maze problems of varying complexity. Furthermore, we show experimental results on the cart pole balancing problem. The results show that for different problems, there can be large performance differences between the different algorithms, and that there is not a single RL algorithm that always performs best, although on average QV-learning scores highest.

AB - This paper describes several new online model-free reinforcement learning (RL) algorithms. We designed three new reinforcement algorithms, namely: QV2, QVMAX, and QVMAX2, that are all based on the QV-learning algorithm, but in contrary to QV-learning, QVMAX and QVMAX2 are off-policy RL algorithms and QV2 is a new on-policy RL algorithm. We experimentally compare these algorithms to a large number of different RL algorithms, namely: Q-learning, Sarsa, R-learning, Actor-Critic, QV-learning, and ACLA. We show experiments on five maze problems of varying complexity. Furthermore, we show experimental results on the cart pole balancing problem. The results show that for different problems, there can be large performance differences between the different algorithms, and that there is not a single RL algorithm that always performs best, although on average QV-learning scores highest.

M3 - Conference contribution

SP - 101

EP - 108

BT - Proceedings of the IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009)

PB - IEEE

ER -