Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

Giovanni Varricchione; Natasha Alechina; Mehdi Dastani; Brian Logan

doi:10.1007/978-3-031-43264-4_21

Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

Giovanni Varricchione^*, Natasha Alechina, Mehdi Dastani, Brian Logan

^*Corresponding author for this work

Sub Intelligent Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. However, current work assumes the multi-agent reward machine to be given. In this paper, we show how reward machines for team tasks can be synthesised automatically from an Alternating-Time Temporal Logic specification of the desired team behaviour and a high-level abstraction of the agents’ environment. We present results suggesting that our automated approach has comparable, if not better, sample efficiency than reward machines generated by hand for multi-agent tasks.

Original language	English
Title of host publication	Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings
Subtitle of host publication	20th European Conference, EUMAS 2023, Naples, Italy, September 14–15, 2023, Proceedings
Editors	Vadim Malvone, Aniello Murano
Pages	328–344
Number of pages	17
ISBN (Electronic)	978-3-031-43264-4
DOIs	https://doi.org/10.1007/978-3-031-43264-4_21
Publication status	Published - 7 Sept 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14282 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

multi-agent reinforcement learning
reward machines
automatic synthesis

Access to Document

10.1007/978-3-031-43264-4_21

978-3-031-43264-4_21Final published version, 668 KBLicence: Taverne

Cite this

Varricchione, G., Alechina, N., Dastani, M., & Logan, B. (2023). Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning. In V. Malvone, & A. Murano (Eds.), Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings: 20th European Conference, EUMAS 2023, Naples, Italy, September 14–15, 2023, Proceedings (pp. 328–344). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14282 LNAI). https://doi.org/10.1007/978-3-031-43264-4_21

Varricchione, Giovanni ; Alechina, Natasha ; Dastani, Mehdi et al. / Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning. Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings: 20th European Conference, EUMAS 2023, Naples, Italy, September 14–15, 2023, Proceedings. editor / Vadim Malvone ; Aniello Murano. 2023. pp. 328–344 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a4034053bb60492fbbfd8d5de05ac947,

title = "Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning",

abstract = "Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. However, current work assumes the multi-agent reward machine to be given. In this paper, we show how reward machines for team tasks can be synthesised automatically from an Alternating-Time Temporal Logic specification of the desired team behaviour and a high-level abstraction of the agents{\textquoteright} environment. We present results suggesting that our automated approach has comparable, if not better, sample efficiency than reward machines generated by hand for multi-agent tasks.",

keywords = "multi-agent reinforcement learning, reward machines, automatic synthesis",

author = "Giovanni Varricchione and Natasha Alechina and Mehdi Dastani and Brian Logan",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.",

year = "2023",

month = sep,

day = "7",

doi = "10.1007/978-3-031-43264-4\_21",

language = "English",

isbn = "978-3-031-43263-7",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "328–344",

editor = "Vadim Malvone and Aniello Murano",

booktitle = "Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings",

}

Varricchione, G , Alechina, N , Dastani, M & Logan, B 2023, Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning. in V Malvone & A Murano (eds), Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings: 20th European Conference, EUMAS 2023, Naples, Italy, September 14–15, 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14282 LNAI, pp. 328–344. https://doi.org/10.1007/978-3-031-43264-4_21

Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning. / Varricchione, Giovanni ; Alechina, Natasha ; Dastani, Mehdi et al.
Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings: 20th European Conference, EUMAS 2023, Naples, Italy, September 14–15, 2023, Proceedings. ed. / Vadim Malvone; Aniello Murano. 2023. p. 328–344 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14282 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

AU - Varricchione, Giovanni

AU - Alechina, Natasha

AU - Dastani, Mehdi

AU - Logan, Brian

PY - 2023/9/7

Y1 - 2023/9/7

N2 - Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. However, current work assumes the multi-agent reward machine to be given. In this paper, we show how reward machines for team tasks can be synthesised automatically from an Alternating-Time Temporal Logic specification of the desired team behaviour and a high-level abstraction of the agents’ environment. We present results suggesting that our automated approach has comparable, if not better, sample efficiency than reward machines generated by hand for multi-agent tasks.

AB - Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. However, current work assumes the multi-agent reward machine to be given. In this paper, we show how reward machines for team tasks can be synthesised automatically from an Alternating-Time Temporal Logic specification of the desired team behaviour and a high-level abstraction of the agents’ environment. We present results suggesting that our automated approach has comparable, if not better, sample efficiency than reward machines generated by hand for multi-agent tasks.

KW - multi-agent reinforcement learning

KW - reward machines

KW - automatic synthesis

UR - https://www.scopus.com/pages/publications/85171998898

U2 - 10.1007/978-3-031-43264-4_21

DO - 10.1007/978-3-031-43264-4_21

M3 - Conference contribution

SN - 978-3-031-43263-7

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 328

EP - 344

BT - Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings

A2 - Malvone, Vadim

A2 - Murano, Aniello

ER -

Varricchione G , Alechina N , Dastani M , Logan B. Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning. In Malvone V, Murano A, editors, Multi-Agent Systems - 20th European Conference, EUMAS 2023, Proceedings: 20th European Conference, EUMAS 2023, Naples, Italy, September 14–15, 2023, Proceedings. 2023. p. 328–344. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-43264-4_21

Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

Abstract

Publication series

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this