Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning

Changxi Zhu; Mehdi Dastani; Shihan Wang

doi:10.5555/3709347.3744030

Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning

Sub Intelligent Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

In decentralized multi-agent deep reinforcement learning (MADRL), communication can help agents to gain a better understanding of the environment to better coordinate their behaviors. Nevertheless, communication may involve uncertainty, which potentially introduces variance to the learning of decentralized agents. In this extended abstract, we report on our research that focuses on a specific decentralized MADRL setting with communication and a theoretical analysis to study the variance caused by communication in policy gradients. We argue for modular techniques to reduce the variance in policy gradients during training. We show a pseudo algorithm to illustrate the integration of the modular techniques into existing decentralized MADRL with communication methods.

Original language	English
Title of host publication	Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025
Editors	Yevgeniy Vorobeychik, Sanmay Das, Ann Nowe
Publisher	International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
Pages	2841-2843
Number of pages	3
ISBN (Electronic)	9798400714269
DOIs	https://doi.org/10.5555/3709347.3744030
Publication status	Published - 5 Jun 2025
Event	24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025 - Detroit, United States Duration: 19 May 2025 → 23 May 2025

Publication series

Name	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
ISSN (Print)	1548-8403
ISSN (Electronic)	1558-2914

Conference

Conference	24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025
Country/Territory	United States
City	Detroit
Period	19/05/25 → 23/05/25

Bibliographical note

Publisher Copyright:
© 2025 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org).

Keywords

Communication
Multi-agent Deep Reinforcement Learning
Variance Reduction

Access to Document

10.5555/3709347.3744030Licence: CC BY

Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement LearningFinal published version, 1.03 MBLicence: CC BY

Cite this

Zhu, C., Dastani, M., & Wang, S. (2025). Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning. In Y. Vorobeychik, S. Das, & A. Nowe (Eds.), Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025 (pp. 2841-2843). (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). https://doi.org/10.5555/3709347.3744030

Zhu, Changxi ; Dastani, Mehdi ; Wang, Shihan. / Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning. Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025. editor / Yevgeniy Vorobeychik ; Sanmay Das ; Ann Nowe. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2025. pp. 2841-2843 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS).

@inproceedings{2b299fa8649f42d8ad9df687dab66944,

title = "Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning",

abstract = "In decentralized multi-agent deep reinforcement learning (MADRL), communication can help agents to gain a better understanding of the environment to better coordinate their behaviors. Nevertheless, communication may involve uncertainty, which potentially introduces variance to the learning of decentralized agents. In this extended abstract, we report on our research that focuses on a specific decentralized MADRL setting with communication and a theoretical analysis to study the variance caused by communication in policy gradients. We argue for modular techniques to reduce the variance in policy gradients during training. We show a pseudo algorithm to illustrate the integration of the modular techniques into existing decentralized MADRL with communication methods.",

keywords = "Communication, Multi-agent Deep Reinforcement Learning, Variance Reduction",

author = "Changxi Zhu and Mehdi Dastani and Shihan Wang",

note = "Publisher Copyright: {\textcopyright} 2025 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org).; 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025 ; Conference date: 19-05-2025 Through 23-05-2025",

year = "2025",

month = jun,

day = "5",

doi = "10.5555/3709347.3744030",

language = "English",

series = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",

publisher = "International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)",

pages = "2841--2843",

editor = "Yevgeniy Vorobeychik and Sanmay Das and Ann Nowe",

booktitle = "Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025",

}

Zhu, C , Dastani, M & Wang, S 2025, Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning. in Y Vorobeychik, S Das & A Nowe (eds), Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), pp. 2841-2843, 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025, Detroit, United States, 19/05/25. https://doi.org/10.5555/3709347.3744030

Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning. / Zhu, Changxi ; Dastani, Mehdi ; Wang, Shihan.
Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025. ed. / Yevgeniy Vorobeychik; Sanmay Das; Ann Nowe. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2025. p. 2841-2843 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning

AU - Zhu, Changxi

AU - Dastani, Mehdi

AU - Wang, Shihan

PY - 2025/6/5

Y1 - 2025/6/5

N2 - In decentralized multi-agent deep reinforcement learning (MADRL), communication can help agents to gain a better understanding of the environment to better coordinate their behaviors. Nevertheless, communication may involve uncertainty, which potentially introduces variance to the learning of decentralized agents. In this extended abstract, we report on our research that focuses on a specific decentralized MADRL setting with communication and a theoretical analysis to study the variance caused by communication in policy gradients. We argue for modular techniques to reduce the variance in policy gradients during training. We show a pseudo algorithm to illustrate the integration of the modular techniques into existing decentralized MADRL with communication methods.

AB - In decentralized multi-agent deep reinforcement learning (MADRL), communication can help agents to gain a better understanding of the environment to better coordinate their behaviors. Nevertheless, communication may involve uncertainty, which potentially introduces variance to the learning of decentralized agents. In this extended abstract, we report on our research that focuses on a specific decentralized MADRL setting with communication and a theoretical analysis to study the variance caused by communication in policy gradients. We argue for modular techniques to reduce the variance in policy gradients during training. We show a pseudo algorithm to illustrate the integration of the modular techniques into existing decentralized MADRL with communication methods.

KW - Communication

KW - Multi-agent Deep Reinforcement Learning

KW - Variance Reduction

UR - https://www.scopus.com/pages/publications/105009817245

U2 - 10.5555/3709347.3744030

DO - 10.5555/3709347.3744030

M3 - Conference contribution

AN - SCOPUS:105009817245

T3 - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

SP - 2841

EP - 2843

BT - Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025

A2 - Vorobeychik, Yevgeniy

A2 - Das, Sanmay

A2 - Nowe, Ann

PB - International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)

T2 - 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025

Y2 - 19 May 2025 through 23 May 2025

ER -

Zhu C , Dastani M , Wang S. Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning. In Vorobeychik Y, Das S, Nowe A, editors, Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). 2025. p. 2841-2843. (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS). doi: 10.5555/3709347.3744030

Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this