Bootstrapped Policy Learning for Task-oriented Dialogue through Goal Shaping

Yangyang Zhao, Ben Niu, Mehdi Dastani, Shihan Wang

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Reinforcement learning shows promise in optimizing dialogue policies, but addressing the challenge of reward sparsity remains crucial. While curriculum learning offers a practical solution by strategically training policies from simple to complex, it hinges on the assumption of a gradual increase in goal difficulty to ensure a smooth knowledge transition across varied complexities. In complex dialogue environments without intermediate goals, achieving seamless knowledge transitions becomes tricky. This paper proposes a novel Bootstrapped Policy Learning (BPL) framework, which adaptively tailors progressively challenging subgoal curriculum for each complex goal through goal shaping, ensuring a smooth knowledge transition. Goal shaping involves goal decomposition and evolution, decomposing complex goals into subgoals with solvable maximum difficulty and progressively increasing difficulty as the policy improves. Moreover, to enhance BPL`s adaptability across various environments, we explore various combinations of goal decomposition and evolution within BPL, and identify two universal curriculum patterns that remain effective across different dialogue environments, independent of specific environmental constraints. By integrating the summarized curriculum patterns, our BPL has exhibited efficacy and versatility across four publicly available datasets with different difficulty levels.
Original languageEnglish
Title of host publicationProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Place of PublicationMiami, Florida, USA
PublisherAssociation for Computational Linguistics (ACL)
Pages4566-4580
DOIs
Publication statusPublished - 1 Nov 2024

Fingerprint

Dive into the research topics of 'Bootstrapped Policy Learning for Task-oriented Dialogue through Goal Shaping'. Together they form a unique fingerprint.

Cite this