Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation

Ruizhe Li, Xiao Li, Guanyi Chen, Chenghua Lin

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.
Original languageEnglish
Title of host publicationProceedings of the 28th International Conference on Computational Linguistics
EditorsDonia Scott, Nuria Bel, Chengqing Zong
Place of PublicationBarcelona
PublisherInternational Committee on Computational Linguistics
Pages2381-2397
Number of pages17
DOIs
Publication statusPublished - 1 Dec 2020

Fingerprint

Dive into the research topics of 'Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation'. Together they form a unique fingerprint.

Cite this