Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification

Dani Krebbers; Heysem Kaya; Alexey Karpov

doi:10.1007/978-3-031-20980-2_34

Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification

Dani Krebbers, Heysem Kaya^*, Alexey Karpov

^*Corresponding author for this work

Utrecht University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Native Language Identification is a prominent paralinguistic study with applications ranging from biometric analysis to speaker adaptation. Former studies on this task have benefited from alternative acoustic feature representations and pre-trained neural networks. In this work, we explore the Native Language Identification performance of contextual acoustic (wav2vec 2.0) and linguistic (BERT) embeddings as state-of-the-art feature representations and combine them with acoustic features at different levels. We encode acoustic and linguistic features using Fisher Vectors, applying Fisher Vector encoding on BERT word embeddings and wav2vec 2.0 for the first time for a paralinguistic task. We compare this approach with conventional functional summarization. In line with our former study using only acoustic modality, the results indicate the superiority of Fisher Vectors encoding over the traditional techniques. Moreover, we show the efficacy of combining alternative representations now in both acoustic and linguistic modalities. Results indicate a notable contribution of the transformer-based contextual auditory and linguistic feature representations to bimodal Native Language Identification systems.

Original language	English
Title of host publication	Speech and Computer
Subtitle of host publication	24th International Conference, SPECOM 2022, Gurugram, India, November 14–16, 2022, Proceedings
Editors	S. R. Mahadeva Prasanna, Alexey Karpov, K. Samudravijaya, Shyam S. Agrawal
Publisher	Springer
Chapter	34
Pages	391-403
Number of pages	13
Edition	1
ISBN (Electronic)	978-3-031-20980-2
ISBN (Print)	978-3-031-20979-6
DOIs	https://doi.org/10.1007/978-3-031-20980-2_34
Publication status	Published - 10 Nov 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13721 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Bibliographical note

Funding Information:
Acknowledgments. Work of A. Karpov is supported by the RSF (project No. 22-11-00321). We also thank the data donors and challenge organizers for making this research possible.

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Keywords

BERT
Computational Paralinguistics
Fisher Vector
Native Language Identification
Wav2vec 2.0

Access to Document

10.1007/978-3-031-20980-2_34

978-3-031-20980-2_34Final published version, 787 KBLicence: Taverne

Cite this

Krebbers, D., Kaya, H., & Karpov, A. (2022). Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification. In S. R. M. Prasanna, A. Karpov, K. Samudravijaya, & S. S. Agrawal (Eds.), Speech and Computer: 24th International Conference, SPECOM 2022, Gurugram, India, November 14–16, 2022, Proceedings (1 ed., pp. 391-403). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13721 LNAI). Springer. https://doi.org/10.1007/978-3-031-20980-2_34

Krebbers, Dani ; Kaya, Heysem ; Karpov, Alexey. / Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification. Speech and Computer: 24th International Conference, SPECOM 2022, Gurugram, India, November 14–16, 2022, Proceedings. editor / S. R. Mahadeva Prasanna ; Alexey Karpov ; K. Samudravijaya ; Shyam S. Agrawal. 1. ed. Springer, 2022. pp. 391-403 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{0326d82d47f94aa6a978d7adb2a8e50e,

title = "Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification",

abstract = "Native Language Identification is a prominent paralinguistic study with applications ranging from biometric analysis to speaker adaptation. Former studies on this task have benefited from alternative acoustic feature representations and pre-trained neural networks. In this work, we explore the Native Language Identification performance of contextual acoustic (wav2vec 2.0) and linguistic (BERT) embeddings as state-of-the-art feature representations and combine them with acoustic features at different levels. We encode acoustic and linguistic features using Fisher Vectors, applying Fisher Vector encoding on BERT word embeddings and wav2vec 2.0 for the first time for a paralinguistic task. We compare this approach with conventional functional summarization. In line with our former study using only acoustic modality, the results indicate the superiority of Fisher Vectors encoding over the traditional techniques. Moreover, we show the efficacy of combining alternative representations now in both acoustic and linguistic modalities. Results indicate a notable contribution of the transformer-based contextual auditory and linguistic feature representations to bimodal Native Language Identification systems.",

keywords = "BERT, Computational Paralinguistics, Fisher Vector, Native Language Identification, Wav2vec 2.0",

author = "Dani Krebbers and Heysem Kaya and Alexey Karpov",

note = "Funding Information: Acknowledgments. Work of A. Karpov is supported by the RSF (project No. 22-11-00321). We also thank the data donors and challenge organizers for making this research possible. Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.",

year = "2022",

month = nov,

day = "10",

doi = "10.1007/978-3-031-20980-2_34",

language = "English",

isbn = "978-3-031-20979-6",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "391--403",

editor = "Prasanna, {S. R. Mahadeva} and Alexey Karpov and K. Samudravijaya and Agrawal, {Shyam S.}",

booktitle = "Speech and Computer",

edition = "1",

}

Krebbers, D, Kaya, H & Karpov, A 2022, Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification. in SRM Prasanna, A Karpov, K Samudravijaya & SS Agrawal (eds), Speech and Computer: 24th International Conference, SPECOM 2022, Gurugram, India, November 14–16, 2022, Proceedings. 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13721 LNAI, Springer, pp. 391-403. https://doi.org/10.1007/978-3-031-20980-2_34

Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification. / Krebbers, Dani; Kaya, Heysem; Karpov, Alexey.
Speech and Computer: 24th International Conference, SPECOM 2022, Gurugram, India, November 14–16, 2022, Proceedings. ed. / S. R. Mahadeva Prasanna; Alexey Karpov; K. Samudravijaya; Shyam S. Agrawal. 1. ed. Springer, 2022. p. 391-403 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13721 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification

AU - Krebbers, Dani

AU - Kaya, Heysem

AU - Karpov, Alexey

N1 - Funding Information: Acknowledgments. Work of A. Karpov is supported by the RSF (project No. 22-11-00321). We also thank the data donors and challenge organizers for making this research possible. Publisher Copyright: © 2022, Springer Nature Switzerland AG.

PY - 2022/11/10

Y1 - 2022/11/10

N2 - Native Language Identification is a prominent paralinguistic study with applications ranging from biometric analysis to speaker adaptation. Former studies on this task have benefited from alternative acoustic feature representations and pre-trained neural networks. In this work, we explore the Native Language Identification performance of contextual acoustic (wav2vec 2.0) and linguistic (BERT) embeddings as state-of-the-art feature representations and combine them with acoustic features at different levels. We encode acoustic and linguistic features using Fisher Vectors, applying Fisher Vector encoding on BERT word embeddings and wav2vec 2.0 for the first time for a paralinguistic task. We compare this approach with conventional functional summarization. In line with our former study using only acoustic modality, the results indicate the superiority of Fisher Vectors encoding over the traditional techniques. Moreover, we show the efficacy of combining alternative representations now in both acoustic and linguistic modalities. Results indicate a notable contribution of the transformer-based contextual auditory and linguistic feature representations to bimodal Native Language Identification systems.

AB - Native Language Identification is a prominent paralinguistic study with applications ranging from biometric analysis to speaker adaptation. Former studies on this task have benefited from alternative acoustic feature representations and pre-trained neural networks. In this work, we explore the Native Language Identification performance of contextual acoustic (wav2vec 2.0) and linguistic (BERT) embeddings as state-of-the-art feature representations and combine them with acoustic features at different levels. We encode acoustic and linguistic features using Fisher Vectors, applying Fisher Vector encoding on BERT word embeddings and wav2vec 2.0 for the first time for a paralinguistic task. We compare this approach with conventional functional summarization. In line with our former study using only acoustic modality, the results indicate the superiority of Fisher Vectors encoding over the traditional techniques. Moreover, we show the efficacy of combining alternative representations now in both acoustic and linguistic modalities. Results indicate a notable contribution of the transformer-based contextual auditory and linguistic feature representations to bimodal Native Language Identification systems.

KW - BERT

KW - Computational Paralinguistics

KW - Fisher Vector

KW - Native Language Identification

KW - Wav2vec 2.0

UR - http://www.scopus.com/inward/record.url?scp=85142678384&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-20980-2_34

DO - 10.1007/978-3-031-20980-2_34

M3 - Conference contribution

SN - 978-3-031-20979-6

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 391

EP - 403

BT - Speech and Computer

A2 - Prasanna, S. R. Mahadeva

A2 - Karpov, Alexey

A2 - Samudravijaya, K.

A2 - Agrawal, Shyam S.

PB - Springer

ER -

Krebbers D, Kaya H, Karpov A. Multi-level Fusion of Fisher Vector Encoded BERT and Wav2vec 2.0 Embeddings for Native Language Identification. In Prasanna SRM, Karpov A, Samudravijaya K, Agrawal SS, editors, Speech and Computer: 24th International Conference, SPECOM 2022, Gurugram, India, November 14–16, 2022, Proceedings. 1 ed. Springer. 2022. p. 391-403. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-20980-2_34