Video-and-Language (VidL) models and their cognitive relevance

A Zonneveld; A Gatt; I Calixto

Video-and-Language (VidL) models and their cognitive relevance

A Zonneveld, A Gatt, I Calixto

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

In this paper we give a narrative review of multi-modal
video-language (VidL) models. We introduce the current
landscape of VidL models and benchmarks, and draw inspiration from neuroscience and cognitive science to propose avenues for future research in VidL models in particular and artifcial intelligence (AI) in general. We argue
that iterative feedback loops between AI, neuroscience, and
cognitive science are essential to spur progress across these
disciplines. We motivate why we focus specifcally on VidL
models and their benchmarks as a promising type of model
to bring improvements in AI and categorise current VidL efforts across multiple ‘cognitive relevance axioms’. Finally,
we provide suggestions on how to effectively incorporate
this interdisciplinary viewpoint into research on VidL models in particular and AI in general. In doing so, we hope to
create awareness of the potential of VidL models to narrow
the gap between neuroscience, cognitive science, and AI.

Original language	English
Title of host publication	What is next in Multimodal Foundational Models? Proceedings of the ICCV Workshop.
Place of Publication	Paris, France
Publisher	IEEE
Pages	1-14
Number of pages	14
Publication status	Published - 2023

Keywords

vision and language

Access to Document

https://openaccess.thecvf.com/content/ICCV2023W/MMFM/papers/Zonneveld_Video-and-Language_VidL_models_and_their_cognitive_relevance_ICCVW_2023_paper.pdf

Cite this

@inproceedings{5bc38d3e87bb406e975d1f6aacc55dbd,

title = "Video-and-Language (VidL) models and their cognitive relevance",

abstract = "In this paper we give a narrative review of multi-modalvideo-language (VidL) models. We introduce the currentlandscape of VidL models and benchmarks, and draw inspiration from neuroscience and cognitive science to propose avenues for future research in VidL models in particular and artifcial intelligence (AI) in general. We arguethat iterative feedback loops between AI, neuroscience, andcognitive science are essential to spur progress across thesedisciplines. We motivate why we focus specifcally on VidLmodels and their benchmarks as a promising type of modelto bring improvements in AI and categorise current VidL efforts across multiple {\textquoteleft}cognitive relevance axioms{\textquoteright}. Finally,we provide suggestions on how to effectively incorporatethis interdisciplinary viewpoint into research on VidL models in particular and AI in general. In doing so, we hope tocreate awareness of the potential of VidL models to narrowthe gap between neuroscience, cognitive science, and AI.",

keywords = "vision and language",

author = "A Zonneveld and A Gatt and I Calixto",

year = "2023",

language = "English",

pages = "1--14",

booktitle = "What is next in Multimodal Foundational Models? Proceedings of the ICCV Workshop.",

publisher = "IEEE",

address = "United States",

}

Zonneveld, A, Gatt, A & Calixto, I 2023, Video-and-Language (VidL) models and their cognitive relevance. in What is next in Multimodal Foundational Models? Proceedings of the ICCV Workshop.. IEEE, Paris, France, pp. 1-14. <https://openaccess.thecvf.com/content/ICCV2023W/MMFM/papers/Zonneveld_Video-and-Language_VidL_models_and_their_cognitive_relevance_ICCVW_2023_paper.pdf>

TY - GEN

T1 - Video-and-Language (VidL) models and their cognitive relevance

AU - Zonneveld, A

AU - Gatt, A

AU - Calixto, I

PY - 2023

Y1 - 2023

N2 - In this paper we give a narrative review of multi-modalvideo-language (VidL) models. We introduce the currentlandscape of VidL models and benchmarks, and draw inspiration from neuroscience and cognitive science to propose avenues for future research in VidL models in particular and artifcial intelligence (AI) in general. We arguethat iterative feedback loops between AI, neuroscience, andcognitive science are essential to spur progress across thesedisciplines. We motivate why we focus specifcally on VidLmodels and their benchmarks as a promising type of modelto bring improvements in AI and categorise current VidL efforts across multiple ‘cognitive relevance axioms’. Finally,we provide suggestions on how to effectively incorporatethis interdisciplinary viewpoint into research on VidL models in particular and AI in general. In doing so, we hope tocreate awareness of the potential of VidL models to narrowthe gap between neuroscience, cognitive science, and AI.

AB - In this paper we give a narrative review of multi-modalvideo-language (VidL) models. We introduce the currentlandscape of VidL models and benchmarks, and draw inspiration from neuroscience and cognitive science to propose avenues for future research in VidL models in particular and artifcial intelligence (AI) in general. We arguethat iterative feedback loops between AI, neuroscience, andcognitive science are essential to spur progress across thesedisciplines. We motivate why we focus specifcally on VidLmodels and their benchmarks as a promising type of modelto bring improvements in AI and categorise current VidL efforts across multiple ‘cognitive relevance axioms’. Finally,we provide suggestions on how to effectively incorporatethis interdisciplinary viewpoint into research on VidL models in particular and AI in general. In doing so, we hope tocreate awareness of the potential of VidL models to narrowthe gap between neuroscience, cognitive science, and AI.

KW - vision and language

M3 - Conference contribution

SP - 1

EP - 14

BT - What is next in Multimodal Foundational Models? Proceedings of the ICCV Workshop.

PB - IEEE

CY - Paris, France

ER -

Video-and-Language (VidL) models and their cognitive relevance

Abstract

Keywords

Access to Document

Fingerprint

Cite this