What Vision-Language Models `See' when they See Scenes

Michele Cafagna; Kees van Deemter; A. Gatt

doi:10.48550/arXiv.2109.07301

What Vision-Language Models `See' when they See Scenes

Michele Cafagna, Kees van Deemter, A. Gatt

Research output: Working paper › Preprint › Academic

Abstract

Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to align descriptions of both types with images. We compare 3 state-of-the-art models, VisualBERT, LXMERT and CLIP. We find that (i) V&L models are susceptible to stylistic biases acquired during pretraining; (ii) only CLIP performs consistently well on both object- and scene-level descriptions. A follow-up ablation study shows that CLIP uses object-level information in the visual modality to align with scene-level textual descriptions.

Original language	English
Publisher	arXiv
Pages	1-12
DOIs	https://doi.org/10.48550/arXiv.2109.07301
Publication status	Published - 15 Sept 2021

Access to Document

10.48550/arXiv.2109.07301Licence: CC BY

2109.07301 (1)Submitted manuscript, 2.54 MBLicence: CC BY

https://arxiv.org/abs/2109.07301Licence: CC BY

Cite this

@techreport{da543e48abbf4c9d8287e7eb04c84b4f,

title = "What Vision-Language Models `See' when they See Scenes",

abstract = "Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to align descriptions of both types with images. We compare 3 state-of-the-art models, VisualBERT, LXMERT and CLIP. We find that (i) V&L models are susceptible to stylistic biases acquired during pretraining; (ii) only CLIP performs consistently well on both object- and scene-level descriptions. A follow-up ablation study shows that CLIP uses object-level information in the visual modality to align with scene-level textual descriptions.",

author = "Michele Cafagna and {van Deemter}, Kees and A. Gatt",

year = "2021",

month = sep,

day = "15",

doi = "10.48550/arXiv.2109.07301",

language = "English",

pages = "1--12",

publisher = "arXiv",

type = "WorkingPaper",

institution = "arXiv",

}

TY - UNPB

T1 - What Vision-Language Models `See' when they See Scenes

AU - Cafagna, Michele

AU - van Deemter, Kees

AU - Gatt, A.

PY - 2021/9/15

Y1 - 2021/9/15

N2 - Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to align descriptions of both types with images. We compare 3 state-of-the-art models, VisualBERT, LXMERT and CLIP. We find that (i) V&L models are susceptible to stylistic biases acquired during pretraining; (ii) only CLIP performs consistently well on both object- and scene-level descriptions. A follow-up ablation study shows that CLIP uses object-level information in the visual modality to align with scene-level textual descriptions.

AB - Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to align descriptions of both types with images. We compare 3 state-of-the-art models, VisualBERT, LXMERT and CLIP. We find that (i) V&L models are susceptible to stylistic biases acquired during pretraining; (ii) only CLIP performs consistently well on both object- and scene-level descriptions. A follow-up ablation study shows that CLIP uses object-level information in the visual modality to align with scene-level textual descriptions.

U2 - 10.48550/arXiv.2109.07301

DO - 10.48550/arXiv.2109.07301

M3 - Preprint

SP - 1

EP - 12

BT - What Vision-Language Models `See' when they See Scenes

PB - arXiv

ER -

What Vision-Language Models `See' when they See Scenes

Abstract

Access to Document

Fingerprint

Cite this