Combining pretest methods to investigate comprehension problems in attitude questions: Evidence from Voting Advice Applications

Bregje Holleman, Naomi Kamoen

Research output: Contribution to conferenceOtherOther research output


In two studies, we investigated how respondents understand attitude questions and how they respond in case of comprehension difficulties. We investigated this in the context of Voting Advice Applications (VAAs): online tools that provide a voting advice based on a comparison between a respondent's answer and the positions of political parties.

Study 1 consists of cognitive interviews with 60 users filling out 30 VAA statements prior to municipal elections in Utrecht. Results show that VAA users encounter comprehension problems for 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question, often triggered by vague quantifying term (e.g., 'extra'), which make the users realize they lack knowledge about the current state of affairs ('How high is the budget now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions. We performed a statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. We will discuss the methodological advantages of combining methods, and we will outline the implications of our findings for questionnaire design.
Original languageEnglish
Publication statusPublished - Nov 2016
EventQDET2 conference - Miami, Florida, Miami, United States
Duration: 9 Nov 201613 Nov 2016


ConferenceQDET2 conference
Country/TerritoryUnited States


  • cognitive interviewing
  • statistical analysis
  • 'no opinion'
  • political attitude questions


Dive into the research topics of 'Combining pretest methods to investigate comprehension problems in attitude questions: Evidence from Voting Advice Applications'. Together they form a unique fingerprint.

Cite this