Can Survey Item Characteristics Relevant to Measurement Error Be Coded Reliably? A Case Study on 11 Dutch General Population Surveys

F. Bais*, J.G. Schouten, P.J. Lugtig, V. Toepoel, Judit Arends-Tóth, salima DOUHOU, Natalia Kieruj, M. Morren, Corrie Vis

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Item characteristics can have a significant effect on survey data quality and may be associated with measurement error. Literature on data quality and measurement error is often inconclusive. This could be because item characteristics used for detecting measurement error are not coded unambiguously. In our study, we use a systematic coding procedure with multiple coders to investigate the extent to which the coding of item characteristics could be done reliably. For this purpose, we constructed an item characteristics scheme that is based on typologies of characteristics. High intercoder reliability indicates a clear relation between item characteristic, item content, and measurement error. Our results show that intercoder reliability is often low, especially for item characteristics that are hard to code due to subjectivity. Low intercoder reliability complicates comparisons between studies about item characteristics and measurement error. We give suggestions for coping with low intercoder reliability.

Original languageEnglish
Pages (from-to)263-295
Number of pages33
JournalSociological Methods and Research
Volume48
Issue number2
Early online date2017
DOIs
Publication statusPublished - 1 May 2019

Keywords

  • data quality
  • intercoder reliability
  • item characteristics
  • measurement error
  • mode-specific measurement error
  • questionnaire profiles

Fingerprint

Dive into the research topics of 'Can Survey Item Characteristics Relevant to Measurement Error Be Coded Reliably? A Case Study on 11 Dutch General Population Surveys'. Together they form a unique fingerprint.

Cite this