TY - JOUR
T1 - Can Survey Item Characteristics Relevant to Measurement Error Be Coded Reliably?
T2 - A Case Study on 11 Dutch General Population Surveys
AU - Bais, F.
AU - Schouten, J.G.
AU - Lugtig, P.J.
AU - Toepoel, V.
AU - Arends-Tóth, Judit
AU - DOUHOU, salima
AU - Kieruj, Natalia
AU - Morren, M.
AU - Vis, Corrie
PY - 2019/5/1
Y1 - 2019/5/1
N2 - Item characteristics can have a significant effect on survey data quality and may be associated with measurement error. Literature on data quality and measurement error is often inconclusive. This could be because item characteristics used for detecting measurement error are not coded unambiguously. In our study, we use a systematic coding procedure with multiple coders to investigate the extent to which the coding of item characteristics could be done reliably. For this purpose, we constructed an item characteristics scheme that is based on typologies of characteristics. High intercoder reliability indicates a clear relation between item characteristic, item content, and measurement error. Our results show that intercoder reliability is often low, especially for item characteristics that are hard to code due to subjectivity. Low intercoder reliability complicates comparisons between studies about item characteristics and measurement error. We give suggestions for coping with low intercoder reliability.
AB - Item characteristics can have a significant effect on survey data quality and may be associated with measurement error. Literature on data quality and measurement error is often inconclusive. This could be because item characteristics used for detecting measurement error are not coded unambiguously. In our study, we use a systematic coding procedure with multiple coders to investigate the extent to which the coding of item characteristics could be done reliably. For this purpose, we constructed an item characteristics scheme that is based on typologies of characteristics. High intercoder reliability indicates a clear relation between item characteristic, item content, and measurement error. Our results show that intercoder reliability is often low, especially for item characteristics that are hard to code due to subjectivity. Low intercoder reliability complicates comparisons between studies about item characteristics and measurement error. We give suggestions for coping with low intercoder reliability.
KW - data quality
KW - intercoder reliability
KW - item characteristics
KW - measurement error
KW - mode-specific measurement error
KW - questionnaire profiles
UR - http://www.scopus.com/inward/record.url?scp=85064330460&partnerID=8YFLogxK
U2 - 10.1177/0049124117729692
DO - 10.1177/0049124117729692
M3 - Article
AN - SCOPUS:85064330460
SN - 0049-1241
VL - 48
SP - 263
EP - 295
JO - Sociological Methods and Research
JF - Sociological Methods and Research
IS - 2
ER -