Can survey item characteristics relevant to mode-specific measurement error be coded reliably?

In multi-mode questionnaire design, usually some consideration is given to mode-specific measurement error. Despite this consideration, however, these measurement effects can be unexpectedly large. For this reason, there is a strong incentive to better predict measurement effects. This may be done by constructing profiles of a questionnaire, in which relevant item characteristics are summarized. For all items of a survey, these item characteristics need to be coded and combined. In this paper, we evaluated a list of item characteristics that literature has reported as relevant to mode-specific measurement error. Most importantly, we evaluated the reliability of the coding of such characteristics. Our results showed that intercoder reliability can be low for the most relevant characteristics. This may be explained by the difficulty of defining the item characteristics and the inherent subjectivity with which these item characteristics are coded. Finally, some suggestions are made for coping with low intercoder reliability.