Smileys, Stars, Hearts, Buttons, Tiles or Grids: Influence of Response Format on Substantive Response, Questionnaire Experience and Response Time

Studies of the processes underlying question answering in surveys suggest that the choice of (layout for) response categories can have a significant effect on respondent answers. In recent years, the use of pictures, such as emojis or stars, is often used in online communication. It is unclear if pictorial answer categories can replace traditional verbal formats as measurement instruments in surveys. In this article we investigate different versions of a Likert-scale to see if they generate similar results and user experiences. Data comes from the non-probability based Flitspanel in the Netherlands. The hearts and stars designs received lower average scores compared to the other formats. Smileys produced average answer scores in line with traditional radio buttons. Respondents evaluated the smiley design most positively. Grid designs were evaluated more negatively. People wanting to compare survey outcomes should be aware of these effects and only compare results when similar response formats are used.