Using Response Generation Behaviors to Improve the Quality of Survey Data

Abstract

Despite their prevalent use, online surveys are vulnerable to data quality problems that may be attributed to several factors, including respondent induced measurement errors that cause discrepancies between respondent attributes and their responses. This research explores how a respondent’s response generation data manifests in their human computer interaction (HCI) device usage; and how these fine-grained HCI data may be used to identify poor-quality responses in online surveys. The larger goal of this research is to establish a structured set of guidelines to address the presence, impact, and appropriate mitigation for poor-quality survey responses. To this end, this dissertation provides a framework that utilizes individual characteristics, survey characteristics, as well as other external factors to determine one's response generation process and consequently the usefulness of that response.This dissertation establishes the prevalence of poor-quality responses in survey research and examines how behavioral differences between respondents exhibiting previously known undesirable behaviors may be used to detect poor-quality responses. ‘Essay One’ examines the presence of poor quality data among professional survey respondents and how HCI based behavioral data can be used to identify them. ‘Essay Two’ examines the same problem among another common participant pool, university students. ‘Essay Three’ alleviates typical Type I concerns in ‘Essay One’ and ‘Essay Two’ by examining whether specific undesirable behaviors exhibited by respondents while answering a question may be captured using appropriate metrics. The overall results obtained suggest that behavioral differences during the response generation process may be captured through HCI devices to create appropriate metrics that identify poor quality responses.