“Half My Survey Data Is Bad; The Problem Is, I Can't Tell Which Half”: Evaluating the Validatability of Large-Scale Survey Data Using 176 Algorithmically Generated Personas

Abstract

Survey data is foundational to much user research, including design artifacts such as personas. However, if survey data is invalidatable, the credibility of any downstream analysis is fundamentally undermined. This work starts with the premise that distinguishing valid from invalid survey data solely through internal survey checks, such as attention checks, is practically infeasible. We conducted a large-scale empirical survey of social media users (N ≈ 20,000) and used persona creation as an analytical lens. Of these responses, 8,140 were classified as (ostensibly) valid and 11,860 as invalid based on passing or failing attention checks. We construct ten datasets by progressively replacing ‘valid’ responses with ‘invalid’ ones in increments of 10%. From each dataset, we generate 16 personas, resulting in a total of 176, and compare their divergence. Results were stable despite data degradation. Persona demographics remained unchanged in most conditions, with demographic consistency at nearly 90%; an average of 12% of persona attributes were unchanged. More than 80% of the survey item diversity was consistent across datasets. The findings are that large-scale survey data may be unverifiable through internal checks alone due to the Validity Masking Effect, and that artifacts in such data can obscure underlying data quality issues, highlighting the risks of relying solely on survey data.