C05: Voice Interaction Design and Evaluation: Practices Based on Psycholinguistics and Psychometrics

Back to Courses' Program

Sunday, 22 June 2025, 13:30 - 17:30 CEST (Central European Summer Time - Sweden)

James R. Lewis (short bio)
MeasuringU, USA

Modality

on-line

Target Audience

Anyone involved in the design or evaluation of voice interaction systems including researchers/academia, students, professionals, and industry.

Abstract

Successful voice interaction design requires many different strategies drawn from many different disciplines (e.g., speech technologies, human factors engineering/UX research, psychology, linguistics, statistics, sociology, pragmatics, market research, service science). In this course we focus on two platform-independent topics: design practices informed by psycholinguistics (a subdiscipline of psychology) and assessment with standardized questionnaires developed using psychometrics (a subdiscipline of statistics). For the first part, we'll discuss voice interaction design principles based on a wide variety of psycholinguistic research including implicit linguistic knowledge, phonology, coarticulation, prosody, conversational discourse, conversational maxims, grammaticality, discourse markers, timing/turn-taking, and social considerations in conversation. In the second part we'll briefly go over how standardized questionnaires are developed, then review the content and application of different versions of the Mean Opinion Scale (MOS), the questionnaire for the Subjective Assessment of Speech System Interfaces (SASSI), the questionnaire for Speech User Interface Service Quality (SUISQ), and the User Experience Questionnaire's scales for measuring the user experience quality of voice assistants (UEQ+).

Benefits for attendees

Attendees will learn the psycholinguistic bases for ten research-based voice interaction design practices and the current inventory of standardized questionnaires that can be used to assess different aspects of the user experience of voice systems.

Course Content

Goals and objectives

Provide an introduction to voice interaction design, followed by specific design practices informed by psycholinguistic research and discussion of standardized questionnaires that have been developed to assess different aspects of voice interaction experiences.

Brief introduction to voice interaction design

Review ten design strategies informed by psycholinguistic research:

  1. Implicit linguistic knowledge: Language knowledge we have that's hidden from introspection
  2. Phonology: Understanding how the mechanics of speech production affect voice interaction design
  3. Coarticulation: A slightly deeper dive into the mechanics of speech production and its consequences for design
  4. Prosody: Understanding the importance of the intonational patterns of speech
  5. Conversational discourse: Applying the principles of everyday conversation to voice interaction design
  6. Conversational maxims: Using the Gricean maxims to inform conversational design
  7. Grammaticality: Understanding the role of the "rules of proper English" in conversational design
  8. Discourse markers: Learning how to use short phrases to signal acknowledgement and upcoming topic shifts in conversational design
  9. Timing and turn-taking: Discussing the critical role of timing in voice interaction design
  10. Social considerations in conversation: As soon as two people talk with each other, there is a social situation to manage, even if one of those people is artificial

Review psychometrically qualified questionnaires for assessing perceptions of voice interaction designs:

  1. Discussion of how to develop a standardized questionnaire -- there's more to it than just writing a set of rating scales
  2. The Mean Opinion Scale (MOS) and its variants (MOS-X and MOS-X2): Covering the history of thedevelopment of the original MOS (measuring intelligibility and naturalness of synthetic speech) throughthe MOS-X2 (measuring and grading synthetic speech intelligibility, naturalness, prosody, and socialimpression)
  3. The Subjective Assessment of Speech System Interfaces (SASSI): Measurement of six aspects of voice interaction (system response accuracy, likeability, cognitive demand, annoyance, habitability, and speed)
  4. Speech User Interface Service Quality (SUISQ): Assessment focused on service quality of speech-enabled IVRs measuring user goal orientation, customer service behaviors, speech characteristics, and verbosity.
  5. User Experience Questionnaire (voice interaction scales): As part of a modular series of semantic differentials for measuring user experience, the UEQ+ has additional modules for the assessment of response behavior, response quality, and comprehensibility

Final discussion and wrap-up

Bio Sketch of Course instructor

James R. (Jim) Lewis graduated with an M.A. in Engineering Psychology in 1982 from New Mexico State University and received his Ph.D. in Experimental Psychology (Psycholinguistics) from Florida Atlantic University in 1996. He was a human factors engineer at IBM from 1981 to 2019 with over 15 years’ experience in voice interaction design and evaluation, after which he joined MeasuringU as their Distinguished User Experience Researcher. He is co-editor in chief of the Journal of User Experience and the author of "Practical Speech User Interface Design" plus over 100 peer-reviewed journal articles and conference papers. He is a BCPE Certified Human Factors Professional, an IBM Master Inventor with over 90 issued US patents issued to date, past president of the Association for Voice/Conversational Interaction Design, and a member of the Academy of Science, Engineering, and Medicine of Florida.