Comparison of measures of speech quality for listening tests of text-to-speech systems

Abstract

Text-to-speech system improvements are best assessed by periodic listening tests. The paper reports on the results of two listening test methodologies: measurement of perceptions of speech quality using a single-item scale and a nine-item scale. The tests engender two important conclusions. First, TTS systems, as well as individual synthesized sentences used to test the system, are factors that affect a listener's perception. Second, the single-item scale can be a reliable and valid substitute for the nine-item scale with appropriate empirical justification when pragmatic considerations such as respondent fatigue necessitate the use of a shorter scale.