Improved language modeling for conversational applications using sentence quality

Abstract

In this paper, we propose a new approach to build language models for conversationals system using a a corpus of text as a opposed to a live or a Wizard-of-Oz collection. Each sentence in the corpus is assigned a "quality" that reflects the developer's intuition for how likely that sentence is to be spoken by a real user to the live system. Language Models (LM) are built for each sentence quality and these are subsequently interpolated to produce the final model. We also have built a classifier that assigns sentence qualities to the data, and whose subsequent language models achive similar improvements in word and turn error rate. ©2010 IEEE.