On the effect of word error rate on automated quality monitoring

This paper studies the effect of word-error-rate (WER) on an automated quality monitoring application for call centers. The system consists of a speech recognition module and a call ranking module. The call ranking module combines direct question answering with a maximum-entropy classifier to automatically monitor the calls that enter a call center, and label them as "good" or "bad". We find that, in the monitoring regime where only a small fraction of the calls are monitored, we achieve 80% precision and 50% recall in classifying whether a call belongs to the bottom 20%. Additionally, the correlation between human and computer-generated scores turns out to be highly sensitive to word error rate. ©2006 IEEE.