How predictable is ASR confidence in dialog applications?

Abstract

ASR confidence is a metric that reflects, to a large extent, the conditions under which a recognition task is being carried out as well as the reliability of the result. Because of this, ASR confidence constitutes a potentially useful feature in frameworks that attempt to asses the state of a dialog. In this paper we evaluate the predictability of ASR confidence based on knowledge of previously observed context-dependent confidences. We find out that the contextual confidence can be predicted with a standard prediction deviation less than 10% of the dynamic range of the confidence score, which represents a almost 40% relative reduction in standard deviation measure to a static confidence assumption baseline. Because our prediction is based on context, this predictability can be leveraged to produce an estimate of the expected average confidence until the end of a call based on the context path expected to be traversed.