TaskDiff: A Similarity Metric for Task-Oriented Conversations

Abstract

The popularity of conversational digital assistants has resulted in efforts to improve user experience by extracting insights from the logs. These approaches utilize distance based metrics to identify similarities between user conversations. These metrics are typically designed to compare text snippets and do not take advantage of the unique conversational features in dialogues that are absent from other textual sources. To address this gap, in this work, we present \textit{TaskSim}, a novel conversational similarity metric that utilizes different dialogue components (e.g.utterances, intents, and slots) with optimal transport. Extensive experimental evaluation of the \textit{TaskSim} metric on a benchmark dataset demonstrate its superior performance over other traditional similarity approaches.