Improving response accuracy for classification- based conversational IT services

Abstract

Conversational IT services are expected to reduce user wait times and improve overall customer satisfaction. Cloud-based solutions are readily available for enterprise subject matter experts (SMEs) to train user-question classifiers and build conversational services with little effort. However, methodologies that the SMEs can use to improve the response accuracy and conversation quality are merely stated and evaluated. In complex service scenarios such as software support, the scope of topics is typically large and the training samples are often limited. Thus, training the classifier based on labeled samples of plain user utterances is not effective in most cases. In this paper, we identify several methods for improving classification quality and evaluate them in concrete training set scenarios. Particularly, a process-based methodology is described that builds and refines on top of service domain knowledge in order to develop a scalable solution for training accurate conversation services. Enterprises and service providers are continuously seeking new ways to improve customer experience on working with IT systems, where user wait times and service resolution quality are critical business metrics. One of the latest trends is the use of conversational IT services. Customers can interact with a conversational service to express their questions in natural language and the system can automatically return relevant answers or execute back-end processes for automated actions. Various text classification techniques have been developed and applied to understand the user questions and trigger the correct responses. For instance, in the context of IT software support, customers can use conversational systems to get answers about software product errors, licenses, or upgrade processes. While the potential benefits of building conversational services are huge, it is often difficult to effectively train classification models that cover well the scope of realistically complex services. In this paper, we propose a training methodology that addresses the limitations in both the scope of topics and the scarcity of the training set. We further evaluate the proposed methodology in a real service support scenario and share the lessons learned.