Conversational multi-modal browser: An integrated multi-modal browser and dialog manager

Abstract

This paper describes a multi-modal, free flow browser. A multi-modal browser allows the user to perform tasks via any of several modalities, such as speech and GUI or devices available sequentially or concurrently. A free-flow or mixed initiative browser allows the user to choose the navigation across an application and the order in which tasks are completed. This paper describes a system to provide flow conversational capability across modalities or devices. Multi-modal interaction is provided using a DOM based MVC multi-modal browser extended with an event-driven form-based dialog manager. We use XForms-based device independent authoring for the application. This paper details the XForms-based programming model of conversational applications, the principles and architecture of an XForms-based and event driven, task oriented, mixed initiative dialog manager and the integration of a dialog manager and a multi-modal browser using a standard-based DOM event driven interface. The dialog manager relies on a DOM based representation of the form-based framework presented by Papineni (1999) and Davies et al. (1999) to provide free-flow dialog management. It determines the user's intention by measuring the relevance of user input to different tasks. It also provides focus determination by determining the next logical turn in the conversation.