About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
CHI 2024
Workshop paper
Towards a No-code Resilient Intelligent Automation System
Abstract
The advancement of Large Language Models (LLMs) has opened new avenues for addressing the complexities of UI automation, presenting solutions that cater not only to developers but also to non-technical users. This paper discusses IDA, an innovative no-code UI automation system that leverages LLMs to address two major challenges in UI automation: resilient element matching and the management of conditional UI logic complexities. IDA's effectiveness is evidenced by its success in the IPA challenge, demonstrating remarkable resilience by adapting to dynamic UI changes and making UI automation accessible to non-developers. Through an examination of IDA's components and its application in automating tasks with varying degrees of complexity, we illustrate the potential of LLMs to simplify UI automation processes and enhance the accessibility of no-code tools for non-technical users. Our findings underscore the significance of integrating AI to overcome technical barriers and extend the benefits of UI automation across a wider range of enterprise applications.