About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
IUI 2017
Conference paper
UI X-Ray: Interactive mobile UI testing based on computer vision
Abstract
User Interface/eXperience (UI/UX) significantly affects the lifetime of any software program, particularly mobile apps. A bad UX can undermine the success of a mobile app even if that app enables sophisticated capabilities. A good UX, however, needs to be supported of a highly functional and user friendly UI design. In spite of the importance of building mobile apps based on solid UI designs, UI discrepancies- inconsistencies between UI design and implementation-Are among the most numerous and expensive defects encountered during testing. This paper presents UI X-RAY, an interactive UI testing system that integrates computer-vision methods to facilitate the correction of UI discrepancies- such as inconsistent positions, sizes and colors of objects and fonts. Using UI X-RAY does not require any programming experience; therefore, UI X-RAY can be used even by non-programmers-particularly designers-which significantly reduces the overhead involved in writing tests. With the feature of interactive interface, UI testers can quickly generate defect reports and revision instructions-which would otherwise be done manually. We verified our UI X-RAY on 4 developed mobile apps of which the entire development history was saved. UI X-RAY achieved a 99.03% true-positive rate, which significantly surpassed the 20.92% true-positive rate obtained via manual analysis. Furthermore, evaluating the results of our automated analysis can be completed quickly (< 1 minute per view on average) compared to hours of manual work required by UI testers. On the other hand, UI X-RAY received the appreciations from skilled designers and UI X-RAY improves their current work flow to generate UI defect reports and revision instructions. The proposed system, UI X-RAY, presented in this paper has recently become part of a commercial product.