UI X-Ray: Interactive mobile UI testing based on computer vision

Abstract

User Interface/eXperience (UI/UX) significantly affects the lifetime of any software program, particularly mobile apps. A bad UX can undermine the success of a mobile app even if that app enables sophisticated capabilities. A good UX, however, needs to be supported of a highly functional and user friendly UI design. In spite of the importance of building mobile apps based on solid UI designs, UI discrepancies- inconsistencies between UI design and implementation-Are among the most numerous and expensive defects encountered during testing. This paper presents UI X-RAY, an interactive UI testing system that integrates computer-vision methods to facilitate the correction of UI discrepancies- such as inconsistent positions, sizes and colors of objects and fonts. Using UI X-RAY does not require any programming experience; therefore, UI X-RAY can be used even by non-programmers-particularly designers-which significantly reduces the overhead involved in writing tests. With the feature of interactive interface, UI testers can quickly generate defect reports and revision instructions-which would otherwise be done manually. We verified our UI X-RAY on 4 developed mobile apps of which the entire development history was saved. UI X-RAY achieved a 99.03% true-positive rate, which significantly surpassed the 20.92% true-positive rate obtained via manual analysis. Furthermore, evaluating the results of our automated analysis can be completed quickly (< 1 minute per view on average) compared to hours of manual work required by UI testers. On the other hand, UI X-RAY received the appreciations from skilled designers and UI X-RAY improves their current work flow to generate UI defect reports and revision instructions. The proposed system, UI X-RAY, presented in this paper has recently become part of a commercial product.