A screenshot-based task mining framework for disclosing the drivers behind variable human actions

A. Martínez-Rojas*, A. Jiménez-Ramírez, J. G. Enríquez, H. A. Reijers

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Robotic Process Automation (RPA) enables subject matter experts to use the graphical user interface as a means to automate and integrate systems. This is a fast method to automate repetitive, mundane tasks. To avoid constructing a software robot from scratch, Task Mining approaches can be used to monitor human behavior through a series of timestamped events, such as mouse clicks and keystrokes. From a so-called User Interface log (UI Log), it is possible to automatically discover the process model behind this behavior. However, when the discovered process model shows different process variants, it is hard to determine what drives a human's decision to execute one variant over the other. Existing approaches do analyze the UI Log in search for the underlying rules, but neglect what can be seen on the screen. As a result, a major part of the human decision-making remains hidden. To address this gap, this paper describes a Task Mining framework that uses the screenshot of each event in the UI Log as an additional source of information. From such an enriched UI Log, by using image-processing techniques and Machine Learning algorithms, a decision tree is created, which offers a more complete explanation of the human decision-making process. The presented framework can express the decision tree graphically, explicitly identifying which elements in the screenshots are relevant to make the decision. The framework has been evaluated through a case study that involves a process with real-life screenshots. The results indicate a satisfactorily high accuracy of the overall approach, even if a small UI Log is used. The evaluation also identifies challenges for applying the framework in a real-life setting when a high density of interface elements is present.

Original languageEnglish
Article number102340
Number of pages16
JournalInformation Systems
Volume121
DOIs
Publication statusPublished - Mar 2024

Bibliographical note

Publisher Copyright:
© 2023 The Author(s)

Funding

This publication is part of the project PID2022-137646OB-C31, funded by MCIN/AEI/10.13039/501100011033/and by the "European Union"; the Centro para el Desarrollo Tecnologico Industrial (CDTI) of Spain under the LINGER project (EXP 00141834/IDI-20220145) ; the DISCOVERY project (2021/C005/00148631) , funded by Union Europea NextGeneration EU and "Plan de Recuperacion, Transformacion y Resiliencia" of the Ministry of Economic and Digital Transformation; the FPU scholarship (FPU20/05984) funded by MCIN/AEI/10.13039/501100011033 and "FSE invierte en tu futuro", and its mobility grants (EST23/00732).

FundersFunder number
MCIN/AEIPID2022-137646OB-C31, EST23/00732
European Union
Centro para el Desarrollo Tecnologico Industrial (CDTI) of Spain under the LINGER projectEXP 00141834/IDI-20220145
Union Europea NextGeneration EU2021/C005/00148631
Ministry of Economic and Digital Transformation
FPU scholarship - MCIN/AEIFPU20/05984
FSE invierte en tu futuro

    Keywords

    • Decision model discovery
    • Robotic Process Automation
    • Task mining
    • UI Log
    • User behavior mining

    Fingerprint

    Dive into the research topics of 'A screenshot-based task mining framework for disclosing the drivers behind variable human actions'. Together they form a unique fingerprint.

    Cite this