Utilizing Large Language Models to Analyze PSR.exe Recorded Input for Computer Use

Краткое изложение: The rapid advancement of Large Language Models (LLMs) has opened new frontiers in automating complex workflows. This paper explores an innovative approach to computer use simulation by leveraging Large Language Models (LLMs) to parse and interpret data recorded by PSR.exe, a tool designed to capture user’s mouse and keyboard operations. We propose a method to extract, analyze, and replicate user interactions recorded in MHT files. By decoding screenshots and extracting action sequences, we aim to develop an automated process that enables applications to emulate user operations effectively. The workflow combines BeautifulSoup for XML parsing, base64 for image decoding, and LLMs for semantic analysis. Results show that our method is lightweight, versatile, and capable of ensuring precision and adaptability while reducing dependency on external tracking tools.