Teach-and-Repeat: Accurately Extracting Operational Knowledge from Mobile Screen Demonstrations to Empower GUI Agents

Researchers have introduced Teach VLM, a model that translates mobile screen trajectories into operational knowledge, and proposed the Teach-and-Repeat paradigm for task automation, achieving state-of-the-art performance in operation semantics prediction.

Teach VLM extracts and analyzes operation-related keyframes from demonstration videos to generate operational knowledge, which is then used to guide downstream screen-based execution agents^[1]. A systematic data flywheel was developed to address the scarcity of aligned training data. GUI agents, increasingly used to automate complex computer tasks, have seen improvements with the introduction of experiential memory, including visual memory that stores and retrieves screenshots from past interactions^[2]. However, the effect of visual memory in GUI agents remains insufficiently understood, with a taxonomy of four GUI agent failures introduced: cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error. A study found that visual memory has a divergent effect on failure distribution and proposed a new framework, Action-Grounded Visual Memory (AGMem), to improve task success rates^[2].