VISUALSKILL: Multimodal Skills for Computer-Use Agents

Researchers have proposed VISUALSKILL, a hierarchical multimodal skill framework for computer-use agents that retains visual figures in the skill artifact, addressing a limitation in existing text-only skill libraries [1]. Computer-use agents, or CUAs, have approached human-level performance on standardized benchmarks but continue to falter on long-horizon tasks and when encountering unfamiliar software [1]. Existing approaches to improving agent reliability rely on reusable skill libraries, yet these libraries represent skills exclusively as text, ignoring the inherently visual nature of graphical user interface interaction [1]. The VISUALSKILL framework counters this by constructing a hierarchical multimodal skill tailored to each target application, organized as a central index over per-topic files [1]. The agent accesses the skill through a load_topic MCP tool that fetches the relevant topic's text and figures on demand [1]. Each skill is built using a two-stage pipeline that combines authored documentation with live-application UI exploration [1]. On two computer-use-agent benchmarks, CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 achieved an average score of 0.456 when equipped with VISUALSKILL [1]. This represents a 15.3-point absolute lift over the no-skill baseline of 0.303 [1]. To isolate the contribution of the visual modality, the researchers compared VISUALSKILL against a matched text-only skill generated from the same source content and differing only in modality [1]. The text-only skill scored 0.373, meaning VISUALSKILL yielded a further 8.3-point absolute gain [1]. The authors state this provides direct evidence that retaining visual figures in the skill artifact, rather than verbalizing them away, helps the agent both identify UI elements and verify workflow state after each action [1]. The work arrives as interest in autonomous GUI agents intensifies across the research community. The paper was submitted to arXiv on June 16, 2026, under the Computation and Language category, and the accompanying code has been made publicly available on GitHub [1]. The framework was developed within arXivLabs, a platform that allows collaborators to build and share new features directly on the arXiv website under principles of openness, community, excellence, and user data privacy [1].