Task Robustness via Re-Labelling Vision-Action Robot Data

A new framework called TREAD uses large vision-language models to relabel existing robotics datasets, improving how policies handle novel tasks and follow instructions without requiring additional data collection, according to research published on arXiv [1][2]. The framework, formally named Task Robustness via Re-Labelling Vision-Action Robot Data, addresses a persistent weakness in robot learning: policies that can generalize to new physical scenarios but struggle to follow varied linguistic commands [1][2]. Researchers attribute this gap to limited linguistic and action-sequence diversity in current robotics datasets [2]. TREAD operates in three stages. First, a pretrained vision-language model generates semantic sub-tasks from original instruction labels and initial scenes. Next, it segments demonstration videos based on those sub-tasks. Finally, it produces diverse instructions that incorporate object properties, decomposing longer demonstrations into grounded language-action pairs [1][2]. The approach also augments data with linguistically varied versions of text goals to further strengthen robustness [2]. Evaluations were conducted on LIBERO, a benchmark for robot manipulation [1][2]. Policies trained on TREAD-augmented datasets showed improved performance on novel, unseen tasks and goals compared to baselines [1][2]. The researchers report gains in both planning generalization—achieved through trajectory decomposition—and language-conditioned policy generalization, driven by the increased linguistic diversity of the training data [2]. The work sits within a broader push to scale robot learning through larger models. Since the 2020s, generative AI has expanded rapidly, with transformer architectures and large-scale neural networks enabling systems that perceive environments and take goal-directed actions [3]. Automation technologies have historically reduced human intervention in processes ranging from factory control loops to modern aircraft stabilization [4]. In robotics specifically, the challenge has shifted from basic motion control to instruction-following in unstructured settings, where datasets for tasks such as object detection and multi-label classification provide the foundation for training [5]. TREAD’s contribution is a data-centric method that extracts more signal from existing demonstrations rather than collecting new ones, leveraging the transferable knowledge embedded in large vision-language models [2].