Orchestra-o1: Omnimodal Agent Orchestration

A research team has introduced Orchestra-o1, an open-source framework that coordinates multiple AI agents to handle tasks spanning text, images, audio, and video simultaneously, according to a paper posted to arXiv on June 10, 2026 [1]. The framework addresses a gap in current multi-agent systems, which the authors say are typically confined to a narrow set of modalities and falter when heterogeneous data types interact [1]. Orchestra-o1 uses a unified orchestration mechanism that performs modality-aware task decomposition, assigns sub-tasks to specialized sub-agents online, and executes independent sub-tasks in parallel [1][4]. The main agent maintains a structured memory, breaks the task into dependency-aware sub-goals, selects suitable backends and tools for each, and iteratively compresses returned evidence until the answer is sufficiently supported [4]. The design decouples high-level planning from specialized perception and action execution, which the authors argue makes the system modular and scalable: new modalities, tools, or sub-agent models can be integrated through the same requirement-matching interface [4]. The dependency-aware scheduler improves latency whenever multiple independent sub-tasks can be solved concurrently [4]. On the OmniGAIA benchmark, Orchestra-o1 outperformed the second-best approach by 10.3% in accuracy [1][5]. When using GPT-5 as the main agent, the framework achieved a 32.8% improvement over the AOrchestra baseline and a 10.3% improvement over Gemini-3-Pro [6]. The researchers also trained an 8-billion-parameter orchestrator model, Orchestra-o1-8B, using a technique called decision-aligned group relative policy optimization, or DA-GRPO, which they report achieves state-of-the-art performance against all existing open-source omnimodal agents [1][5]. The code and model weights have been released publicly on GitHub and Hugging Face [5]. The authors note they tested the pipeline on an 8 × H20 compute setup and have not verified a minimum configuration for lower-resource settings [5]. For users with limited compute, they recommend using a smaller model such as Qwen3-4B or applying parameter-efficient fine-tuning methods like LoRA [5].