ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

A new inference framework called ViCoStream enables streaming video large language models to process video at more than 100 frames per second, according to a preprint posted to arXiv. The system coordinates multiple processing stages to sustain real-time performance on a single GPU. The framework, described in a paper submitted on June 18, 2026, formulates streaming VideoLLM inference as a coordinated pipeline that spans visual preprocessing, visual encoding, token dropping, and LLM prefilling and decoding [1]. The authors argue that prior work focused on accelerating individual modules — such as visual encoding, token pruning, or KV-cache compression — without demonstrating whether the full system can maintain real-time streaming [1]. ViCoStream addresses this by combining chunk-wise execution, CUDA-stream overlap, visual token control, bounded visual attention, and query-side retrieval to bound per-chunk computation and memory costs [1]. The paper reports that experiments using Qwen2.5-VL-3B and Qwen2.5-VL-7B-Instruct models across multiple streaming benchmarks achieved 134 FPS video throughput and a time-to-first-token of less than 50 ms on a single A100 GPU [1]. Accuracy remained close to full-history baselines, the authors state [1]. The research also includes a systematic study of bottleneck migration, examining how chunk size, token retention, attention locality, and retrieval scope influence the trade-off between throughput and accuracy [1]. Streaming video-language models must continuously ingest incoming frames while responding to queries with low latency, a dual requirement that makes both ingestion throughput and query-time responsiveness critical for real-world deployment [1]. The ViCoStream approach coordinates these stages rather than optimizing any single component in isolation, which the authors say provides a more complete picture of real-time streaming viability [1]. The work appears as a preprint and has not yet undergone peer review. The paper includes references to supporting tools and code repositories, though no official implementation was linked at the time of posting [1].