EvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation

A new framework called EvoPrompt aims to solve a persistent problem in machine learning: adapting large vision-language models to specialized tasks with only a handful of labeled examples, without causing the model to forget its broad pre-trained knowledge [1]. The framework, detailed in a paper submitted to arXiv on 10 March 2026 and revised on 3 June 2026, introduces a method that treats the fine-tuning of prompts as an evolutionary process that must be carefully guided [1][2]. The core insight is that governing the "evolutionary path of prompts is essential for forgetting-free adaptation" [2]. Unlike conventional prompt tuning, which the authors argue can lead to "catastrophic forgetting of pre-trained knowledge," EvoPrompt is designed to explicitly steer this trajectory [2][3]. To achieve this, the framework employs a Modality-Shared Prompt Projector, or MPP, which generates hierarchical prompts from a unified embedding space rather than using isolated, per-layer prompts [1][3]. This projector establishes "a bridge for cross-layer and cross-modal synergy" [3]. The training process itself is then managed by an evolutionary strategy that decouples low-rank updates into two distinct components: direction and magnitude. The paper explains that this method works by "preserving early-learned semantic directions while only adapting their magnitude, thus enabling prompts to evolve without discarding foundational knowledge" [1][2]. The process is further stabilized by a technique called Feature Geometric Regularization, or FGR, which enforces feature decorrelation to prevent what is known as representation collapse, a common failure mode when training on small datasets [1][3]. The authors report that their experiments show EvoPrompt achieves state-of-the-art performance in few-shot learning scenarios while "robustly preserving the original zero-shot capabilities of pre-trained VLMs" [1][2]. The concept of applying evolutionary principles to prompt engineering is not entirely new, though previous implementations have differed significantly. A separate 2023 framework, also named EvoPrompt, connected large language models with evolutionary algorithms to automatically optimize discrete, human-readable prompts for tasks like language understanding, without requiring access to model parameters or gradients [4]. More recently, a method called PromptEvolver used a vision-language model within a genetic algorithm to iteratively refine text prompts for text-to-image generation, operating entirely in natural language space [5]. The new EvoPrompt for vision-language model adaptation is distinct in its focus on preserving pre-trained knowledge by controlling the trajectory of internal prompt representations during fine-tuning, rather than evolving the text of the prompts themselves [2][3].