Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

Researchers have found that safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs, despite efforts to align them with safety protocols.

Recent studies attribute this vulnerability to 'shallow safety,' where alignment concentrates in the first few output tokens^[1]. Short token injections at any generation step can substantially alter subsequent safety behavior, and a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection^[1]. Autoregressive consistency, a property of LLMs, can concentrate alignment updates on early tokens, making safety alignment fragile^[2]. This phenomenon can be exploited by 'random insertion attacks,' which insert a short harmful span into an otherwise safe refusal trajectory and sustain the resulting harmful branch^[2]. To address this, researchers argue that robust safety alignment requires training on the generation process itself, not only its outputs^[1]. One proposed solution is to align models directly on generation trajectories constructed by simulating mid-sequence perturbation, which has been shown to improve robustness to mid-sequence injection and generalize to attacks that exploit early-token generation^[1].