Frequency-Domain Latent Attention Gating for Cross-Domain Token Aggregation

Researchers have introduced FLaG, a plug-in module that performs token aggregation in the frequency domain, aiming to address a common bottleneck in models that map token representations to sample-level predictions [1]. The method uses a real FFT to transform tokens, summarizes spectral components with learnable latent queries, and applies a channel-wise gate before reconstructing enhanced time-domain tokens for final pooling [1][2]. The module was evaluated across three distinct domains: antimicrobial peptide (AMP) activity prediction using ESM2, image classification with ResNet18 on CIFAR-10 and CIFAR-100, and text classification with RoBERTa on IMDB and GLUE [1][2]. The clearest performance gains were observed on the ESM2-8M antimicrobial peptide tasks and on CIFAR-100, while the method remained competitive with established text baselines on IMDB and GLUE [1][2]. Probing experiments on the AMP setting revealed that low-frequency bands contribute the most to model performance [1][2]. The channel-wise gate functions as a broadly shared spectral reweighting stage, while cross-attention patterns are sample-specific with mild query-wise differentiation [1][2]. The study also found that higher-helix peptides exhibit stronger average spectral sensitivity in both bacteria tested [1][2]. Frequency-domain interventions in attention mechanisms have been explored in other recent work. A separate study introduced Attention Frequency Modulation (AFM), a training-free inference-time intervention that edits token-wise pre-softmax cross-attention logits via frequency-selective reweighting during denoising [3]. That work interprets attention-derived concentration maps as spatial signals on a latent grid, revealing a stable coarse-to-fine spectral progression over denoising steps [3]. In the domain of cross-domain few-shot segmentation, researchers proposed a Frequency-aware Matching Network (FAMNet) that performs support-query matching in specific frequency bands to reduce support-query bias [4]. The approach incorporates a Multi-Spectral Fusion module that uses mid-frequency information to extract domain-invariant features from high and low-frequency components while suppressing domain-variant information [4]. Another related direction is Energy-Gated Attention (EGA), which augments standard transformer attention by gating value aggregation according to the spectral energy of key token embeddings [5]. Motivated by turbulence theory and grounded in the Wiener–Khinchin theorem, EGA amplifies tokens with high spectral energy—corresponding to informationally dense positions such as syntactic heads and discourse markers—while suppressing low-energy background tokens [5]. The FLaG source code and supplementary materials have been released on the project website and GitHub repository [1][2].