Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation

A new fusion method called Bidirectional Cross-Attention Fusion (BCAF) aligns high-resolution RGB with low-resolution hyperspectral imaging for multimodal semantic segmentation, avoiding common preprocessing steps that degrade data quality, according to a paper published on arXiv [1]. The approach, detailed by Jonas Vilho Funk and colleagues, uses two independent backbones: a standard Swin Transformer for RGB and a hyperspectral-adapted Swin variant that preserves spectral structure through 3D tokenization with spectral self-attention [1]. The method aligns the two modalities at their native grids via localized, bidirectional cross-attention, bypassing pre-upsampling or early spectral collapse that can occur in other fusion pipelines [1]. On the benchmark SpectralWaste dataset, BCAF achieved 75.4% accuracy while processing 55 images per second [1]. The researchers also evaluated the method on a novel industrial dataset, K3I-Cycling, where it reached 62.3% mean intersection-over-union for material segmentation across categories such as paper, metal, and plastic, and 66.2% mIoU for plastic-type segmentation distinguishing PET, PP, HDPE, LDPE, and PS [1]. The first RGB subset of the K3I-Cycling dataset has been released on Fordatis [1]. The authors describe BCAF as modality-agnostic, applicable to any co-registered RGB input paired with a lower-resolution, high-channel auxiliary sensor [1]. Code and model checkpoints are publicly available on GitHub [1]. The paper appeared on arXiv in March 2026 and was last revised in June 2026 [1]. Hugging Face, a platform that indexes machine learning papers and associated models, datasets, and demos, has integrated with arXiv to embed interactive demos directly alongside paper abstracts [5]. This integration allows users to find and run community-built demos for papers like BCAF without writing code, provided the paper is linked from a Hugging Face repository or Space [5]. The platform also supports paper pages that aggregate related artifacts and enable authorship verification [4].