Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels

A new study posted to arXiv finds that feature-learning kernel models only achieve delayed generalization, a phenomenon known as grokking, when a specific symmetry in the training data is broken. The work examines how the Recursive Feature Machine algorithm learns algebraic tasks. The paper, submitted on 31 Mar 2026 and revised on 24 Jun 2026, focuses on grokking, where a model reaches high training accuracy long before it begins to generalize to unseen test data [1][2]. This behavior was first documented on algebraic problems like modular arithmetic by Power et al. in 2022 [1][2]. The authors, led by Marcel Tomas Bernal, investigate this effect using the Recursive Feature Machine (RFM) algorithm, a method introduced by Radhakrishnan et al. in 2024 [1][2]. RFM iteratively updates feature matrices through the Average Gradient Outer Product (AGOP) of an estimator to learn task-relevant features [1][2]. The central experimental finding is that generalization occurs only when a certain symmetry in the training set is broken [1][2]. The study empirically demonstrates that RFM generalizes by recovering the underlying invariance group action inherent in the data, with the learned feature matrices encoding specific elements of that invariance group [1][2]. This explains the dependence of generalization on symmetry breaking [1][2]. The research was posted on arXiv, an open-access repository for electronic preprints that has been operating since August 14, 1991, and as of November 2024 receives about 24,000 submissions per month [7]. The platform hosts papers across mathematics, physics, computer science, and related fields, and its content is moderated but not peer-reviewed [7]. The paper's abstract page also links to several community-developed tools under the arXivLabs framework, a formalized collaboration space launched in 2020 that allows third-party developers to build features directly on the site [5][6]. These tools include the Bibliographic Explorer for navigating citation trees and the CORE Recommender for surfacing related open-access papers [5][6]. arXivLabs partners must adhere to the repository's values of openness, community, excellence, and user data privacy, and are granted only minimal, anonymized data access [5].