AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani

Researchers have introduced AmchiBias, the first benchmark designed to measure socio-cultural stereotypical bias in natural language processing models for the Indian state of Goa, a region with a distinct multicultural history [1]. The benchmark, detailed in a paper submitted to arXiv on June 13, 2026, addresses a gap in NLP evaluation, which often considers bias only at the national level, overlooking rich subnational socio-cultural structures [1]. AmchiBias covers various Goan identity groups and comprises 313 minimal pairs across eight sociodemographic dimensions, presented in both English and Devanagari Konkani [1]. The dataset allows researchers to probe whether language models hold stereotypical associations tied to hyperlocal community identities [2]. The authors evaluated five multilingual encoder models on the AmchiBias benchmark [1]. When queried in Konkani, the models returned near-chance scores, a result the paper attributes to language incompetence for general multilingual models and a lack of Goan cultural competence for Indian language models [1]. This finding highlights a critical gap in low-resource multilingual NLP evaluation for hyperlocal community identities [2]. In English-language queries, the performance shifted. Models with a stronger Indian language coverage showed higher bias for pan-Indian groups than for hyperlocal Goan groups [1]. The paper suggests this indicates that the English signal reflects pan-Indian pretraining associations rather than genuine Goan cultural knowledge [1]. The work underscores how large language models, which are trained on vast amounts of text through self-supervised learning, can absorb and reproduce biases present in their training data [7]. The release of AmchiBias comes as the machine learning community continues to develop tools for identifying and debugging biases in models. Platforms such as Hugging Face Spaces have collaborated with arXiv to embed interactive demos directly alongside papers, allowing a wider audience to explore research and identify issues without writing code [3][4]. Researchers can link their models and demos to arXiv papers through specific integrations, making it easier for others to reproduce results and probe for biases [5]. The AmchiBias paper's availability on arXiv places it within this ecosystem, where community-built demos can amplify the visibility of such evaluation frameworks [3].