AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani

The article introduces AmchiBias, a benchmark designed to assess socio-cultural stereotypical bias for Goan identity groups, featuring 313 minimal pairs across eight sociodemographic dimensions in both English and Konkani. Evaluation of five multilingual encoder models revealed near-chance performance in Konkani, indicating inadequate language competence and cultural representation in existing models. This highlights the need for improved multilingual NLP systems that can accurately reflect hyperlocal identities, which is crucial for practitioners focusing on low-resource language settings.