IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

Researchers have introduced two new benchmarks, TukaBench and IndoBias, to evaluate the safety and bias of Large Language Models (LLMs) in African and Indonesian languages.

TukaBench, introduced in a paper on arXiv^[1], is a jailbreak benchmark for seven African languages that assesses LLMs' safety across four settings, including human translation of existing prompts and culturally adapted prompts. The study found that prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to the least refusal^[1]. However, the evaluation also revealed structural limitations, including model comprehension failures and reduced reliability when using LLMs as judges in Low-Resource Languages. Meanwhile, IndoBias, presented in another arXiv paper^[2], is a bias benchmark for Indonesian and three local languages. It features dual evaluation tracks: depth-oriented and breadth-oriented. Existing LLMs showed strong bias towards prototypical sentences in Indonesian, while local languages suffered higher bias under the Ideology and Religion category^[2]. The study also found that Common Crawl texts introduce more bias during pretraining in Indonesian compared to human-reviewed article texts.