BERTomelo: Your Portuguese Encoder Best Friend

A team of researchers has released BERTomelo, a monolingual encoder for Portuguese built on the ModernBERT architecture and trained from scratch on a corpus of 106 million documents, according to a paper published on arXiv [1]. The model, introduced by authors including Luís Paulo Faina Garcia, is designed to address what the researchers describe as a gap in Portuguese natural language processing, where existing monolingual encoders such as BERTimbau and Albertina have lagged behind English benchmarks in scalability and efficiency [1][2]. BERTomelo is offered in Base and Large versions, both featuring a 1,024-token context window and hardware-level optimizations including FlashAttention and alternating attention mechanisms [1][4]. The model was pretrained on ClassiCC-PT, a high-quality Portuguese corpus that the authors say ensures alignment with contemporary usage of the language [1][2]. The Base variant contains 136 million parameters across 22 layers, while the Large variant holds 377 million parameters across 28 layers, according to model cards published on Hugging Face [5][6]. Pretraining encompassed 386.48 billion tokens, and the custom tokenizer was sourced from ModBERTBr and trained using a Unigram algorithm on the BrWaC corpus and Portuguese Wikipedia [5][6]. In benchmark evaluations, BERTomelo Large achieved a Named Entity Recognition F1 score of 91.05 percent, with recall at 92.75 percent and precision at 89.42 percent, outperforming mBERT, BERTimbau Large, and Albertina 900M on that task [5][6]. On Recognizing Textual Entailment, the Large model reached an F1 of 89.21 percent and accuracy of 89.26 percent [6]. For Semantic Textual Similarity, it recorded a mean squared error of 0.401 and a Pearson correlation of 0.849 [6]. The release comes as the Portuguese NLP community continues to build evaluation infrastructure. A separate initiative, the CLARIN-PT-LDB leaderboard, was recently launched to assess open large language models on European Portuguese using ten benchmarks, including novel tests for cultural alignment and model safeguards [9]. The BERTomelo authors note that standardized long-context benchmarks for Portuguese remain unavailable and that the current release does not yet incorporate a sequence packing mechanism [5][6].