moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT

The article presents moBERTo, an encoder-only transformer model specifically adapted for Portuguese by continuing the pretraining of the ModernBERT-base checkpoint on a dataset of 60 billion tokens. Key architectural features include rotary positional embeddings and alternating local-global attention, while evaluation results indicate that moBERTo achieves superior performance in information retrieval and other NLP tasks, particularly with long-context capabilities of up to 8,192 tokens. This model is significant for practitioners as it demonstrates the effectiveness of continued pretraining and tokenizer adaptation in enhancing performance on language-specific tasks, and the model weights and training data are publicly available for further research and application.