SOURCE-LINKED INTELLIGENCE

Polish ModernBERT: The Long and Short of Polish Language Understanding

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Encoder-only Transformers remain effective for discriminative and representation-learning tasks, yet Polish encoders still largely rely on BERT/RoBERTa-style architectures. We introduce \textbf{Polish ModernBERT}, a family of four Polish encoders available at Base and Large scales, each with 512-token and 8K context variants. We adapt the ModernBERT pretraining recipe through staged selection experiments and release a long-context benchmark covering legal topic classification, ideological decision-direction prediction, factual-consistency assessment over literary plot summaries, and human-righ

Read original source ↗ Open in workspace

recordType

paper

region

Global