SOURCE-LINKED INTELLIGENCE

POSPAN: Position-Constrained Span Masking for Language Model Pre-training

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position co

Read original source ↗ Open in workspace

recordType

paper

region

Global