SOURCE-LINKED INTELLIGENCE

Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Large language models are typically trained under uniform token weighting, which allows frequent and low-information tokens to dominate learning and can increase the tendency to memorize surface-level text spans. To address this, we present an information-weighted cross-entropy loss that rescales token-level contributions using TF-IDF statistics, emphasizing semantically informative tokens while down-weighting ubiquitous ones. Experiments on five decoder-only LLMs ranging from 1.1B to 13B parameters show consistent reductions in memorized substring length while preserving perplexity and downst

Read original source ↗ Open in workspace

recordType

paper

region

Global