SOURCE-LINKED INTELLIGENCE

Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

Retrieval-Augmented Generation (RAG) enhances language models with external knowledge, but the lengthy retrieved context inflates the input and degrades inference efficiency. Soft context compression encodes each document into a substantially shorter embedding sequence. However, most existing approaches are trained by distilling outputs from uncompressed RAG systems, inherently limiting their performance relative to the original model. To address this limitation, we propose DEX-Comp, a two-stage training recipe: Pure Distillation warm-starts the compression model on the uncompressed RAG's corr

Read original source ↗ Open in workspace

recordType

paper

region

Global