SOURCE-LINKED INTELLIGENCE

Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

arXiv · AI, language, vision and robotics · article · Sep 15, 2026 · UTC

Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error.

Read original source ↗ Open in workspace

recordType

paper

region

Global