SOURCE-LINKED INTELLIGENCE

Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Selecting a retrieval model for a production RAG system requires reliable comparative evaluation, but obtaining relevance judgments at scale is expensive and difficult to repeat as new candidate systems arrive. We study pooled LLM evaluation, in which an LLM judges the union of documents retrieved by the current set of candidate systems, and the pool is then expanded incrementally as new systems are introduced by judging only the new documents they contribute. These judgments are reused to evaluate all systems on a common basis. We validate this approach on four retrieval benchmarks with 11 sy

Read original source ↗ Open in workspace

recordType

paper

region

Global