SOURCE-LINKED INTELLIGENCE

Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, eva

Read original source ↗ Open in workspace

recordType

paper

region

Global