SOURCE-LINKED INTELLIGENCE

Task-Aware Federated Fine-Tuning for MoE-based Large Language Models

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

Mixture-of-Experts (MoE) has become a widely adopted architecture for Large Language Models (LLMs), as it improves model capacity while limiting computational overhead through sparse expert activation. This property makes MoE-based LLMs particularly attractive for resource-constrained distributed environments. However, federated fine-tuning of MoE-based LLMs remains challenging under heterogeneous client data. Since clients often correspond to different task preferences, directly aggregating their local updates may weaken expert specialization and introduce conflicting update directions on sha

Read original source ↗ Open in workspace

recordType

paper

region

Global