SOURCE-LINKED INTELLIGENCE

RAPTOR: Role-Aware Private Training for Mixture-of-Experts

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

Differentially private (DP) fine-tuning methods treat sparse Mixture-of-Experts (MoE) models as a single dense block, ignoring that shared layers see all data while experts only see routed records. We identify and formally characterize three resulting failure modes: global clipping suppresses expert gradients, batch-level normalization dilutes sparse expert updates, and fixed privacy noise degrades signal-to-noise ratio on low-load experts. We introduce RAPTOR - a Role-Aware Private Training framework, which alternates shared and expert optimization and targets each failure directly, using exp

Read original source ↗ Open in workspace

recordType

paper

region

Global