SOURCE-LINKED INTELLIGENCE

LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 12, 2026 · UTC

We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with a lightweight per-layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters), training both jointly under a gate-regularized language-modeling objective. Across 10 independently-seeded training runs, LayerRoute converges to an identical skip-pattern structure in every run - a consistent

Read original source ↗ Open in workspace

recordType: paper
region: Global

Evidence & attribution

arXiv · AI, language, vision and robotics · 2026-09-12T03:30:59.000Z

First collected: 2026-09-20T16:41:15.630Z. This is not the publication date.