SOURCE-LINKED INTELLIGENCE

FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 15, 2026 · UTC

Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference. FlexEE makes early exiting practical for LLM decoding through layer-wise exit supervision for reliable inte

Read original source ↗ Open in workspace

recordType

paper

region

Global