SOURCE-LINKED INTELLIGENCE

EFQ-Softmax: Exp-Free Quantization for Softmax

arXiv · AI, language, vision and robotics · article · Sep 9, 2026 · UTC

Low-bit attention accelerates Transformer inference by moving the $QK^\top$ and $PV$ matrix multiplications to FP8 or FP4 matrix engines. However, the softmax path often evaluates shifted-score exponentials in higher precision, forms a temporary probability block, and quantizes it before low-bit $PV$ multiplication. This exp-then-quantize path creates a mismatch between a high-precision probability producer and a low-bit matrix consumer. We propose EFQ-Softmax (Exp-Free Quantization for Softmax), a low-bit probability-generation method that directly maps shifted attention scores to block-scale

Read original source ↗ Open in workspace

recordType

paper

region

Global