SOURCE-LINKED INTELLIGENCE

Low-Rank Ternary Adaptation for Fine-Tuning Transformers

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to t

Read original source ↗ Open in workspace

recordType

paper

region

Global