BiLoCo: Binary Low-Rank Corrections for LLM FP4 Decode

Post-training quantization (PTQ) to FP4 has emerged as a key technique for reducing large language model inference cost, especially on Blackwell GPUs which offer native FP4 tensor core support. Direct PTQ to FP4, however, still leaves a noticeable accuracy gap. Recent work on SVDQuant mitigates the errors introduced through FP4 quantization for diffusion models by adding high-precision low-rank corrections obtained via SVD. However, this high-precision low-rank correction requires many bits per rank, and moreover competes on tensor core utilization with the main FP4 GEMM. We introduce BiLoCo, a correction whose left and right factors are stored as binary $\{\pm 1\}$ sign vectors rather than as BF16 numbers. The signed format uses $16\times$ less storage per rank-one term, and since the correction now consists of additions/subtractions, this can be run on CUDA cores concurrently with the FP4 tensor-core GEMM. On 4–32B LLMs, BiLoCo matches or improves memory-matched SVDQuant for PTQ. We implement BiLoCo on B200 with a correctness-preserving CUDA schedule that runs end-to-end decode at $1.13$–$1.15\times$ FP4-only, with the residual gap coming from the final output combine rather than from one-bit arithmetic.