On the Cross-Branch Correlation of Quantization Error in Classifier-Free Guidance

Classifier-free guidance runs a network twice per step, with and without the condition, and extrapolates along the difference of the two predictions. Quantized models are judged per branch, yet the guided error equals $\text{Ec}+s(\text{Ec}-\text{Eu})+s(1+s)V$, where $\text{Ec}$ and $\text{Eu}$ are the branch errors, $s$ is the guidance scale, and $V$, the mean squared difference of the two errors, depends on a cross term that no per-branch metric sees. In six inference stacks, $75.8\%$ of the activation quantizers give the branches different scales or random numbers, and on these, $19\%$ to $28\%$ of the values held by both branches receive different codes. We propose PairRound, which lets the branches share what decides their codes: the scale, and the random numbers where rounding is stochastic. Codes then differ only where values differ, and a quadratic in $s$ states when the coarser grid that sharing gives one branch pays off, a condition our measurements meet. At matched bits and seeds, the shared scale lowers the trajectory distance to the BF16 sampler by $60\%$ to $68\%$ on eleven image, video and language backbones, improves FID, VBench or perplexity on every one, and composes with eight published methods and improves each of them.