Fidelity Is Not Selection: Low-Bit Optimizer States Across Codebooks and Bit Widths

Quantizing Adam’s persistent moment states reduces optimizer memory, but representations with the same nominal bit width can train differently, making exhaustive comparison expensive. A cheap selector is the error a representation introduces into the next Adam update. However, this score reads a reference state once, whereas training repeatedly applies the codec to states it has already encoded; whether its ranking remains valid for selection is unclear. We test this prospectively by locking each choice before candidate training across codebooks, bit widths, training working points, and an external low-bit optimizer. At a common 130M condition across 12 independent training units, the error retained the same ordering of our three representations, yet its choice was best by point estimate under a uniform map in all 12 units, intermediate under a floating-point map in all 12, and never best under a dynamic-exponent map, where it was worst in 11. In one paired 4+4-bit condition the ordering persisted and the mismatch did not recur. Similar fidelity values also accompanied sharply different stability outcomes at larger working points. In the external optimizer, the error rule beat the published default in all six units at the longer budget but missed the lowest-loss setting in five. Thus, the failure mode is not an unstable fidelity ranking, but a stable one whose training meaning changes across settings.