- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PolyLoRA starts from a problem with quantized diffusion models: an update learned during LoRA training may have little effect on the model that is actually used. Some changes disappear during rounding, and integer merging can weaken those that remain. The authors build on IntLoRA to address these two issues, using FAC to encourage changes across rounding thresholds and MAC to give more weight to changes that have a stronger effect after merging. Getting an update through these steps does not necessarily make it useful for denoising, though, so STS also uses layer activations to guide the update directions. These goals are incorporated into the training loss through a regularizer and an adaptive barrier, leaving IntLoRA’s integer merge scheme in place. In the reported experiments, PolyLoRA does better than the quantized baselines on both subject personalization and style customization.
Critical strengths
I see the deployment perspective as one of the paper’s strengths. A changed integer code is not enough on its own; the change also has to make it through the merge and be useful for denoising.
A change in the adapter code can have very different effects after merging, depending on the integer anchor. MAC takes this into account by giving more weight to coordinates where a code change has a larger effect on the merged weights. Since this weighting is used during training, it also fits into IntLoRA without changing the basic integer merge procedure.
Critical weaknesses
My concern is with how the crossing mechanism is validated. Appendix D measures movement from an EMA reference and counts anything above half a quantization step as a crossing. Since that reference need not lie at the center of a quantization cell, passing this test does not necessarily change the rounded code. For a unit step, moving from −0.4 to 0.4 passes the test, although both values round to 0. The reverse can also happen: moving from 0.49 to 0.51 changes the code with a displacement of only 0.02. This leaves me unsure what Table 5 tells us about actual crossings. Comparing the rounded codes between consecutive training steps would help show whether FAC really makes those crossings more frequent.
Questions for authors
In the provided code, change_main is detached before the barrier is computed. Was this intentional, and was this version used for the reported experiments? With no direct gradient through the score, what role does the barrier play when ?
Table 5 measures movement relative to an EMA reference. How often do the rounded codes actually change during training, and does adding FAC increase that rate? Are the runs with more code changes also the ones with better image quality?
What differs between the experiments in Tables 1 and 4? Under W4A8, full PolyLoRA has a DINO score of 0.4758 in Table 1 and 0.6032 in Table 4.