PolyLoRA: Discrete-Continuous Transversal Updates for Quantized Diffusion Adaptation

Large-scale diffusion models are often customized via low-rank adaptation and then quantized for deployment to reduce memory costs. However, a fundamental mismatch remains. The adapter is optimized in continuous space but deployed after discrete integer merge. This leads to two failures. First is semantic facet sticking, where updates that reduce the training loss may not cross a quantization boundary and leave the deployed model unchanged. Second is asymmetric merging distortion, where even when a boundary is crossed, the base integer anchor non-uniformly suppresses or amplifies the update. We propose PolyLoRA to reshape the adaptation trajectory along transversality in polyhedral space for deployable semantic changes. It introduces three lightweight mechanisms. Forward-Aligned Crossing promotes transverse boundary crossing in the actual pre-merge adapter, directly countering facet sticking. Merge-Anisotropy Calibration weights these crossings according to the integer-merge geometry, steering updates toward coordinates where post-merge leverage is strong to mitigate distortion. Semantic Transversality Shaping further uses an activation-derived Jacobian-Gram proxy to align merge-retained displacements with denoising-relevant response patterns, ensuring that survived updates actually benefit generation. These mechanisms are coupled with the denoising loss via a dual-barrier strategy at no inference overhead. By enforcing transversality, PolyLoRA ensures that optimizer updates cross quantization boundaries and survive integer merge in semantically relevant directions. Under W4A8 DreamBooth, PolyLoRA further improves DINO and CLIP-I over IntLoRA by 0.0162 and 0.0083, respectively, while preserving the direct integer-merge inference path.