acceptodds
Under review as a conference paper at ICLR 2027

PolyLoRA: Discrete-Continuous Transversal Updates for Quantized Diffusion Adaptation

Abstract

Large-scale diffusion models are often customized via low-rank adaptation and then quantized for deployment to reduce memory costs. However, a fundamental mismatch remains. The adapter is optimized in continuous space but deployed after discrete integer merge. This leads to two failures. First is semantic facet sticking, where updates that reduce the training loss may not cross a quantization boundary and leave the deployed model unchanged. Second is asymmetric merging distortion, where even when a boundary is crossed, the base integer anchor non-uniformly suppresses or amplifies the update. We propose PolyLoRA to reshape the adaptation trajectory along transversality in polyhedral space for deployable semantic changes. It introduces three lightweight mechanisms. Forward-Aligned Crossing promotes transverse boundary crossing in the actual pre-merge adapter, directly countering facet sticking. Merge-Anisotropy Calibration weights these crossings according to the integer-merge geometry, steering updates toward coordinates where post-merge leverage is strong to mitigate distortion. Semantic Transversality Shaping further uses an activation-derived Jacobian-Gram proxy to align merge-retained displacements with denoising-relevant response patterns, ensuring that survived updates actually benefit generation. These mechanisms are coupled with the denoising loss via a dual-barrier strategy at no inference overhead. By enforcing transversality, PolyLoRA ensures that optimizer updates cross quantization boundaries and survive integer merge in semantically relevant directions. Under W4A8 DreamBooth, PolyLoRA further improves DINO and CLIP-I over IntLoRA by 0.0162 and 0.0083, respectively, while preserving the direct integer-merge inference path.

Primary area:
generative models
Keywords:
Diffusion Models, Model Quantization, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dNn7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PolyLoRA starts from a problem with quantized diffusion models: an update learned during LoRA training may have little effect on the model that is actually used. Some changes disappear during rounding, and integer merging can weaken those that remain. The authors build on IntLoRA to address these two issues, using FAC to encourage changes across rounding thresholds and MAC to give more weight to changes that have a stronger effect after merging. Getting an update through these steps does not necessarily make it useful for denoising, though, so STS also uses layer activations to guide the update directions. These goals are incorporated into the training loss through a regularizer and an adaptive barrier, leaving IntLoRA’s integer merge scheme in place. In the reported experiments, PolyLoRA does better than the quantized baselines on both subject personalization and style customization.

Critical strengths

I see the deployment perspective as one of the paper’s strengths. A changed integer code is not enough on its own; the change also has to make it through the merge and be useful for denoising.

A change in the adapter code can have very different effects after merging, depending on the integer anchor. MAC takes this into account by giving more weight to coordinates where a code change has a larger effect on the merged weights. Since this weighting is used during training, it also fits into IntLoRA without changing the basic integer merge procedure.

Critical weaknesses

My concern is with how the crossing mechanism is validated. Appendix D measures movement from an EMA reference and counts anything above half a quantization step as a crossing. Since that reference need not lie at the center of a quantization cell, passing this test does not necessarily change the rounded code. For a unit step, moving from −0.4 to 0.4 passes the test, although both values round to 0. The reverse can also happen: moving from 0.49 to 0.51 changes the code with a displacement of only 0.02. This leaves me unsure what Table 5 tells us about actual crossings. Comparing the rounded codes between consecutive training steps would help show whether FAC really makes those crossings more frequent.

Questions for authors

In the provided code, change_main is detached before the barrier is computed. Was this intentional, and was this version used for the reported experiments? With no direct gradient through the score, what role does the barrier play when λtr=0\lambda_{\mathrm{tr}}=0?

Table 5 measures movement relative to an EMA reference. How often do the rounded codes actually change during training, and does adding FAC increase that rate? Are the runs with more code changes also the ones with better image quality?

What differs between the experiments in Tables 1 and 4? Under W4A8, full PolyLoRA has a DINO score of 0.4758 in Table 1 and 0.6032 in Table 4.

Reviewer_LaYwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose PolyLoRA which reshapes the adaptation trajectory in a “polyhedral transversality” view to address the core mismatch in quantized diffusion adaptation. The method comprises three lightweight mechanisms: Forward-Aligned Crossing (FAC), Merge-Anisotropy Calibration (MAC), and Semantic Transversality Shaping (STS). Extensive experiments show consistent gains over prior quantized LoRA baselines, especially under W4A8 DreamBooth and style customization.

Critical strengths
  • The work addresses an important problem at the intersection of PEFT and quantization for diffusion models to make low-bit adapters effective after integer merge.

-This paper identifies and formalizes two deployment-specific failure modes with a simple but compelling proposition connecting cell-crossing to deployed weight changes.

  • Evaluates on subject-driven and style-customization settings under both W8A8 and challenging W4A8 regimes shows consistent improvements over existing strong baselines.
Critical weaknesses
  • Evaluation is limited to SDv1.5 with relatively small-scale personalization/style benchmarks. no results on SDXL/DiT backbones where quantization behavior may differ.

  • The authors only compare under quantization bits-width setup and do not propvide a practical runtime speedup comparison with different methods.

  • The evaluated downstream task is restricted in subject-driven generation and style customization, which may impact the generalization assessment of the proposed method.

Questions for authors

Why there is a numerical inconsistency across tables: the W4A8 DINO score for PolyLoRA differs between the main table (0.4758) and ablation table (0.6032)?

Reviewer_SYmjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PolyLoRA for customizing diffusion models with low-bit adapters that are merged directly with quantized base weights. It addresses updates that disappear after rounding or have little effect after integer merging. The proposed training objective encourages quantization-boundary crossings, accounts for the merge geometry, and uses input activations to guide the resulting updates. These components are combined with the denoising loss through a dual-barrier formulation. Experiments cover subject and style customization, including controlled image comparisons, component ablations, and evaluation on additional backbones such as FLUX.

Critical strengths
  1. The paper identifies a concrete problem in low-bit adaptation: changing the continuous adapter parameters does not necessarily change the deployed weights. Accounting for both rounding and integer merging gives the method a clear practical motivation, while retaining the existing inference procedure.

  2. The image comparisons provide useful evidence of fine-grained customization. The clock-ring and shoe-stripe examples test whether changes in the prompt remain visible while preserving the subject’s structure. The subject and style evaluations, together with the FLUX results, support the usefulness of the approach beyond individual examples.

Critical weaknesses
  1. The evidence for the boundary-crossing explanation is incomplete. The reported diagnostic checks whether displacement from an EMA reference exceeds half a quantization cell. This can miss a small update that crosses a nearby boundary and count a larger update that leaves the rounded code unchanged. An increase in this diagnostic therefore does not necessarily mean that more actual boundary crossings occur.

  2. The interpretation of STS as denoising sensitivity needs stronger support. Table 10 reports several diagnostics, but I could not determine how "relative deployed gain" is computed or how it relates to changes in the denoiser’s output. It is also unclear how the cross-module entries of GG represent semantic diversity when each module uses its own activation basis.

  3. The relationship between the main results and the component ablation is unclear. The full W4A8 model obtains DINO/CLIP-I scores of 0.4758/0.7035 in Table 1 and 0.6032/0.7814 in Table 4. The paper needs to explain the evaluation differences behind this gap so that the contribution of each component can be interpreted alongside the main results.

Questions for authors
  1. How often does the half-cell diagnostic agree with actual changes in the rounded codes? What proportion of crossings does it miss, and how often does it count a crossing that did not occur?

  2. How is "relative deployed gain" in Table 10 defined? Does it directly measure a change in the denoiser’s output or error?

  3. What makes the response vectors from independently constructed activation bases comparable when forming GG?

  4. Which evaluation settings differ between Tables 1 and 4, and how do they account for the substantially different scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.