acceptodds
Under review as a conference paper at ICLR 2027

Compressed Adaptation of Diffusion Transformers via Globally-Sensitive Sketch

Abstract

Adapting diffusion transformers under tight memory budgets requires both compression and parameter-efficient fine-tuning. Existing sketch-based compression methods use locally sensitive sketches, where each lookup table is tied to a predetermined spatial region and cannot be shared by statistically similar weight groups at different locations. We propose Globally-Sensitive Sketch (GSS), a unified framework for compression and parameter-efficient fine-tuning. GSS clusters weight groups within each weight matrix by normalized distributional shape, allowing similar groups to share learnable sketches across arbitrary matrix positions. Each weight stores a 4-bit index into its group's assigned sketch, and only the shared sketch entries are updated during fine-tuning. GSS is extensively evaluated on six diffusion transformers spanning 2B to 20B parameters. We consider two settings: adaptation for subject-driven personalization and paired image editing, and standalone text-to-image generation. GSS has the lowest mean weight quantization error among all evaluated 4-bit baselines and preserves text-to-image generation quality across all six backbones. In addition, GSS achieves competitive quality on both subject-driven personalization and paired image editing compared to full fine-tuning while using over 100 fewer trainable parameters. We provide our code at https://anonymous.4open.science/r/gss-anonymization/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Compression, PEFT, Sketching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8eXvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Globally-Sensitive Sketch (GSS), a framework for compressing and efficiently fine-tuning diffusion transformers under limited memory budgets. The key observation is that weight groups at different positions may have similar distributions, but existing sketch-based methods typically assign codebooks based on fixed spatial locations, preventing such groups from sharing parameters. To address this limitation, GSS clusters weight groups according to their normalized distributional shapes and allows similar groups to share learnable sketches across different positions. During fine-tuning, only the shared sketch entries are updated, while the quantization codes and routing assignments remain fixed, significantly reducing the number of trainable parameters.

The authors evaluate GSS on six diffusion transformers ranging from 2B to 20B parameters, covering subject-driven personalization, paired image editing, and standalone text-to-image generation. Experimental results show that GSS achieves competitive adaptation quality compared with full fine-tuning and existing PEFT methods while using substantially fewer trainable parameters. It also demonstrates strong 4-bit quantization performance, achieving lower weight reconstruction errors and preserving generation quality across different backbones. Additional ablation studies investigate the effects of sketch pool size and density-weighted fitting.

Critical strengths
  1. Novel and practical solution. The paper identifies an interesting limitation of existing sketch-based compression methods and proposes a simple yet novel solution through distribution-based global sketch sharing. I appreciate this kind of intuitive and effective approach that addresses a practical problem.

  2. Comprehensive experiments. The authors evaluate their method across six diffusion transformers of different scales and multiple tasks, with comparisons against existing baselines. The broad experimental coverage provides convincing evidence of the method's effectiveness and practical value.

Critical weaknesses
  1. Lack of qualitative visualization results. Personally, I believe that for a paper involving image generation, showing actual generated images is quite important. However, the authors do not provide any qualitative visual comparisons. I would really like to see how GSS performs visually compared to other methods, as numerical metrics alone may not fully reflect the perceptual quality of generated images.

  2. Insufficient ablation on the core design. The main motivation of this paper is distribution-based global sketch sharing, but the ablation studies mainly focus on sketch pool size and density-weighted fitting. There is no direct comparison with alternative routing strategies, such as random or position-based assignment, under the same sketch budget. This makes it difficult to isolate the actual contribution of global routing, which I consider an important missing piece in validating the core idea.

Questions for authors
  1. Qualitative visualization results. Could the authors provide some visual comparisons of generated images between GSS and the baseline methods? I believe such comparisons are important for image generation tasks, as they would help us better understand the actual perceptual quality and potential limitations of the proposed method.

  2. Ablation on global sketch sharing. Could the authors provide a more direct ablation study comparing the proposed distribution-based routing with alternative strategies, such as random or position-based assignment, under the same sketch budget? I think this would help clarify how much of the improvement comes specifically from the global routing mechanism rather than other design choices.

3 comments

Rebuttal for Reviewer 8eXv

We thank the reviewer for the thoughtful feedback and for highlighting the novel and practical design and the comprehensive experiments of our work. We address the comments and questions below, and we have revised the paper accordingly, with all changes highlighted in red.

[W1 - Qualitative visual comparisons] --> We add four image figures, covering every backbone and task.

A. Qualitative results. We add four figures to the appendix of the revised paper, covering every backbone and task. The first compares GSS, Full FT, LoRA, QLoRA and SketchTune on subject-driven personalization on all six backbones, and the second compares the same five methods on image editing with FLUX.1-Kontext. The third and fourth compare BF16, GSS, NF4, OrbitQuant and SketchTune on text-to-image generation after 4-bit quantization on all six backbones, one prompt per figure. Images compared side by side use the same prompt.

[W2 - Ablation of distribution-based routing] --> SketchTune already assigns by position, and we add a direct ablation against random and position-based assignment.

A. Our main baseline already assigns by position. SketchTune shares its tables by row position, and the paper compares GSS with it throughout.

B. A direct ablation at the same budget. We now also change only the assignment rule inside GSS. Every arm uses 1024 sketches per matrix, groups of 64 and 4-bit codes, and refits its pool the same way, so bits per weight and trainable parameters are identical. Random assignment gives each group a uniformly random sketch; position assignment gives each sketch one contiguous band of groups. We report the weight reconstruction error over all 456 linear layers of FLUX.1-dev:

FLUX.1-dev, 456 layersMean |W−Ŵ| ↓Std |W−Ŵ| ↓Top-1% mean |W−Ŵ| ↓
Distribution (GSS)2.37e-32.07e-34.29e-3
Random2.93e-3 (+24%)2.55e-3 (+23%)6.10e-3 (+42%)
Position2.92e-3 (+23%)2.52e-3 (+22%)6.00e-3 (+40%)

As the table shows, distribution-based routing gives the lowest error on every metric. Random and position assignment perform about the same as each other, and both raise the mean weight error by 23-24%, its standard deviation by 22-23%, and the mean error on the largest 1% of errors by 40-42%. The gain therefore comes from routing groups by distribution, not from the shared pool alone.

[Q1 - Qualitative visualization results] --> We add four image figures, covering every backbone and task.

See W1.

[Q2 - Ablation of global sketch sharing] --> SketchTune already assigns by position, and we add a direct ablation against random and position-based assignment.

See W2.

Thank you for the detailed response and for conducting the additional experiments. I appreciate the authors' efforts in addressing my concerns.

The newly added qualitative comparisons and the controlled ablation study on different routing strategies have largely addressed my main concerns, particularly regarding the effectiveness of distribution-based global sketch sharing.

Given these additional results and clarifications, I am satisfied with the authors' response and will raise my score accordingly. I appreciate your efforts and wish you the best of luck!

Dear Reviewer 8eXv,

Thank you so much for your time, thoughtful feedback, and for raising your score! We are glad the additional experiments and clarifications addressed your concerns. Your comments helped us improve the paper significantly, and we truly appreciate your constructive engagement throughout the review process.

Sincerely, Paper51039 authors

Reviewer_UHdgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Globally-Sensitive Sketch (GSS), a unified framework for 4-bit compression and parameter-efficient fine-tuning of diffusion transformers. Unlike prior sketch-based methods that tie each lookup table to a fixed row, GSS normalizes weight groups and clusters them by distributional shape, so that similar groups anywhere in a matrix share learnable sketches from a matrix-wide pool. During fine-tuning, codes, scales, and assignments stay frozen, and only the shared sketch entries are updated. Across six diffusion transformers from 2B to 20B parameters, GSS achieves the lowest mean weight quantization error among the evaluated 4-bit baselines and competitive personalization and editing quality with far fewer trainable parameters than full fine-tuning.

Critical strengths
  1. The paper unifies compression and PEFT in a single representation. Since the sketches within the compressed representation are trained directly without a separate adapter, neither training nor inference requires a full-precision backbone. The design is simple and conceptually clear.
  2. The paper provides an actual system implementation and cost measurements. The authors implement forward and backward CUDA kernels that operate directly on the packed representation, and Appendix B reports memory and time for both training and inference.
  3. The hyperparameter search protocol is clearly specified. The same number of learning rates (16) is searched for all methods, and the checkpoint selection procedure with separate validation and test splits is described in Algorithm 3.
Critical weaknesses
  1. The efficiency claims are not well supported by the measurements. According to Appendix B, the training step time of GSS is 1735.8 ms, which is slower than LoRA and QLoRA, making it the slowest among the PEFT methods. Its inference memory (11.6 GiB) is larger than that of QLoRA, and its inference time (10.8 s) is also longer than that of QLoRA. Compared with QLoRA, its most direct competitor, GSS only slightly reduces training memory while being disadvantaged in speed and inference memory. The paper emphasizes the small number of trainable parameters, but this does not translate into faster training in practice, which limits its significance.

  2. The reliability of the adaptation experiments is limited. (a) Full fine-tuning obtains substantially lower LLM scores than the PEFT methods on almost all backbones. This suggests that full fine-tuning may not have been properly tuned, which makes it difficult to claim that GSS is comparable to full fine-tuning based on this baseline. (b) The rank of LoRA and QLoRA is fixed at 4, which the authors themselves state was chosen for convenience. Since the rank has a large impact on baseline performance, comparisons with higher ranks are also necessary.

  3. The limited expressiveness of training only the sketches is not discussed. During fine-tuning, only 1024 × 16 values per matrix are updated, and all weights that share the same sketch move together in the same direction. There is no analysis of how such a heavily constrained update is sufficient for injecting localized knowledge, as required in subject personalization. In addition, no qualitative results comparing generated images are provided, even in the appendix, so visual comparison is not possible.

  4. The novelty is limited. Grouping weights and assigning codebooks shared across the entire matrix through clustering, regardless of position, is conceptually close to vector quantization methods such as AQLM and GPTVQ. Although GSS differs in direction from these VQ methods, the idea of not assigning sketches by position is itself not particularly novel.

Questions for authors
  1. Could you report the total training time until convergence, together with the number of training steps for each method, rather than only the per-step time? Could you also explain why GSS trains more slowly than QLoRA and report the offline compression time?

  2. Could you compare against LoRA and QLoRA with higher ranks, including ranks that match the number of trainable parameters of GSS?

  3. Could you analyze the expressiveness of updates that only modify the shared sketches, for example by applying LoRA on top of the GSS-compressed backbone or evaluating on more challenging adaptation tasks?

  4. Could you clarify how GSS differs from existing approaches that share codebooks across a matrix regardless of position, such as vector quantization methods?

5 comments

Rebuttal for Reviewer UHdg (Part 1/n)

We thank the reviewer for the thoughtful feedback and for highlighting the unified compression and fine-tuning design, the system implementation, and the clear tuning protocol of our work. We address the comments and questions below, and we have revised the paper accordingly, with all changes highlighted in red.

[W1 - Efficiency claims] --> GSS targets training memory and quality; its time cost is comparable, and int16 ids close part of the inference-memory gap.

A. Training memory and quality matter the most. We agree that GSS is not the fastest method. Our goal is to make users able to personalize large models with good quality. We explain in more detail below:

  • Training memory is what stops users from personalizing large models. Whether a model can be fine-tuned on a given GPU is decided by peak memory, not by speed. GSS has the lowest training memory of all methods, so it brings personalization of large DiTs to smaller GPUs.
  • Training time is comparable, and it is not the bottleneck. GSS takes 1735.8 ms per step, against 1304.4 ms for QLoRA. A PEFT personalization run finishes in minutes either way, so this gap matters far less than whether the run fits in memory.
  • Inference time is comparable. The difference in inference time is negligible in practice, and users care more about the quality of the generated images than about a small difference in latency. This paper targets quality and training memory, and our kernels are not yet optimized for speed: GSS has a weight reconstruction stage before each matrix multiplication, which leads to extra memory reads/writes.
  • Speed can be improved with fused kernels. Fused kernels, in the spirit of FlashAttention, could further cut both training and inference time. That would be the subject of a separate systems paper, and we see it as follow-up work.

B. Inference memory. Part of the gap came from our kernels storing the cluster IDs as int32, which costs 0.5 bits per weight. A cluster ID only needs 10 bits (K=1024K = 1024), so we now store it as int16. This brings GSS to 4.51 bits per weight on FLUX.1-dev, the same as NF4 (4.50), and saves 0.29 GiB. On FLUX.1-Kontext at 1024 px, the GSS transformer weights now take 8.7 GiB, the same as NF4 in QLoRA, and peak inference memory of the transformer is 11.3 GiB against 11.2 GiB for QLoRA (Appendix, Memory and Latency).

[W2 - Full fine-tuning and LoRA rank] --> Full FT is easy to overfit and hard to tune, and every method gets the same tuning budget.

A. Full FT is easy to overfit and hard to tune with hyperparameter search.

  • Updating all of a DiT's billions of weights from a handful of subject images easily overfits and causes language drift [1-3]. Full FT is not guaranteed to outperform PEFT methods.
  • Full FT is also the most expensive method to tune. It requires lots of GPU memmory and time.
  • Nevertheless, Full FT gets the best results on three backbones, Qwen-Image, SD3.5-large and Lumina-Image-2.0. A badly tuned baseline would not lead on any backbone.

B. Every method gets the same number of trials to ensure fairness.

  • We have done our best to ensure a fair comparison by giving every method the same tuning budget: 16 learning-rate trials of up to 750 steps each.

The rank of LoRA and QLoRA is discussed in Q2.

[W3 - Expressiveness of sketch-only updates and qualitative results] --> Our experiments show that GSS adapts well, and we add an image gallery.

A. A constrained update is enough for PEFT, and its expressiveness is shown by experiments. Fine-tuning large models has a low intrinsic dimension: even an update restricted to a small random subspace reaches most of the full fine-tuning quality [7]. Existing theory of PEFT expressiveness, such as that for LoRA [10], holds only under idealized assumptions, and what an update learns in practice depends on the nonconvex training dynamics of a nonlinear network with billions of parameters. The effectiveness of a PEFT method, LoRA [4] included, therefore has to be justified by experiments, and we justify GSS the same way. As the main results in our paper show, GSS adapts well.

B. Qualitative results. We add two figures to the appendix of the revised paper, covering every backbone and task in Table 1. The first shows personalization on all six backbones, with the reference photo followed by the images from GSS, Full FT, LoRA, QLoRA and SketchTune. The second shows image editing on FLUX.1-Kontext, with the source image followed by the edits from the same five methods. Every image in a row uses the same prompt.

Rebuttal for Reviewer UHdg (Part 2/n)

[W4 - Novelty relative to vector quantization] --> GSS clusters weight groups by distribution shape, which makes its pool compact enough to fit in on-chip GPU memory.

A. Our novelty: clustering by distribution shape. GSS clusters groups of weights by the shape of their value distribution, with scale and order removed, into one pool of 16-level grids shared across the whole matrix. Each weight keeps its own 4-bit code on its group's grid. Vector quantization, such as AQLM [8] and GPTVQ [9], instead encodes blocks of weights by their values position by position, so blocks with a similar distribution but a different order receive different codes.

B. Our unique advantage: a compact pool that still gives the best approximation quality. Because each weight's own code handles the order and the group scale handles the magnitude, a grid only has to match a distribution shape. So 1024 grids of 16 values are enough for a whole matrix, and GSS still has the lowest mean, standard deviation and top-1% mean weight quantization error of all 4-bit baselines on every backbone, as discussed in Section 3.2. Vector quantization cannot separate out the order this way, so its accurate settings need far larger codebooks:

Shared entries per matrixValues per entryTotal shared values per matrixSize in 16-bit
GSS1,024 grids1616,38432 KB
AQLM [8]65,536 vectors8524,2881 MB

C. The whole pool fits in fast on-chip GPU memory. Every weight is decoded by looking up its grid, a random read into the pool. At 32 KB, the GSS pool fits in the shared memory of one streaming multiprocessor (228 KB on an H100), so these lookups, and the gradient accumulation into the pool during fine-tuning, stay on-chip. AQLM's 1 MB codebook does not fit, so its lookups go to off-chip memory, which slows down both inference and training.

[Q1 - Time to convergence, why GSS is slower, and compression time] --> We report total training time, explain the slowdown, and report compression time.

A. Total training time. The total training time is simply the time per step multiplied by the number of steps. Of the two, only the time per step is a property of the method: the number of steps depends on the task, the data and the hyperparameters. Under our protocol, every trial of every method trains for at most 750 steps. The table gives the time of these 750 steps on FLUX.1-Kontext, excluding screening evaluations:

MethodSubject-driven personalization (min)Image editing (min)
Full FT (2 GPUs)30.015.6
LoRA13.28.4
QLoRA16.311.1
SketchTune16.111.3
GSS21.714.2

B. Why GSS trains more slowly than QLoRA. The extra time comes from the sketch update. Unlike QLoRA, whose base weights receive no gradient, GSS updates the shared sketches, so every backward pass needs an extra matrix multiplication that forms the full weight gradient, writes it to GPU memory, and reduces it into the sketch gradient. A FlashAttention-style fused kernel could keep this work on chip, decoding weights in registers and accumulating the weight-gradient tiles directly into the sketch gradient. The extra matrix multiplication would remain, so such a kernel would reduce the overhead rather than remove it. We leave it to future systems work.

C. Offline compression time. GSS compresses a backbone once, offline. To keep this fast, we run every k-means step with fast implementations: the exact 1-D k-means that summarizes each group and fits each grid is solved by dynamic programming in C++, and the clustering of groups into the 1024 grids of a matrix runs on the GPU with FlashKMeans. As a result, compressing any of the six backbones takes only minutes, and it only has to be done once:

Model (one H100 GPU)Linear layersGSS compression time (min)
FLUX.1-dev45610.5
FLUX.1-Kontext4569.2
Qwen-Image84010.7
SD3.5-large5294.7
SD3-medium3331.6
Lumina-Image-2.02382.2

Rebuttal for Reviewer UHdg (Part 3/n)

[Q2 - Higher and parameter-matched LoRA/QLoRA ranks] --> We now run LoRA/QLoRA at ranks that match GSS's trainable parameters; a higher rank brings no consistent gain, and a very high rank is not free in memory.

A. Rank is a design choice, and we keep the comparison fair through the tuning budget. Choosing the rank per layer or per task is a research problem in its own right [5, 6], so most work fixes a small rank, and a larger rank is not reliably better: on language tasks, LoRA [4] finds that ranks as small as 1 or 2 match much larger ones. What keeps the comparison fair is that every method gets the same tuning budget (W2).

B. LoRA and QLoRA at matched trainable parameters. We now train LoRA and QLoRA at the smallest rank, and no less than 4, that gives them at least as many trainable parameters as GSS, and use these runs in Table 1. On FLUX.1-dev and FLUX.1-Kontext, rank 4 already has 11.2 M trainable parameters, 1.5 times the 7.5 M of GSS. On the other four backbones, the matched rank is 5 to 10. Going from rank 4 to the matched rank changes the LLM score with no consistent direction, and GSS has a higher LLM score than both LoRA and QLoRA on all six backbones. Part 4 gives the numbers for each backbone.

C. A very high rank is not free in memory, and Table 1 already covers its upper limit. Training keeps four copies of every trainable parameter: its weight, its gradient, and the two AdamW moments. On FLUX.1-dev, each unit of rank adds 2.8 M trainable parameters, so even with all four copies in bfloat16, rank 16 adds 0.33 GiB, rank 64 adds 1.3 GiB, and rank 128 adds 2.7 GiB, before counting the activations of the adapter branch. The NF4 backbone that QLoRA trains on takes 8.7 GiB, so rank 128 adds about 30% on top of it and gives back much of what 4-bit compression saves. The limit of increasing the rank is full fine-tuning, which updates every weight without any rank constraint, and Table 1 compares GSS with it on all six backbones.

[Q3 - Analysis of the expressiveness of sketch-only updates] --> Our experiments already include a harder adaptation task, and adding LoRA on top would measure LoRA rather than GSS.

A. Our experiments already test sketch-only updates on a harder task. Besides personalization on six backbones, the paper evaluates instruction-based image editing, which changes the model's behavior far more broadly than learning a single subject.

B. LoRA on top of GSS is possible but not necessary, as sketch-only updates already work well without it. GSS replaces each linear layer, so a LoRA adapter can be added on top of it, just as QLoRA adds one on top of NF4. Doing so only enlarges the space of updates, so it would not make GSS less expressive. Sketch-only updates already obtain a higher LLM score than both LoRA and QLoRA on all six backbones, even with LoRA and QLoRA given at least as many trainable parameters as GSS, so adding LoRA would bring extra trainable parameters without a clear need.

C. Keeping GSS alone gives a clean answer. Our claim is that GSS by itself has strong compressed adaptation capability. With LoRA on top, it would be hard to tell whether a gain comes from the sketches or from the adapter, and the adapter would add memory for its weights, gradients and optimizer states.

[Q4 - Difference from vector quantization (AQLM, GPTVQ)] --> GSS is scalar quantization with shared, distribution-assigned grids.

See W4.

[1] Ruiz et al., DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR 2023.

[2] Kumari et al., Multi-Concept Customization of Text-to-Image Diffusion (Custom Diffusion), CVPR 2023.

[3] Han et al., SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, ICCV 2023.

[4] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.

[5] Zhang et al., Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (AdaLoRA), ICLR 2023.

[6] Valipour et al., DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation, EACL 2023.

[7] Aghajanyan et al., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL 2021.

[8] Egiazarian et al., Extreme Compression of Large Language Models via Additive Quantization (AQLM), ICML 2024.

[9] van Baalen et al., GPTVQ: The Blessing of Dimensionality for LLM Quantization, arXiv:2402.15319, 2024.

[10] Zeng and Lee, The Expressive Power of Low-Rank Adaptation, ICLR 2024.

Reviewer_GjLtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Globally-Sensitive Sketch (GSS), which provides a 4-bit weight compression method and enables parameter-efficient fine-tuning for diffusion transformers (DiTs). A weight matrix is divided into groups that share and reuse a fixed number of codebooks. The allocation is based on distributional shapes of each group instead of its location. Density-weighted sketch optimization is further proposed to improve the compression accuracy on rare or outlier values. The authors evaluate the method on six DiT-based backbones in two settings: (1) Compressed adaptation for subject-driven personalization and paired image editing. (2) Post training quantization on MJHQ and sDCI.

Critical strengths
  1. The paper is well-motivated. It is reasonable to make codebook assignment of weight groups depend on their distribution rather than their positions, so that similar groups in different rows can share one sketch and the number of sketches no longer grows with the row count.
  2. The method is intuitive and data-free, and is specified clearly in Algorithms 1-2. The motivation of each step of the compression algorithm is clearly stated.
  3. The experimental setup across 6 DiT-based backbones and two settings is extensive.
Critical weaknesses
  1. Besides the 4-bit codes, the algorithm also introduces some metadata like sketch pool LL, scale SS, cluster id II, etc, which may cause additional burden of GPU memory. However, the bits per weight is not reported in the paper. Can the authors please provide the relevant information and evaluate if the method outperforms the baselines with the same or comparable bits per weight?
  2. The details of the LLM judge in Table 1 are not provided, including which LLM is used and with what prompts and inputs.
Questions for authors

The method looks model-agnostic. I am curious why the authors focus on DiTs only. SketchTune already reports results on LLMs, so an LLM experiment would give a direct comparison on its original benchmarks.

3 comments

Rebuttal for Reviewer GjLt

We thank the reviewer for the thoughtful feedback and for highlighting the clear motivation, the intuitive data-free design, and the extensive experiments of our work. We address the comments and questions below, and we have revised the paper accordingly, with all changes highlighted in red.

[W1 - Bits per weight and comparison at matched storage] --> We report the bits per weight of GSS and of every baseline.

We break the storage of GSS into its four parts, in bits per weight, on each of the six backbones.

ModelCodes CCScale SSCluster ID IISketch pool LLEffective bits per weight
FLUX.1-dev4.0000.2500.2500.0124.51
FLUX.1-Kontext4.0000.2500.2500.0124.51
Qwen-Image4.0000.2500.2500.0114.51
SD3.5-large4.0000.2500.2500.0174.52
SD3-medium4.0000.2500.2500.0434.54
Lumina-Image-2.04.0000.2500.2500.0244.52

We compare the effective bits per weight of GSS against each baseline. All methods use between 4.0 and 4.6 bits per weight, so their storage is comparable, and GSS stays within 0.04 bits of NF4 on every backbone. At this comparable bit width, GSS gives the best weight approximation, with the lowest mean, standard deviation and top-1% mean weight error on every backbone (Section 3.2).

ModelNF4SketchTuneOrbitQuantGSS
FLUX.1-dev4.504.304.084.51
FLUX.1-Kontext4.504.304.084.51
Qwen-Image4.504.284.004.51
SD3.5-large4.504.354.014.52
SD3-medium4.504.564.014.54
Lumina-Image-2.04.504.424.014.52

[W2 - Details of the LLM judge] --> We give the judge model, its inputs, and its prompts.

We list the judge model, its inputs, its prompt and its output for each of the two tasks in Table 1.

Personalization (DreamBooth)Editing (InstructPix2Pix)
Judge modelMuse Spark 1.2Muse Spark 1.2
Input imagesThe generated image and the subject's reference photosThe edited image and the target image
Input textThe generation prompt and the judge prompt belowThe edit instruction and the judge prompt below
Prompt"Rate the generated image from 0 to 10, jointly considering (a) how faithfully it preserves the subject identity in the reference photos, (b) how well it follows the prompt, and (c) overall visual quality. Reply with ONLY a single integer from 0 to 10 and nothing else.""Rate the edited image from 0 to 10 on how well it applies the requested edit while preserving the source structure and overall quality. Reply with ONLY a single integer from 0 to 10 and nothing else."
OutputOne integer, 0-10One integer, 0-10

We divide the judge's output score by 10 to normalize it to [0, 1].

[Q1 - Why DiTs only, and results on LLMs] --> We consider DiTs the harder setting, and leave LLMs to future work.

GSS can replace any nn.Linear, so it is indeed not specific to DiTs. We focus on DiTs because we consider them a harder target for quantization than LLMs, for two reasons:

  • Errors compound over sampling. A DiT applies the same weights at every one of its 20-50 denoising steps, so a quantization error is reapplied at each step and accumulates along the sampling trajectory [1].
  • Existing PTQ methods do not transfer. LLM PTQ methods such as GPTQ [5] and AWQ [6] calibrate on activations, but in diffusion models the activation distribution shifts across timesteps, so these methods work poorly there. GSS is data-free and is not affected by this shift. The dedicated diffusion methods [1-4] were developed on much smaller models and do not hold up on today's large DiTs.

We therefore believe DiTs show the advantages of GSS more clearly. We leave LLMs to future work. A fair evaluation of an LLM quantizer needs more than perplexity: general and reasoning abilities are known to be sensitive to quantization [7, 8], and measuring them properly requires a careful evaluation design of its own.

[1] Li et al., Q-Diffusion: Quantizing Diffusion Models, ICCV 2023.

[2] Shang et al., Post-training Quantization on Diffusion Models (PTQ4DM), CVPR 2023.

[3] Wu et al., PTQ4DiT: Post-training Quantization for Diffusion Transformers, NeurIPS 2024.

[4] Zhao et al., ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation, ICLR 2025.

[5] Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023.

[6] Lin et al., AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration, MLSys 2024.

[7] Li et al., Evaluating Quantized Large Language Models, ICML 2024.

[8] Liu et al., Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models, COLM 2025.

Thanks for the clear rebuttal, which resolves my main concerns. I remain positive and will keep my score.

Dear Reviewer GjLt,

Thank you for your time and thoughtful feedback, and for your continued positive assessment of our work! We are glad our rebuttal resolved your main concerns. We really appreciate your insightful comments, which helped us improve the paper.

Sincerely, Paper51039 Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Paper Revision Summary

Paper Revision Summary

We thank all the reviewers and the area chair(s) for their time and effort in helping us improve the paper. We are glad that the reviewers found the motivation clear (R GjLt) and the design novel and practical (R 8eXv), with an intuitive data-free formulation (R GjLt) that unifies compression and fine-tuning in one system implementation (R UHdg). The reviewers also appreciated the extensive and comprehensive experiments (R GjLt, R 8eXv) and the clear tuning protocol (R UHdg).

We have updated the paper to address all reviewer concerns. The major revisions are summarized below, with all changes highlighted in red, and the changes made during the discussion phase in blue.

  • [GjLt] added the bits per weight of GSS and of every baseline on all six backbones in Section 2.2 and Table 8 of Appendix C, showing that all methods use a comparable 4.0-4.6 bits per weight.
  • [GjLt] added the judge model, inputs, prompts and output of the LLM judge in Table 5 of Appendix B.
  • [GjLt] clarified why we focus on DiTs in Section 3.
  • [UHdg] reduced the cluster IDs from int32 to int16, which brings GSS to the same transformer weight memory as NF4 (Section 2.2, Figure 4 of Appendix C).
  • [UHdg] clarified the efficiency goals of GSS and explained why its training step is slower than that of QLoRA in Appendix C.
  • [UHdg] added the total training time of every method in Table 6 of Appendix C.
  • [UHdg] added the offline compression time of all six backbones in Table 7 of Appendix C, and described the fast k-means implementations in Section 2.3.
  • [UHdg] added a design for fused kernels as future work in Appendix C and Section 5.
  • [UHdg] updated Table 1 with LoRA and QLoRA at ranks that match the trainable parameters of GSS, and discussed full fine-tuning and the equal tuning budget in Section 3.1.
  • [UHdg] added a comparison with vector quantization (AQLM, GPTVQ) in Section 4, and clarified that the 32 KB sketch pool stays in on-chip shared memory in Section 2.4.
  • [UHdg, 8eXv] added qualitative results for every backbone and task in Figures 5-8 of Appendix E.
  • [8eXv] added an ablation of distribution-based routing against random and position-based routing in Section 3.3 and Table 9 of Appendix D.
AuthorsOpenReview ↗

Summary for Area Chair(s): Reviewers' Positive Feedback and Addressed Concerns (1/n)

Summary for Area Chair(s): Reviewers' Positive Feedback and Addressed Concerns (1/n)

Dear area chair(s),

We sincerely thank the reviewers and area chair(s) for their thoughtful and constructive feedback on our submission. Below, we summarize the positive feedback and the concerns we addressed in our rebuttal.

Reviewers' positive feedback

Novel design:\color{blue}{\textsf{Novel design:}} A well-motivated, novel and practical design that routes weight groups by distribution rather than position

  • R GjLt (R:3 / C:2): The paper is well-motivated. It is reasonable to make codebook assignment of weight groups depend on their distribution rather than their positions, so that similar groups in different rows can share one sketch and the number of sketches no longer grows with the row count.
  • R 8eXv (R:2 / C:3): The paper identifies an interesting limitation of existing sketch-based compression methods and proposes a simple yet novel solution through distribution-based global sketch sharing. I appreciate this kind of intuitive and effective approach that addresses a practical problem.

Unified system:\color{blue}{\textsf{Unified system:}} A simple, data-free method that unifies compression and PEFT in one representation, with a full system implementation

  • R GjLt: The method is intuitive and data-free, and is specified clearly in Algorithms 1-2. The motivation of each step of the compression algorithm is clearly stated.
  • R UHdg (R:2 / C:2): The paper unifies compression and PEFT in a single representation. Since the sketches within the compressed representation are trained directly without a separate adapter, neither training nor inference requires a full-precision backbone. The design is simple and conceptually clear.
  • R UHdg: The paper provides an actual system implementation and cost measurements. The authors implement forward and backward CUDA kernels that operate directly on the packed representation.

Extensive experiments:\color{blue}{\textsf{Extensive experiments:}} Extensive experiments on six DiTs with a clear and fair tuning protocol

  • R GjLt: The experimental setup across 6 DiT-based backbones and two settings is extensive.
  • R 8eXv: The authors evaluate their method across six diffusion transformers of different scales and multiple tasks, with comparisons against existing baselines. The broad experimental coverage provides convincing evidence of the method's effectiveness and practical value.
  • R UHdg: The hyperparameter search protocol is clearly specified. The same number of learning rates (16) is searched for all methods, and the checkpoint selection procedure with separate validation and test splits is described in Algorithm 3.
AuthorsOpenReview ↗

Summary for Area Chair(s): Reviewers' Positive Feedback and Addressed Concerns (2/n)

Summary for Area Chair(s): Reviewers' Positive Feedback and Addressed Concerns (2/n)

Addressed concerns

  • Bits per weight of GSS and of every baseline - Addressed concern raised by R GjLt
  • Details of the LLM judge - Addressed concern raised by R GjLt
  • Why DiTs, and not LLMs - Addressed concern raised by R GjLt
  • Efficiency goals of GSS, and inference memory with int16 cluster IDs - Addressed concern raised by R UHdg
  • Total training time, why GSS trains more slowly than QLoRA, and offline compression time - Addressed concern raised by R UHdg
  • Tuning of full fine-tuning, and LoRA/QLoRA at ranks that match the trainable parameters of GSS - Addressed concern raised by R UHdg
  • Expressiveness of sketch-only updates, and LoRA on top of GSS - Addressed concern raised by R UHdg
  • Difference from vector quantization (AQLM, GPTVQ) - Addressed concern raised by R UHdg
  • Qualitative visual comparisons - Addressed concern raised by R UHdg, R 8eXv
  • Ablation of distribution-based routing against random and position-based routing - Addressed concern raised by R 8eXv

Novelty statement

While novelty is a multifaceted concept in academic research, we believe it can be roughly viewed from two fronts: technical novelty, which focuses on the development of new methodologies, and empirical novelty, which involves uncovering previously unknown properties and behaviors.

  • Technical novelty: As its name indicates, GSS makes sketching globally sensitive: similarly distributed weight groups share the same sketch wherever they are in the matrix. This complements prior work: SketchTune assigns sketches by position, and vector quantization methods such as AQLM encode blocks of weights by their values, whereas GSS shares sketches by distribution. As a result, GSS needs only a small pool of sketches, small enough to fit in on-chip GPU memory, yet still gives the best approximation quality among 4-bit baselines. Fine-tuning then updates only the shared sketches, so one data-free compressed representation serves both compression and PEFT.
  • Empirical novelty: On six large DiTs, GSS has the lowest mean weight quantization error of all 4-bit baselines at a comparable bit width, and our ablation on FLUX.1-dev attributes this gain to distribution-based routing. For fine-tuning, GSS has the lowest training memory of all methods and beats LoRA and QLoRA in LLM score on personalization on all six backbones, even with LoRA and QLoRA given at least as many trainable parameters as GSS.

We hope the above overview will provide our AC and reviewers with a concise way to navigate through the information on this page.

Further, we sincerely hope that our appreciation for a simple yet effective design, routing weights by distribution rather than position, and a single compressed representation for both compression and fine-tuning of large DiTs, can be shared with you and the broader research community working in this important and rapidly evolving field.

Sincerely, Paper51039 authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.