- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Globally-Sensitive Sketch (GSS), a framework for compressing and efficiently fine-tuning diffusion transformers under limited memory budgets. The key observation is that weight groups at different positions may have similar distributions, but existing sketch-based methods typically assign codebooks based on fixed spatial locations, preventing such groups from sharing parameters. To address this limitation, GSS clusters weight groups according to their normalized distributional shapes and allows similar groups to share learnable sketches across different positions. During fine-tuning, only the shared sketch entries are updated, while the quantization codes and routing assignments remain fixed, significantly reducing the number of trainable parameters.
The authors evaluate GSS on six diffusion transformers ranging from 2B to 20B parameters, covering subject-driven personalization, paired image editing, and standalone text-to-image generation. Experimental results show that GSS achieves competitive adaptation quality compared with full fine-tuning and existing PEFT methods while using substantially fewer trainable parameters. It also demonstrates strong 4-bit quantization performance, achieving lower weight reconstruction errors and preserving generation quality across different backbones. Additional ablation studies investigate the effects of sketch pool size and density-weighted fitting.
Critical strengths
-
Novel and practical solution. The paper identifies an interesting limitation of existing sketch-based compression methods and proposes a simple yet novel solution through distribution-based global sketch sharing. I appreciate this kind of intuitive and effective approach that addresses a practical problem.
-
Comprehensive experiments. The authors evaluate their method across six diffusion transformers of different scales and multiple tasks, with comparisons against existing baselines. The broad experimental coverage provides convincing evidence of the method's effectiveness and practical value.
Critical weaknesses
-
Lack of qualitative visualization results. Personally, I believe that for a paper involving image generation, showing actual generated images is quite important. However, the authors do not provide any qualitative visual comparisons. I would really like to see how GSS performs visually compared to other methods, as numerical metrics alone may not fully reflect the perceptual quality of generated images.
-
Insufficient ablation on the core design. The main motivation of this paper is distribution-based global sketch sharing, but the ablation studies mainly focus on sketch pool size and density-weighted fitting. There is no direct comparison with alternative routing strategies, such as random or position-based assignment, under the same sketch budget. This makes it difficult to isolate the actual contribution of global routing, which I consider an important missing piece in validating the core idea.
Questions for authors
-
Qualitative visualization results. Could the authors provide some visual comparisons of generated images between GSS and the baseline methods? I believe such comparisons are important for image generation tasks, as they would help us better understand the actual perceptual quality and potential limitations of the proposed method.
-
Ablation on global sketch sharing. Could the authors provide a more direct ablation study comparing the proposed distribution-based routing with alternative strategies, such as random or position-based assignment, under the same sketch budget? I think this would help clarify how much of the improvement comes specifically from the global routing mechanism rather than other design choices.
3 comments
Thank you for the detailed response and for conducting the additional experiments. I appreciate the authors' efforts in addressing my concerns.
The newly added qualitative comparisons and the controlled ablation study on different routing strategies have largely addressed my main concerns, particularly regarding the effectiveness of distribution-based global sketch sharing.
Given these additional results and clarifications, I am satisfied with the authors' response and will raise my score accordingly. I appreciate your efforts and wish you the best of luck!
Dear Reviewer 8eXv,
Thank you so much for your time, thoughtful feedback, and for raising your score! We are glad the additional experiments and clarifications addressed your concerns. Your comments helped us improve the paper significantly, and we truly appreciate your constructive engagement throughout the review process.
Sincerely, Paper51039 authors
Rebuttal for Reviewer 8eXv
We thank the reviewer for the thoughtful feedback and for highlighting the novel and practical design and the comprehensive experiments of our work. We address the comments and questions below, and we have revised the paper accordingly, with all changes highlighted in red.
[W1 - Qualitative visual comparisons] --> We add four image figures, covering every backbone and task.A. Qualitative results. We add four figures to the appendix of the revised paper, covering every backbone and task. The first compares GSS, Full FT, LoRA, QLoRA and SketchTune on subject-driven personalization on all six backbones, and the second compares the same five methods on image editing with FLUX.1-Kontext. The third and fourth compare BF16, GSS, NF4, OrbitQuant and SketchTune on text-to-image generation after 4-bit quantization on all six backbones, one prompt per figure. Images compared side by side use the same prompt.
[W2 - Ablation of distribution-based routing] --> SketchTune already assigns by position, and we add a direct ablation against random and position-based assignment.A. Our main baseline already assigns by position. SketchTune shares its tables by row position, and the paper compares GSS with it throughout.
B. A direct ablation at the same budget. We now also change only the assignment rule inside GSS. Every arm uses 1024 sketches per matrix, groups of 64 and 4-bit codes, and refits its pool the same way, so bits per weight and trainable parameters are identical. Random assignment gives each group a uniformly random sketch; position assignment gives each sketch one contiguous band of groups. We report the weight reconstruction error over all 456 linear layers of FLUX.1-dev:
As the table shows, distribution-based routing gives the lowest error on every metric. Random and position assignment perform about the same as each other, and both raise the mean weight error by 23-24%, its standard deviation by 22-23%, and the mean error on the largest 1% of errors by 40-42%. The gain therefore comes from routing groups by distribution, not from the shared pool alone.
[Q1 - Qualitative visualization results] --> We add four image figures, covering every backbone and task.See W1.
[Q2 - Ablation of global sketch sharing] --> SketchTune already assigns by position, and we add a direct ablation against random and position-based assignment.See W2.