Compressed Adaptation of Diffusion Transformers via Globally-Sensitive Sketch

Adapting diffusion transformers under tight memory budgets requires both compression and parameter-efficient fine-tuning. Existing sketch-based compression methods use locally sensitive sketches, where each lookup table is tied to a predetermined spatial region and cannot be shared by statistically similar weight groups at different locations. We propose Globally-Sensitive Sketch (GSS), a unified framework for compression and parameter-efficient fine-tuning. GSS clusters weight groups within each weight matrix by normalized distributional shape, allowing similar groups to share learnable sketches across arbitrary matrix positions. Each weight stores a 4-bit index into its group's assigned sketch, and only the shared sketch entries are updated during fine-tuning. GSS is extensively evaluated on six diffusion transformers spanning 2B to 20B parameters. We consider two settings: adaptation for subject-driven personalization and paired image editing, and standalone text-to-image generation. GSS has the lowest mean weight quantization error among all evaluated 4-bit baselines and preserves text-to-image generation quality across all six backbones. In addition, GSS achieves competitive quality on both subject-driven personalization and paired image editing compared to full fine-tuning while using over 100$\times$ fewer trainable parameters. We provide our code at https://anonymous.4open.science/r/gss-anonymization/.