- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper directly optimizes the layer-output reconstruction error after quantization and learns an orthogonal rotation matrix. It uses per-timestep incoherence coefficients as weights, initializes the rotation using the product of the eigenvector matrix of the weighted covariance matrix and a fixed normalized randomized Hadamard transform, preserves orthogonality through Cayley parameterization. Experimental results further demonstrate the effectiveness of SyRA-DiT.
Critical strengths
-
This paper uses the quantized layer output error as the optimization objective to learn an orthogonal rotation matrix. Orthogonality is ensured through the Cayley transformation. The problem of non-differentiability caused by quantization rounding during backpropagation is addressed using STE.
-
This method uses timestep incoherence weights to directly perform a weighted average on the covariance of the activation matrix, obtaining the initial rotation matrix through eigenvalue decomposition combined with the Hadamard matrix. It also uses the same time-step weights to construct GPTQ’s second-order statistics, which guide the final weight rounding and error compensation. Experiments with DiT-XL/2, PixArt-, and Open-Sora demonstrate strong W4A4 performance across image and video generation tasks.
Critical weaknesses
-
Missing comparisons to several recent, closely related methods, such as LRQ-DiT, DiRotQ, and TR-DQ, which are mentioned but did not compared. These are also rotation- or time-step-aware quantization methods aimed at achieving high-quality low-bit quantization. Given the similarities in their design philosophies, including comparisons with recent baselines would strengthen the paper’s positioning.
-
Obtaining the initial rotation matrix requires solving the corresponding covariance matrices for all activation matrices in the calibration set. Additionally, optimizing the quantized layer output errors requires layer-by-layer learning. Since some models have a large number of layers, and certain layers may be very wide, this may introduce significant computational overhead for calibration.
Questions for authors
-
Could the authors report at least one more comparison with a recent baseline method that takes rotation or time-step awareness into account?
-
What is the calibration overhead associated with computing the covariance matrix of activations for each calibration set and learning with STE?
-
What impact does using the normalized time-step weights mentioned in Appendix B have on the actual GPTQ calibration compared to using the original time-step weights ?
## Common response to all reviewers and AC(s)
Dear Area Chair and Reviewers,
We sincerely thank the three reviewers for their careful reading, thoughtful comments, and constructive suggestions, and the Area Chair for overseeing the review process. We are encouraged by the reviewers' recognition of SyRA-DiT's state-of-the-art W4A4 generation quality and its well-motivated design. We also appreciate their insightful feedback on methodological novelty, theoretical consistency, experimental comparisons, and practical efficiency, which has helped us further strengthen the manuscript. In response, we have made the following major revisions:
Methodological clarification and component-wise analysis: We have clarified the novelty of SyRA-DiT relative to existing learned-rotation PTQ methods and introduced controlled ablations to isolate the contributions of rotation learning and temporal calibration.
Expanded experimental comparisons: We have added comparisons with recent competitive W4A4 quantization methods across image, text-to-image, and video generation, including rotation-based PTQ and training-based approaches.
Calibration and inference efficiency: We have quantified calibration budgets, dense rotation storage, and online computational overhead, and investigated structured rotations to improve practical deployment efficiency.
Theoretical consistency and implementation clarity: We have clarified the relationship between timestep-weighted reconstruction and GPTQ calibration, improved the theoretical consistency claim, and refined the algorithm and implementation details.
These revisions provide a more comprehensive evaluation of SyRA-DiT's contributions, performance, and practical limitations. Below, we summarize the main shared concerns and our responses, followed by point-by-point replies to each reviewer.
Common questions and related Revisions
CQ1. Evaluations on rotation cost (by Reviewer p2BP, Reviewer 3NEQ, Reviewer vGiv)
Reply:
We thank the reviewer for this valuable comment. We have revised the efficiency analysis and Table 1 to explicitly account for activation rotation overhead. All configurations use CUDA Graph execution on an A100 GPU to ensure a consistent comparison.
Table 1. Inference cost of DiT-XL/2 on A100 (ImageNet-256, DDIM 50, CFG$=1.5$, batch size 1). Measured W4A4 results use a symmetric per-token INT4 timing prototype, distinct from the asymmetric group-128 quality implementation. † denotes projected latency and estimated memory for the hybrid rotation design, which replaces dense fc2 rotations with structured transforms. Speedups are relative to FP16.
Specifically, SyRA-DiT applies dense FP16 rotations to 112 quantized linear layers. Although weight rotations are absorbed into GPTQ quantization, activation rotations still introduce additional computational and storage overhead, regardless of whether they are learned or fixed. Accounting for these operations, W4A4 incurs 370.4 ms/image latency and 4086 MB peak memory, compared with 341.3 ms/image and 1672 MB for FP16. The dense rotations introduce approximately 1.32 GiB of storage and 177 ms/image of latency overhead, with fc2 contributing approximately 84% of the rotation cost. We also clarify that the 1.77× speedup (193.3 ms/image) corresponds to W4A4 GEMM-only execution without activation rotations, rather than the complete SyRA-DiT pipeline.
To mitigate this overhead, we evaluate a simple hybrid rotation design (denoted by hybrid R† in the table) that replaces dense rotations in fc2 layers with online Hadamard transforms and learned block-diagonal orthogonal rotations, while retaining dense rotations in the remaining layers. The hybrid design yields an estimated latency of 263.4 ms/image and a peak memory footprint of 2984 MB. This corresponds to a 1.30× speedup over FP16, with a modest increase in FID from 6.28 to 6.57 relative to the dense-rotation design.
This initial experiment suggests that structured rotations can reduce inference overhead while largely preserving generation fidelity. Our primary aim is to use rotations to study how weight and activation quantization interact across diffusion timesteps. We leave a systematic comparison of low-cost alternatives, including sparse rotations and block-diagonal special orthogonal rotations, to future work. We thank the reviewers for raising this point and have revised the contribution statement and limitations section to clarify the trade-offs among quantization accuracy, rotation overhead, and inference efficiency.