- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes R²EPA (Reject-REPA), a selective sample rejection mechanism for REPA (Representation Alignment for Generation) training. The authors first connect the REPA objective to a Barber–Agakov variational lower bound on conditional mutual information, and analyze the alignment gradient across cosine-similarity regimes (weak alignment / Case A, strong positive alignment / Case B, and antipodal alignment). Based on this analysis, they propose per-sample loss deviation as a practical indicator of atypical alignment updates. Methodologically, after a warm-up stage, R²EPA applies a dual-detector mechanism (Z-score + MAD with OR fusion) together with cross-batch EMA statistics to reject samples whose REPA alignment losses are unusually high or low; only accepted samples contribute to the alignment term, while the denoising term still uses the full batch. The method requires no additional model parameters. Experiments are conducted on ImageNet 256×256 and COCO, covering SiT-B/L/XL, iREPA, and JiT-B settings, and report FID, sFID, IS, Precision, and Recall, together with ablations, gradient measurements, timestep distributions of rejected samples, and qualitative results.
Critical strengths
-
Simple, practical, and parameter-free. R²EPA is a drop-in replacement for REPA that requires no architectural modification and only filters alignment-loss samples via statistical detectors. It is easy to implement and reproduce, and the experiments show consistent gains across model scales (SiT-B/L/XL) and in the iREPA setting, giving it practical value.
-
Rich experimental and visual evidence. The paper provides extensive quantitative comparisons (ImageNet, COCO, different samplers, different model scales), ablations (warm-up, threshold, Case A/B), gradient measurements, timestep distributions of rejected samples, and qualitative results. In particular, the Case A/B decomposition in Table 4 is analytically useful.
Critical weaknesses
-
Insufficient generalization evidence (most critical). Validation is limited to ImageNet/COCO with DINOv2-B; other datasets and encoders are untested. Moreover, rejected samples concentrate at high noise, suggesting that “outliers” may simply reflect a natural property of high-noise samples rather than genuinely atypical alignment updates. This directly weakens the claim that selective alignment is broadly beneficial.
-
Theory–experiment mismatch and limited explanatory power (most critical). The theory predicts that low-loss samples should also be rejected, but experiments show gains come almost entirely from the high-loss side. Meanwhile, the claimed theoretical calibration of c via Vysochanskij–Petunin does not actually determine c, and the ablation shows c has little effect. Together, this makes the theoretical contribution read more like post-hoc explanation than a predictive, design-guiding analysis.
Questions for authors
-
Rejected samples concentrate heavily at high noise (mean t = 0.825). Have the authors tried treating t as an explicit variable (e.g., stratifying by noise level before rejection) to disentangle “naturally low alignment at high noise” from “genuinely atypical alignment updates”? If the method remains effective after controlling for t, the selective-alignment claim would be substantially strengthened.
-
Does the method remain effective with other encoders (e.g., CLIP, SigLIP, MAE) or on other datasets (e.g., CC3M, LAION subsets, other resolutions)? Geometric differences in encoder feature spaces may significantly alter the loss distribution and outlier structure.
-
Given that Table 4 shows gains come almost entirely from Case A (high-loss side) while Case B (low-loss side) contributes little, have the authors considered one-sided rejection only? Should the theoretical section be revised accordingly to reflect that the low-loss side is not the main source of benefit in practice?
4 comments
Response to Reviewer qcXQ [2/2]
W2(a) & Q3 Given that gains mainly come from rejecting Case A, why also reject Case B, and how should the theoretical discussion reflect this evidence?
Response: Table 4 of the original submission (Table 5 in the revised manuscript) evaluates Case A rejection, which achieves nearly identical ImageNet FID to Case A + Case B rejection. On COCO, Case A + Case B rejection achieves a lower FID (Table 4(c) in the revised manuscript). We therefore retain the Case A + Case B rejection rule: it achieves comparable ImageNet performance and better COCO performance while using a shared threshold to handle both loss tails.
On ImageNet, rejecting Case A alone achieves nearly the same FID as rejecting Case A + Case B. The detector measurements also show that high-loss samples account for 99.38% of pooled flags. The gradient analysis motivates testing both cases, and the ablations compare the results of rejecting each case separately and together. We have clarified this distinction in Section 3.3 and added the COCO comparison and the reason for retaining Case A + Case B rejection to Section 5.3.
W2(b) How does the VP bound guide the choice of , and what does weak threshold sensitivity imply about the theory’s predictive value?
Response: For a unimodal distribution with finite variance, VP gives (approximately 4.94%), using the true mean and standard deviation. This supports choosing to target losses far from the mean without assuming a Gaussian distribution. The bound guides this choice; it does not determine the best threshold or guarantee the rejection rate of the combined Z-score/MAD detector.
The setting lies outside the VP bound's applicable range, , and tests a stricter rejection threshold. All tested thresholds improve over REPA, with relatively small differences. These results show that the method works across the tested thresholds and does not require precise threshold tuning in these experiments.
We have clarified why we choose , when the VP bound applies, and how changing the threshold affects the results in Sections 4.1 and 5.3 and Appendix A.2. We hope the added comparisons and explanations help resolve the concerns raised in the reviews. We welcome further comments and would be glad to discuss any points that remain unclear.
I prefer to maintain the score. My main concern remains the generality of the selective comparison claim and its theoretical explanatory power.
Thank you for your follow-up. We understand your remaining concerns about generality and theoretical insight.
We claim that selective alignment can improve generation in the evaluated settings. The results on ImageNet and COCO, together with the additional MAE experiment, provide evidence across the tested tasks and encoders. Regarding the theory, the MI analysis explains REPA’s objective, while the gradient analysis describes how alignment gradients vary with cosine similarity. These analyses motivate examining which samples contribute to alignment. Our ablation experiments show that filtering samples based on alignment-loss deviations improves generation quality in the tested settings. We have revised the manuscript to explain more clearly how the theoretical analysis motivates the method and how the experiments support its effectiveness.
Our method offers a simple design for REPA by selecting which samples contribute to the alignment objective. We hope you will recognize the value of this contribution, and we thank you for your careful consideration and constructive feedback.
Response to Reviewer qcXQ [1/2]
We thank the reviewer for the detailed evaluation and constructive suggestions on our analysis and experimental scope. We have revised the manuscript and added experimental comparisons to address these concerns. Below, we respond point by point, grouping overlapping weaknesses and questions.
W1 & Q1 Can explicit timestep control distinguish naturally low alignment at high noise from atypical alignment updates?
Response: To examine the role of timestep explicitly, we evaluated a timestep-gated rejection scheme: for t≥0.8, REPA alignment is disabled and only the denoising term is retained; below this threshold, alignment remains active.
The timestep gate improves both metrics over REPA, showing that disabling alignment at high noise can improve generation in this setting. R²EPA achieves better FID and IS than this fixed-cutoff baseline. Its rejected samples also concentrate at high timesteps (Figure 4), but it selects them using their alignment losses instead of rejecting every sample above a fixed timestep cutoff. We have added this comparison to Section 5.3 and Table 4(b), and moved the timestep-distribution analysis and figure into Section 5.3 before the gating comparison.
W1 & Q2 Does R²EPA remain effective with other encoders, datasets, and resolutions despite differences in feature geometry?
Response: We provide an additional comparison using MAE as the alignment encoder:
R²EPA improves both IS and FID-10K with MAE, showing that it also works with an encoder other than DINOv2-B. The detector calculates its thresholds from the observed alignment losses, so the thresholds can adjust when the typical loss values and their spread change. We have included the MAE comparison in Section 5.3 and Table 4(a) of the revised manuscript.