acceptodds
Under review as a conference paper at ICLR 2027

Reject to Align: Selective Sample Rejection for REPA

Abstract

Representation alignment (REPA) has become a standard technique for accelerating Diffusion Transformer (DiT) training by regularising hidden states toward features from a pretrained encoder. However, REPA computes its alignment gradient as a plain arithmetic mean over the batch, assigning equal weight to every sample regardless of alignment quality. We use a variational analysis to relate the population REPA objective to a Barber–Agakov lower bound on the conditional mutual information between DiT and encoder features. This analysis interprets cosine similarity as a positive-pair compatibility score and motivates examining per-sample alignment gradients. The gradient analysis characterises how alignment updates vary with cosine similarity. Complementary empirical measurements link rejected samples to weaker alignment and a higher alignment-to-denoising gradient-norm ratio. These observations motivate using large per-sample loss deviations as a practical indicator of atypical alignment updates. Based on this analysis, we propose Reject-REPA (REPA), a selective sample rejection mechanism that filters alignment-loss outliers via a -based threshold grounded in the Vysochanskij–Petunin inequality, a dual detector combining Z-score + MAD for outlier identification, and cross-batch EMA statistics for adaptive thresholding. REPA introduces no additional model parameters and serves as a drop-in replacement for standard REPA training. Experiments on different generation tasks suggest that selective alignment can improve generation quality in the evaluated settings.

Primary area:
generative models
Keywords:
Representation alignment (REPA), Diffusion Transformer (DiT)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qcXQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes R²EPA (Reject-REPA), a selective sample rejection mechanism for REPA (Representation Alignment for Generation) training. The authors first connect the REPA objective to a Barber–Agakov variational lower bound on conditional mutual information, and analyze the alignment gradient across cosine-similarity regimes (weak alignment / Case A, strong positive alignment / Case B, and antipodal alignment). Based on this analysis, they propose per-sample loss deviation as a practical indicator of atypical alignment updates. Methodologically, after a warm-up stage, R²EPA applies a dual-detector mechanism (Z-score + MAD with OR fusion) together with cross-batch EMA statistics to reject samples whose REPA alignment losses are unusually high or low; only accepted samples contribute to the alignment term, while the denoising term still uses the full batch. The method requires no additional model parameters. Experiments are conducted on ImageNet 256×256 and COCO, covering SiT-B/L/XL, iREPA, and JiT-B settings, and report FID, sFID, IS, Precision, and Recall, together with ablations, gradient measurements, timestep distributions of rejected samples, and qualitative results.

Critical strengths
  1. Simple, practical, and parameter-free. R²EPA is a drop-in replacement for REPA that requires no architectural modification and only filters alignment-loss samples via statistical detectors. It is easy to implement and reproduce, and the experiments show consistent gains across model scales (SiT-B/L/XL) and in the iREPA setting, giving it practical value.

  2. Rich experimental and visual evidence. The paper provides extensive quantitative comparisons (ImageNet, COCO, different samplers, different model scales), ablations (warm-up, threshold, Case A/B), gradient measurements, timestep distributions of rejected samples, and qualitative results. In particular, the Case A/B decomposition in Table 4 is analytically useful.

Critical weaknesses
  1. Insufficient generalization evidence (most critical). Validation is limited to ImageNet/COCO with DINOv2-B; other datasets and encoders are untested. Moreover, rejected samples concentrate at high noise, suggesting that “outliers” may simply reflect a natural property of high-noise samples rather than genuinely atypical alignment updates. This directly weakens the claim that selective alignment is broadly beneficial.

  2. Theory–experiment mismatch and limited explanatory power (most critical). The theory predicts that low-loss samples should also be rejected, but experiments show gains come almost entirely from the high-loss side. Meanwhile, the claimed theoretical calibration of c via Vysochanskij–Petunin does not actually determine c, and the ablation shows c has little effect. Together, this makes the theoretical contribution read more like post-hoc explanation than a predictive, design-guiding analysis.

Questions for authors
  1. Rejected samples concentrate heavily at high noise (mean t = 0.825). Have the authors tried treating t as an explicit variable (e.g., stratifying by noise level before rejection) to disentangle “naturally low alignment at high noise” from “genuinely atypical alignment updates”? If the method remains effective after controlling for t, the selective-alignment claim would be substantially strengthened.

  2. Does the method remain effective with other encoders (e.g., CLIP, SigLIP, MAE) or on other datasets (e.g., CC3M, LAION subsets, other resolutions)? Geometric differences in encoder feature spaces may significantly alter the loss distribution and outlier structure.

  3. Given that Table 4 shows gains come almost entirely from Case A (high-loss side) while Case B (low-loss side) contributes little, have the authors considered one-sided rejection only? Should the theoretical section be revised accordingly to reflect that the low-loss side is not the main source of benefit in practice?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qcXQ [1/2]

We thank the reviewer for the detailed evaluation and constructive suggestions on our analysis and experimental scope. We have revised the manuscript and added experimental comparisons to address these concerns. Below, we respond point by point, grouping overlapping weaknesses and questions.

W1 & Q1 Can explicit timestep control distinguish naturally low alignment at high noise from atypical alignment updates?

Response: To examine the role of timestep explicitly, we evaluated a timestep-gated rejection scheme: for t≥0.8t\geq0.8, REPA alignment is disabled and only the denoising term is retained; below this threshold, alignment remains active.

MethodFID ↓IS ↑
REPA7.90122.60
Timestep-gated rejection7.70126.74
R²EPA7.57128.88

The timestep gate improves both metrics over REPA, showing that disabling alignment at high noise can improve generation in this setting. R²EPA achieves better FID and IS than this fixed-cutoff baseline. Its rejected samples also concentrate at high timesteps (Figure 4), but it selects them using their alignment losses instead of rejecting every sample above a fixed timestep cutoff. We have added this comparison to Section 5.3 and Table 4(b), and moved the timestep-distribution analysis and figure into Section 5.3 before the gating comparison.

W1 & Q2 Does R²EPA remain effective with other encoders, datasets, and resolutions despite differences in feature geometry?

Response: We provide an additional comparison using MAE as the alignment encoder:

MethodIS ↑FID-10K ↓
Baseline with MAE alignment108.5915.41
R²EPA with MAE alignment110.7915.13

R²EPA improves both IS and FID-10K with MAE, showing that it also works with an encoder other than DINOv2-B. The detector calculates its thresholds from the observed alignment losses, so the thresholds can adjust when the typical loss values and their spread change. We have included the MAE comparison in Section 5.3 and Table 4(a) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer qcXQ [2/2]

W2(a) & Q3 Given that gains mainly come from rejecting Case A, why also reject Case B, and how should the theoretical discussion reflect this evidence?

Response: Table 4 of the original submission (Table 5 in the revised manuscript) evaluates Case A rejection, which achieves nearly identical ImageNet FID to Case A + Case B rejection. On COCO, Case A + Case B rejection achieves a lower FID (Table 4(c) in the revised manuscript). We therefore retain the Case A + Case B rejection rule: it achieves comparable ImageNet performance and better COCO performance while using a shared threshold cc to handle both loss tails.

On ImageNet, rejecting Case A alone achieves nearly the same FID as rejecting Case A + Case B. The detector measurements also show that high-loss samples account for 99.38% of pooled flags. The gradient analysis motivates testing both cases, and the ablations compare the results of rejecting each case separately and together. We have clarified this distinction in Section 3.3 and added the COCO comparison and the reason for retaining Case A + Case B rejection to Section 5.3.

W2(b) How does the VP bound guide the choice of cc, and what does weak threshold sensitivity imply about the theory’s predictive value?

Response: For a unimodal distribution with finite variance, VP gives Pr⁡(∣X−μ∣>3σ)≤4/81\Pr(|X-\mu|>3\sigma)\leq4/81 (approximately 4.94%), using the true mean and standard deviation. This supports choosing c=3c=3 to target losses far from the mean without assuming a Gaussian distribution. The bound guides this choice; it does not determine the best threshold or guarantee the rejection rate of the combined Z-score/MAD detector.

The c=1c=1 setting lies outside the VP bound's applicable range, c≥8/3c\geq\sqrt{8/3}, and tests a stricter rejection threshold. All tested thresholds improve over REPA, with relatively small differences. These results show that the method works across the tested thresholds and does not require precise threshold tuning in these experiments.

We have clarified why we choose c=3c=3, when the VP bound applies, and how changing the threshold affects the results in Sections 4.1 and 5.3 and Appendix A.2. We hope the added comparisons and explanations help resolve the concerns raised in the reviews. We welcome further comments and would be glad to discuss any points that remain unclear.

I prefer to maintain the score. My main concern remains the generality of the selective comparison claim and its theoretical explanatory power.

Thank you for your follow-up. We understand your remaining concerns about generality and theoretical insight.

We claim that selective alignment can improve generation in the evaluated settings. The results on ImageNet and COCO, together with the additional MAE experiment, provide evidence across the tested tasks and encoders. Regarding the theory, the MI analysis explains REPA’s objective, while the gradient analysis describes how alignment gradients vary with cosine similarity. These analyses motivate examining which samples contribute to alignment. Our ablation experiments show that filtering samples based on alignment-loss deviations improves generation quality in the tested settings. We have revised the manuscript to explain more clearly how the theoretical analysis motivates the method and how the experiments support its effectiveness.

Our method offers a simple design for REPA by selecting which samples contribute to the alignment objective. We hope you will recognize the value of this contribution, and we thank you for your careful consideration and constructive feedback.

Reviewer_5iV2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a limitation of standard representation alignment for Diffusion Transformers: it computes alignment gradients as a uniform arithmetic mean over a batch, ignoring heterogeneity in per-sample alignment quality. Through variational analysis and gradient geometry analysis, the authors characterize how alignment updates behave across different cosine similarity regimes, motivating the use of per-sample loss deviation as an indicator of atypical updates. They propose R²EPA, a drop-in selective sample rejection mechanism based on the Vysochanskij–Petunin inequality, a dual Z-score + MAD outlier detector, and cross-batch EMA statistics. The method introduces no additional model parameters.

Critical strengths
  1. This work systematically investigates intra-batch sample heterogeneity in REPA training.
  2. The variational analysis linking REPA to a Barber–Agakov lower bound on conditional mutual information provides a principled interpretation of per-sample alignment loss as a compatibility score.
Critical weaknesses
  1. The idea of selective sample weighting/rejection is not new to diffusion model training overall. Loss reweighting, curriculum sampling, and outlier filtering have been widely explored for the denoising objective. This paper applies an analogous paradigm to the alignment loss, which is an incremental refinement rather than a fundamental conceptual breakthrough.
  2. The paper uses hard binary rejection, but does not justify this design choice against smoother alternatives. It remains unclear whether hard rejection is optimal for this task, or whether soft weighting could achieve better or comparable results with more stable gradients.
  3. HASTE addresses temporal gradient conflict globally, while the proposed method addresses sample-level heterogeneity. The two mechanisms are complementary, but the paper does not evaluate their combination or discuss their relative merits and applicable scenarios.
  4. The paper does not analyze behavior when a large fraction of the batch is flagged as outliers. If the valid set becomes too small, alignment gradient estimation could become noisy or unstable. No safeguard is proposed or verified.
  5. Also, current modern large DIT-based generation models (e.g., Flux2) have been widely used in many senses. Is the proposed method proven in these large generation models?
Questions for authors

Please see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5iV2 [1/2]

We appreciate the reviewer's thoughtful assessment and helpful suggestions. These comments have guided our manuscript revisions and expanded experimental evaluation. Our point-by-point responses follow.

W1 What is the contribution beyond existing loss reweighting, curriculum sampling, and outlier filtering methods?

Response: We show that selecting which samples contribute to REPA alignment can improve generation quality. The controlled experiment in Appendix B shows that a small fraction of corrupted alignment targets can destabilize training. In the main experiments, filtering samples with unusually high or low alignment losses improves generation quality over standard REPA. Our method removes selected terms from the alignment objective while retaining every sample for denoising.

The Z-score/MAD filter requires no additional model parameters or architectural changes. It improves generation across the evaluated model scales and samplers and works with both REPA and iREPA. These results show that a simple change to which samples receive alignment can improve existing training methods.

W2 Why use hard rejection rather than soft weighting, and could soft weighting provide comparable performance with more stable gradients?

Response: We have also considered whether soft weighting could offer an effective alternative to hard rejection. In Table 2(a), we explore an adaptive non-uniform weighting strategy, implemented as soft weighting of the REPA alignment terms. R²EPA achieves lower FID and higher IS than this soft-weighting baseline, supporting direct rejection in the evaluated setting.

We believe that extreme alignment terms can still affect the update when assigned small nonzero weights, which may explain why removing them performs better in this comparison. Rejected samples continue to contribute to denoising. We have added this explanation to Section 5.2 of the revised manuscript.

W3 How does R²EPA relate to HASTE in design and applicability, and how could their strengths be combined?

Response: Like HASTE, our warm-up design keeps alignment active early in training. We apply alignment to all samples during warm-up and then only to accepted samples after rejection begins.

We believe that keeping all samples involved in alignment during warm-up helps the model first learn from the encoder representations. Selective rejection is then introduced to reduce the influence of atypical alignment terms. Table 6 (left) shows that the effect is not strictly monotonic: the 100K setting improves IS over immediate rejection but has a slightly higher FID, while 200K gives the best FID and IS among the tested schedules. These results support allowing an initial period of full alignment before applying sample selection.

Table 2(a) also reports better FID and IS for R²EPA than HASTE in the evaluated setting. HASTE controls when alignment ends globally; R²EPA controls which samples contribute after warm-up. The warm-up comparison is reported in Section 5.3 and Table 6 (left) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 5iV2 [2/2]

W4 How is training stability maintained when few samples are accepted, and what safeguards address excessive rejection?

Response: Our implementation includes an explicit fallback: if no samples are accepted, we use the median per-sample alignment loss of the batch instead of averaging an empty set. Alignment supervision is retained, and all samples continue to contribute to denoising. We have clarified this safeguard in Section 4.1 of the revised manuscript.

We choose the default threshold to keep losses near the center of the distribution and reject those far from it. For a unimodal distribution with finite variance, the VP inequality places at least 77/8177/81 (approximately 95.06%) of the probability within μ±3σ\mu\pm3\sigma, using the true mean and standard deviation. This supports the choice of c=3c=3, but does not guarantee a minimum accepted count in each batch for our combined Z-score/MAD detector.

Table 6 (right) shows improvements over REPA for all tested thresholds, including the stricter c=1c=1 setting. Although the VP bound does not apply to c=1c=1, this experiment shows that the method still improves generation with a stricter rejection threshold. We have explained the threshold choice and these results in Appendix A.2 of the revised manuscript.

W5 Has R²EPA been validated on larger generation models, such as Flux2?

Response: We apologize that limited computational resources currently restrict our evaluation to the model scales reported in the paper. We plan to extend validation to larger generation models, including those highlighted by the reviewer, and explore multimodal training scenarios. We have added these directions to Section 6 (Conclusion and Future Work) of the revised manuscript.

Thanks for your response. I will keep the original positive score. Furthermore, I just want to point out that the FLUX.2 family covers different models with diverse scales and parameters. For example, FLUX.2-klein-4B can be trained on a consumer-level GPU.

Thank you for your clarification and helpful suggestion. We appreciate you pointing out FLUX.2-klein-4B as a more accessible option. In future work, we will explore extending our strategy to multimodal settings and a broader range of models.

Reviewer_tAPFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper modifies REPA by inserting a simple filter into the alignment loss. After a warm-up stage, the filter rejects samples whose per-sample alignment loss deviates by more than cc robust standard deviations from the batch mean, i.e., samples with either very low or very high average similarity. This is explained through the lens of gradient geometry: weakly aligned samples yield full-scale residuals whose gradients may conflict with the aggregate direction, while near-perfectly aligned samples yield vanishing gradients. The authors also show, as a standalone result, that minimizing the population REPA objective maximizes a Barber-Agakov lower bound on the conditional mutual information between DiT and frozen-encoder features. Finally, the method is evaluated across different setups, showing modest improvements over plain REPA.

Critical strengths
  1. The improvements are well-substantiated: the authors report consistent FID/sFID/IS gains across SiT-B/L/XL, ODE/SDE samplers at different step counts, and three seeds, plus auxiliary evidence (FDr$^6$, pixel-space JiT-B-16, COCO T2I, qualitative samples) and a training-trajectory comparison suggesting improved efficiency (FID 2.76 at 350K vs REPA's 2.81 at 400K), although the absolute gains are modest.
  2. The method is simple (essentially being a parameter-free drop-in), and is motivated by a gradient-geometry analysis.
  3. The threshold is calibrated via the Vysochanskii-Petunin inequality, which requires only unimodality of the loss distribution (and c≥8/3c \ge \sqrt{8/3}) and yields an explicit upper bound on the Z-score branch's rejection rate, which is a neat distribution robustness property.
Critical weaknesses
  1. The theoretical machinery is largely decorative with respect to the final method.
    • The MI lower bound (Prop. 1) is not used anywhere downstream. In particular, the per-sample "compatibility score" reading follows directly from the vMF parameterization (Eq. (4)) without the Barber-Agakov step.
    • The conflict claim for weakly aligned samples (Prop. 2) is qualitative: the proof establishes only the feature-space limit, and the transfer to parameter space via the sample-dependent Jacobian is argued, but is not shown.
    • The actual decision statistic (the loss deviation) is validated empirically (Table 9), but is not derived. Consequently, the core of the method is essentially a well-motivated, well-validated heuristic, and only the rejection-rate bound is fully rigorous.
  2. (Minor, reproducibility and edge cases) Iwas not able to find the EMA momentum α\alpha (Eqs. (9) and (10), Algorithm 1). The c=1c = 1 ablation (Table 5) also lies outside the Vysochanskii-Petunin regime (c≥8/3c \ge \sqrt{8/3}), so its rejection rate carries no guarantee.
Questions for authors
  1. Is the connection to MI maximization essential to your analysis? What would the paper lose, if anything, if Sec. 3.2 / Prop. 1 were removed?
  2. All experiments use DINOv2-B. How sensitive is R$^2$EPA's benefit to the encoder choice? With a less informative encoder, one would expect a larger fraction of weakly aligned (Case A) samples. Does the gap between R$^2$EPA and REPA widen or shrink in that regime?
1 comment
AuthorsOpenReview ↗

Response to Reviewer tAPF

We thank the reviewer for the careful reading and constructive feedback. In response to the questions and suggestions, we have revised the manuscript and included additional experimental results. We address each point below.

W1 & Q1 Is the MI connection essential, what would be lost without Section 3.2 / Proposition 1, and how does the gradient analysis justify sample rejection?

Response: Proposition 1 links REPA's expected alignment loss to a lower bound on conditional MI. With the data distribution, encoder, and κ\kappa fixed, minimizing expected REPA loss is equivalent to maximizing a specific variational lower bound on conditional MI between the DiT and encoder features.

The vMF model expresses cosine similarity as a positive-pair compatibility score. Proposition 1 shows how the expected score enters the conditional-MI bound. Removing Section 3.2 would remove this explanation of how REPA's cosine loss relates to conditional MI, although the gradient calculations in Section 3.3 would remain valid.

The MI analysis explains the training objective, while the gradient analysis examines how each sample affects the update. Section 3.3 shows that weak alignment produces a large feature-space residual, while near-complete positive alignment produces a vanishing residual. How these residuals affect the model parameters also depends on each sample's network Jacobian. These observations motivate selecting samples based on alignment-loss deviations, and our ablation experiments test whether this improves generation quality. We have clarified the roles of the MI and gradient analyses at the end of Section 3.2.

W2 What EMA momentum α\alpha is used, and how should the c=1c=1 ablation outside the VP regime be interpreted?

Response: We have specified the default EMA momentum α=0.99\alpha=0.99 after Equation (10) in Section 4.1. This setting retains historical loss statistics and reduces batch-to-batch fluctuations.

For a unimodal distribution with finite variance, the VP bound gives Pr⁡(∣X−μ∣>3σ)≤4/81\Pr(|X-\mu|>3\sigma)\leq4/81 (approximately 4.94%), where μ\mu and σ\sigma are the true mean and standard deviation. This supports choosing c=3c=3 to target losses far from the mean. We use the bound to guide the threshold choice; it does not guarantee the rejection rate of the combined Z-score/MAD detector. Since 1<8/31<\sqrt{8/3}, c=1c=1 is outside the bound's applicable range and tests a stricter rejection threshold.

All tested thresholds improve over REPA, with relatively small performance differences. These results show that the method works across the tested thresholds and does not require precise threshold tuning in these experiments. We have clarified when the bound applies and why we test c=1c=1 in Sections 4.1 and 5.3 and Appendix A.2.

Q2 Would a less informative encoder produce more weakly aligned samples, and would the gap between R²EPA and REPA widen or shrink?

Response: We have added an experiment using MAE as the alignment encoder:

MethodIS ↑FID-10K ↓
REPA with MAE alignment108.5915.41
R²EPA with MAE alignment110.7915.13

R²EPA improves both IS and FID-10K with MAE, showing that it also works with an encoder other than DINOv2-B. Changing the encoder can change the alignment-loss distribution. Our Z-score/MAD detector updates its thresholds using the observed losses and selects samples according to how far their losses deviate from the center of that distribution. A less informative encoder therefore does not necessarily lead to larger or smaller gains. We have included the MAE results in Section 5.3 and Table 4(a) of the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of the Revision

We appreciate the reviewers' careful assessment and constructive suggestions. Their feedback helped us explain the analysis more clearly and add comparisons that further evaluate selective alignment. We summarize the main revisions below.

  • Sections 3.2 and 3.3: Clarified the different roles of the MI analysis and the gradient analysis, and explained how the rejection ablations evaluate the contributions of Case A and Case B.
  • Section 5.2: Added an explanation of why removing extreme alignment terms may work better than assigning them small nonzero weights, while keeping all samples in the denoising objective.
  • Section 5.3 and Table 4(a): Added an evaluation with MAE as the alignment encoder, where R²EPA improves both IS and FID-10K over REPA.
  • Section 5.3, Figure 4, and Table 4(b): Brought the timestep distribution of rejected samples into the main text and added a comparison with a fixed timestep gate.
  • Section 5.3 and Table 4(c): Added a COCO comparison of Case A and Case A + Case B rejection and explained why we retain both rejection branches.
  • Section 5.3 and Appendix A.2: Clarified how the VP bound guides the threshold choice and how to interpret the stricter c=1c=1 ablation outside its applicable range.

The main revisions are marked in blue in the manuscript. We look forward to further discussion with the reviewers.

AuthorsOpenReview ↗

Key Contributions and Rebuttal Updates

Dear Area Chair,

Thank you for considering our paper. We would like to highlight the following points:

  • Practical contribution: R²EPA improves REPA by selecting which samples contribute to alignment while retaining all samples for denoising. It requires no additional model parameters or architectural changes.
  • Consistent empirical benefits: Improvements are observed across the evaluated model scales, samplers, REPA/iREPA configurations, and three seeds. Training trajectories also show better FID with fewer training steps.
  • Additional evidence and revisions: We added MAE, timestep-gating, and COCO rejection comparisons. We clarified the roles of the MI and gradient analyses, the threshold choice, and implementation safeguards.
  • Reviewer discussion: Reviewers recognized the method’s simplicity and experimental breadth. Reviewer 5iV2 explicitly maintained the positive score after our response. We also addressed Reviewer qcXQ’s remaining concerns by clarifying the scope of our claims and the supporting evidence.

We respectfully ask the Area Chair to consider the practical value of this simple design for REPA and the strengthened experimental evidence in the revision. We hope you will recognize the value of our contribution. Thank you for your time and careful assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.