- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OiT to combine the strengths of x-pred and v-pred, aiming to improve few-step sampling while retaining the advantages of x-pred. It uses nested channel masking to learn ordered representations and a small feature prefix for velocity correction. The paper also introduces bREPA and a Hann Decoder to improve feature alignment and image reconstruction.
Critical strengths
-
The motivation is interesting. The paper highlights the trade-off between x-pred and v-pred in representation learning and sampling quality. Using velocity correction to combine their strengths is a reasonable direction.
-
The overall results are strong. OiT-XL achieves 1.55 FID on ImageNet-256, with promising results in 10-step sampling. The paper also includes useful ablations on nested masking, prefix width, bREPA, and the Hann Decoder.
Critical weaknesses
-
The gains from the core design are modest. In Table 3, adding nested masking and velocity correction only improves FID from 5.84 to 5.68. Since JiT already trains x-pred with a velocity loss, the need for a separate Correction Decoder is not well justified. Without a compute-matched comparison against a larger Clean Decoder, it is unclear whether the improvement comes from the proposed correction design or simply the extra model capacity.
-
The benefit of Ordered Representation is unclear. The authors observe larger velocity errors along the leading PCA directions, but the connection to the leading feature channels learned through nested masking is not well established. Tables 4 and 6 provide some useful ablations, but do not fully separate the effects of nested masking and prefix conditioning. It remains unclear whether ordered features offer any advantage over a simple low-dimensional representation.
Questions for authors
-
Why is a separate Correction Decoder preferable to simply increasing the capacity of the Clean Decoder, given that JiT already uses a velocity loss? Could the authors compare the two designs with matched parameters and FLOPs, including results at different sampling steps?
-
Is Ordered Representation necessary for velocity correction? What happens if the correction branch uses the first 8 channels or a learned low-dimensional projection without nested masking? Conversely, how does nested masking perform with full-feature correction? These comparisons would help clarify whether the learned ordering itself matters.
4 comments
Dear Reviewer nKHi,
Thank you very much for the time and care you have given our paper and for your thoughtful feedback. We have posted a detailed response to your comments and hope it addresses the concerns you raised. If anything remains unclear or insufficiently addressed, we would be grateful for the chance to clarify it before the discussion period closes. If you feel that the main concerns have been resolved, we would also greatly appreciate your consideration of a higher rating.
Best regards,
Authors of Submission 30812
response to author
Thank you for the detailed response. The rebuttal has addressed most of my concerns, so I have decided to raise my rating.
Thank you very much for carefully considering our response and for your positive reassessment! Your thoughtful feedback has helped us strengthen the experiments and clarify the method. We sincerely appreciate your time and constructive engagement, and we will incorporate the additional results and clarifications into the revised manuscript.
Response to Reviewer nKHi
We thank the reviewer for the thoughtful assessment and for recognizing OiT's motivation and strong results. These concrete suggestions helped us design more direct comparisons of decoder capacity and feature ordering. We address both concerns below.
W1 / Q1: Velocity supervision and decoder capacity
We agree that JiT already uses velocity-space supervision. The distinction is what the network predicts: JiT predicts the clean image, with a v-loss equivalent to a timestep-weighted x-loss (Eq. (7)). Sampling still relies on a fixed x-to-v conversion that amplifies reconstruction errors. OiT retains clean-image prediction while learning a separately supervised velocity residual to refine the converted field. As shown in Table 1, x-pred remains weaker than direct v-pred in few-step sampling.
Following the suggestion, we removed OiT-B's two-layer Correction Decoder and expanded the Clean Decoder from two to four layers to closely match the parameter and FLOP budgets. We also include a control with two two-layer Clean Decoders whose outputs are averaged:
Relative to the four-layer Clean Decoder, OiT improves probing accuracy by 1.07 percentage points and reduces FID by 3.23/0.45 at 10/50 steps. Its advantage over both capacity controls supports the contribution of explicit velocity correction.
Regarding the modest gain in Table 3, the baseline is already much stronger after the bREPA stage, leaving less room for further improvement. We therefore also evaluated correction before adding bREPA, keeping all other settings unchanged.
Before bREPA, correction reduces FID by 0.92 and improves IS by 18.4.
W2 / Q2: Feature ordering and compact correction inputs
(1) Why nested masking encourages ordering
The ordering mechanism has a simple explanation in a linear orthogonal model. Let x be centered data and ui orthonormal directions. Reconstructing from a prefix of length K gives x^K=∑i=1K(ui⊤x)ui. When K is sampled independently of x,
=\mathbb E\|x-\hat{x}_K\|_2^2 =\mathbb E\|x\|_2^2 -\sum_i\Pr(K\ge i)\,\mathbb E[(u_i^\top x)^2].$$ The first term is fixed total variance; the sum rewards captured variance weighted by how often each direction is retained. Since earlier directions are retained at least as often, placing the leading principal components first minimizes this objective, consistent with the linear theory of Nested Dropout (Rippel et al., 2014). Related ordering is also observed empirically in nonlinear models. FlexTok (Bachmann et al., 2025, Sec. 5.1) shows that short prefixes capture high-level semantic and geometric structure, while additional tokens refine details. Semanticist (Wen et al., 2025, Sec. 5.3, Fig. 4) reports a PCA-like hierarchy in tokens' contributions to reducing diffusion loss. These observations support the motivation for learning informative leading channel groups in OiT. Here, "PCA-like" refers to importance ordering. **(2) Controlled ablations** We appreciate the suggestion to separate the effects of nested masking and correction input width. We added the two requested controls to Table 4(a). All variants use the same training and evaluation settings. | Nested masking | Correction input | FID↓ | IS↑ | | --- | --- | ---: | ---: | | No | None | 5.84 | 206.7 | | No | Full features | 5.81 | 208.8 | | No | First 8 channels | 5.84 | 207.0 | | Yes | None | 5.77 | 209.9 | | Yes | Full features | 5.75 | 210.6 | | Yes | First 8 channels | **5.68** | **211.4** | With the correction input fixed to eight channels, nested masking improves FID from 5.84 to 5.68. With nested masking enabled, the eight-channel prefix also outperforms full-feature correction (5.68 vs. 5.75 FID). Restricting the input width without nested masking gives no FID improvement over the baseline without correction. These results support learning informative prefixes for correction. We especially appreciate the reviewer's precise suggestions, which led to direct tests of both design choices. Thank you for the time and care devoted to examining the method and helping us strengthen its justification.