- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AnisoFlow. It replaces the hand-designed prediction targets in flow models with a learnable one by using a learnable orthogonal basis where each direction gets its own noise weight while the data term stays fixed. It consistently improves ImageNet and text-to-image generation, and also stabilizes low-precision inference. Interestingly, the learned basis converges to the data PCA basis and the weights grow nearly linearly with directional variance, which matches the theoretical analysis.
Critical strengths
-
The paper takes something (velocity vs. x vs. ε) treats as a fixed choice, the prediction target, and makes it learnable. Nothing else in the pipeline has to change, it never hurts performance, and the gains in pixel space are solid. So I think it is likely useful for other tasks too.
-
The learned basis lines up with PCA and the weights follow the data spectrum. This regularity is worth reporting regardless of how it is explained, and it gives the community a concrete takeaway beyond FID numbers.
Critical weaknesses
-
The main conclusion, that the preferred weight scales near-linearly with data variance, depends on several strong assumptions, like the noise and data errors being uncorrelated. None of them are checked. They seem picked to fit the observed results, so the theory does not really predict anything new.
-
The learned weights basically converge to a simple analytic curve. But the paper never tests just using that fixed curve directly. So I cannot tell if the gains come from the anisotropic shape itself or from the learning process. This is a confusion for me.
-
The operator only works inside each patch and is shared everywhere, so it ignores image content entirely, and the paper never explains why that is fine. Worse, on the mainstream latent-space setup it barely helps and just falls back to the default target. Please explain.
If you can convince me, I will increase my score,Question 1 is the most important, please convince me. (They seem picked to fit the observed results, so the theory does not really predict anything new. )
Questions for authors
See weaknesses,thank you.
4 comments
Thank you for your reply,
W1: The new error analysis is a useful sanity check, but it still feels mostly post hoc since it’s based on the trained model. It doesn’t yet convince me that the theory has real predictive power.
W2: The key fixed-weight baseline is still missing. Without it, I can’t tell whether the gains come from the target shape itself or from the learning process.
W3: The explanation for the shared operator sounds reasonable, but there’s no supporting experiment.
For latent models, the authors confirm that the method doesn’t improve over the default target.
So, the pixel space results of the paper are indeed good, so this is not a work lacking value. But, I think the author's response is not enough to change the core flaw of the paper. If a fixed weight baseline is submitted later and it is proven that the learning version is significantly better, the paper will be more reasonable.
We thank the reviewer for their follow-up and for acknowledging the value of our pixel-space results. We would like to clarify the theoretical contribution and report the completed fixed-weight baseline, which directly addresses the reviewer's main concern.
W1: Predictive power of the theoretical analysis
We agree that our new error analysis is a consistency check on a trained model. The prediction errors in our theory inherently depend on the trained network and the chosen target, so measuring them necessarily requires a trained model. However, our theory goes beyond explaining the observed near-linear scaling: it identifies conditions under which the timestep dependence of prediction errors cancels the explicit factor, favoring shared weights, unlike the strongly time-dependent zero-skill solution, which we evaluated and found much worse (4.26 vs. 2.30 FID at 160 epochs).
Importantly, this implication was already tested in our original submission (Fig. 5): four independently learned timestep-bucket profiles nearly coincide, and time-dependent weights do not improve FID (4.92 vs. 4.96). Our new analysis further supports this cancellation: estimated preferred weights vary by only across timesteps versus for (Fig. 8(c)). Thus, the theory identifies a separate, testable implication beyond a post-hoc explanation of spectral scaling.
W2: Fixed-weight baseline and the value of learning
We have completed the requested baseline with a frozen PCA basis and fixed . At 600 epochs, it achieves 1.65 FID, compared with 1.90 for JiT, showing that the anisotropic target shape indeed accounts for much of the improvement. However, our full method, jointly learning weights and basis from , achieves 1.55 FID. This additional 0.10 improvement over the fixed baseline represents nearly 30% of the total FID reduction from JiT.
More importantly, the value of learning is not solely measured by its improvement over a carefully chosen fixed profile. The fixed baseline uses , obtained from our trained AnisoFlow model rather than determined independently. Our analysis does not predict , and the learned coefficient varies across model sizes ( for JiT-B/16 versus for JiT-H/16). For a new architecture, selecting would require additional tuning, potentially involving costly training sweeps, whereas AnisoFlow learns it automatically in a single training run.
Furthermore, learning provides a more general approach than fixing an analytic target shape. Our default initialization requires neither a precomputed PCA basis nor a manually specified weight profile. Instead, AnisoFlow learns both the weights and basis automatically, adapting to different architectures and representations with negligible overhead. Thus, the value of learning extends beyond outperforming a fixed profile: it provides an efficient, general way to discover suitable prediction targets across architectures and representations, while also achieving the best FID in our experiments.
W3: Empirical evidence for the shared operator
We conducted an additional analysis using the estimated error statistics from Appendix A.3, grouping 480,000 patches by spatial position and content. Across five spatial groupings, estimated preferred weights vary by at most , and by only for four of the five. Grouping patches by high-frequency content fraction yields variation across quartiles, while local patch variance produces a somewhat larger variation. In contrast, preferred weights vary by over three orders of magnitude across PCA directions. These results suggest that directional variation dominates spatial and content-dependent variation, supporting our choice of a simple shared operator.
We hope these additional experiments address the reviewer's central concerns. We would also appreciate clarification on what further evidence would be needed to change their assessment, and how they weigh the remaining weaknesses against the paper's empirical improvements, simplicity, and generality. We kindly ask the reviewer to reconsider their score.
Thanks for the additional experiments.
For W1, I still see the spectral analysis mainly as a consistency check, but the time-invariance prediction and the zero-skill comparison make the theoretical argument more convincing. For W2, the fixed-weight baseline answers my main question: much of the gain comes from the anisotropic profile itself, while learning provides a further improvement and avoids choosing the profile manually.
For W3, the spatial and content analysis gives reasonable support for the shared operator.
The limited gains in latent space remain a limitation, but I think the pixel-space results and the added controls make the paper substantially stronger, if the experiment is true.
Sooo, I am therefore inclined to raise my score to 3.
We thank the reviewer for the positive comments on the simplicity and generality of our learnable prediction target, the empirical improvements in pixel space, and the spectral structure of the learned weights and basis, which provides insights beyond the FID improvements.
W1: Checking the assumptions behind the near-linear scaling (e.g., uncorrelated noise and data errors)
Checking these assumptions exactly is difficult, for two reasons. First, the prediction target shapes the errors the network learns, while these errors in turn determine the preferred target. Second, AnisoFlow predicts a single combined target, so separating the noise- and data-estimation errors requires additional assumptions. Our analysis therefore treats the errors as locally fixed and explains the near-linear scaling under these conditions. We will state this clearly in the revised paper.
Within these limits, we add an error analysis of a trained AnisoFlow-H/16 model as a consistency check (Appendix A.3, Fig. 8). It estimates effective noise- and data-error scales that include their correlation, so it does not assume uncorrelated errors. The noise-error scale is nearly constant across directions (∝λi−0.09), while the data-error scale grows linearly (∝λi1.00). Their ratio scales as λi1.09, close to the λi1.07 of the learned weights.
W1: Whether the assumptions are picked to fit the results, and whether the theory predicts anything new
Beyond the near-linear scaling, our analysis predicts that the preferred weights stay approximately constant across timesteps. This is not obvious: the zero-skill solution, which corresponds to direction-wise EDM (Appendix A.2), scales with αt/σt and thus changes strongly over time. The time-bucketed experiment in the original submission supports our prediction (Fig. 5): the four timestep-bucket profiles nearly coincide, and time-dependent weights do not improve FID (4.92 for shared wi vs. 4.96 for time-dependent wi(b) after 160 epochs). Our new error analysis agrees: while αt/σt varies by 11.7× across timesteps, the estimated preferred weights vary by only 1.19× (Fig. 8(c)).
W2: Why learn the weights if they converge to a simple analytic curve
Our analysis suggests that the learned weights approximately follow wi≈cλi, but does not determine c in advance. Moreover, the coefficient depends on the model size: our smaller JiT-B/16 models learn c≈0.2, compared with c≈0.15 for JiT-H/16. Choosing c for a new model would therefore require a costly hyperparameter sweep over full training runs, whereas learning the weights adds negligible overhead (Table 1).
Nonetheless, we are running a baseline with frozen PCA basis and fixed weights wi=0.15λi, using the coefficient obtained from our trained model. Note that even if this baseline performs well, it uses a coefficient obtained from our trained model rather than one selected independently. Results will be added later during the rebuttal period.
W3: Patch-wise operator shared across the image, ignoring image content
We use a shared operator across patches because the prediction target primarily aims to capture low-level statistics, such as directional data variance and prediction difficulty, rather than image content. Making the target content-dependent would introduce additional complexity, whereas our shared operator already achieves substantial improvements in pixel space with negligible overhead. We are also investigating how prediction-error statistics vary across spatial positions and patch content, and will report additional results during the rebuttal period if available.
W3: Limited gains in latent space, where AnisoFlow falls back to the default target
We do not claim FID improvements in latent space. The point of these experiments is that AnisoFlow does not simply learn the linear scaling but adapts to each representation: for SiT-REPA and RAEv2, it learns profiles close to their default velocity and data prediction targets (Fig. 7). This is why the target should be learned rather than fixed.
We hope our responses and additional experiments address the reviewer's concerns, and we kindly ask them to consider raising their score.