acceptodds
Under review as a conference paper at ICLR 2027

AREX: Affine–Residual Exponential Integrator for Few-Step Sampling in Flow Matching

Abstract

We introduce AREX, a training-free sampler for pretrained flow matching models that uses the target mean and covariance to capture an analytically tractable part of the sampling dynamics. We show that the velocity field of the moment-matched Gaussian target is the -optimal affine approximation to the marginal velocity field. This motivates decomposition of the learned dynamics into an affine component over the whole sampling path, determined by the first two target moments, and a neural residual term. AREX keeps the affine component and integrates it using an explicit matrix-valued propagator. In turn, we only require to integrate over the residual term. This differs from scalar exponential integrators, which analytically handle only isotropic linear dynamics. Across image and text-to-image generation tasks, AREX consistently improves sample fidelity in the few-step sampling regime without retraining the underlying model.

Primary area:
generative models
Keywords:
Generative models, flow matching, few-step sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_71vcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a novel training-free fast sampling method for the flow-matching generative models. Based on the idea of exponential integrator, the author splits the velocity field into the affine velocity field and the residual to the true velocity field, integrate the affine velocity field exactly, and put the residual into an integral with Adams-Bashforth approximation. Theoretical derivations demonstrated several nice properties of the proposed method. Comprehensive experiments in unconditional, class-conditional, and text-conditional generations show the effectiveness of the method under different circumstances.

Critical strengths

This paper provides a novel idea that splits the velocity field into the Gaussian affine velocity field part and its residual part, and then apply the exponential integrator to the splitted form. Rigorous derivations are provided inside of the paper. The experiments are comprehensive and covers different types of generation tasks.

Critical weaknesses

See the questions below.

Questions for authors

The paper is solid in general. Here are some of my questions:

  1. For the text-conditional generations, how are the distribution mean and covariance matrix estimated? More precisely, it's mentioned inside Appendix E.2 that it's done by "a self sampling for 2000 template prompts and 25 samples for each prompt", how are these prompts selected/created?
  2. Would the estimation of mean and covariance matrix largely affect the final generation quality in the text-conditional case? For example, when a new prompt is given for text-conditional generation, does the generation quality of the image relates to how closely the new prompt is related to the estimation template prompts?
1 comment
AuthorsOpenReview ↗

Thank you for the positive feedback

Thank you for the careful review and the constructive comments.

Calibration prompts and moment estimation (Q1). Since the SANA training set is not available, we estimate the target statistics from model self-samples. We construct 2,000 template prompts by independently sampling a subject, a style, and a modifier uniformly from 20 subjects, 10 styles, and 10 modifiers. This produces 1,263 distinct prompts, none of which appears in MJHQ-30K.

For each template prompt, we generate 25 samples using the released SANA pipeline with 20 sampling steps and CFG scale w=4.5w=4.5, giving 50,000 latent samples in total. As described in Appendix B, the pooled mean m1m_1 is estimated by averaging all 50,000 latents, while the within-condition covariance Sˉw\bar S_w is estimated by averaging the unbiased sample covariances of the 25 samples for each template prompt. We do not store a separate conditional mean for each prompt. Instead, at sampling time, the first network evaluation provides an endpoint prediction x^1,w\hat x_{1,w}, from which we estimate the trajectory-specific affine center as m1+α(x^1,w−m1)m_1+\alpha(\hat{x}_{1,w}-m_1). The per-template sample means are used only during calibration to fit the scalar α\alpha, whose fitted value is 0.261 on SANA. We have added the details of the template prompt generator to Appendix E.2.

Sensitivity to calibration prompts (Q2). We examined both the amount of calibration data and the coverage of the template prompts. First, we re-estimated the calibrated statistics using different numbers of conditions and self-samples per condition. The following table reports FID for AREX-C-AB2 on SANA:

Calibration settingConditions × samples (Ccal×KC_{\rm cal}\times K)dof / ddFID NFE 4FID NFE 8
Fewer samples2000 × 20.2499.5112.1
Fewer samples2000 × 50.989.07.0
Fewer samples2000 × 102.28.46.6
Fewer conditions125 × 250.3755.161.4
Fewer conditions250 × 250.7314.212.3
Fewer conditions500 × 251.58.56.7
Fewer conditions1000 × 252.98.46.5
Full calibration2000 × 255.98.36.5

Here, dof=Ccal(K−1)\mathrm{dof}=C_{\rm cal}(K-1) is the effective degrees of freedom of the within-condition covariance estimate, and d=8192d=8192 is the latent dimension. The covariance estimate is sensitive to the amount of calibration data. When dof<d\mathrm{dof}<d, the estimate is rank-deficient, and generation quality can deteriorate substantially. With 2,000 conditions and 10 samples each, the degrees of freedom reach approximately 2.2d2.2d, and the FID is already within 0.1 of the full-calibration result at both NFEs. Thus, reliable covariance estimation requires enough samples, but the full 50,000-sample calibration is not necessary to obtain similar FID in this experiment. In contrast, the fitted α\alpha is relatively stable across calibration subsets, varying from 0.257 to 0.262 compared with 0.261 using the full calibration set.

Second, we examined whether generation quality depends on how closely the evaluation prompts match the calibration templates. We grouped MJHQ-30K categories by their coverage in the templates before evaluating the results. We evaluated each group on disjoint prompts from the remaining 29,000 MJHQ-30K samples, using approximately 2,000–3,000 images for each subgroup FID comparison. The smaller evaluation sets lead to relatively high absolute FID values, so comparisons are most meaningful between samplers within the same group.

FID: AREX-C-AB2 vs. DPM-Solver-v3NFE 4NFE 8
Covered (animals, food, indoor, landscape, vehicles)27.08 vs. 29.2724.49 vs. 25.39
Partly covered (art, plants)27.39 vs. 30.9424.65 vs. 26.10
Not covered (fashion, logo, people)28.73 vs. 32.2425.76 vs. 26.64

AREX-C-AB2 improves over DPM-Solver-v3 in all three groups. At NFE 4, the FID improvements are 2.19, 3.55, and 3.51 for the covered, partly covered, and uncovered groups, respectively. At NFE 8, the corresponding improvements are 0.90, 1.45, and 0.88. Thus, the relative improvement does not systematically decrease as template coverage decreases. This is consistent with the construction of AREX-C. The covariance and residual profile are shared across prompts, while the affine center is estimated separately for each new prompt using its first network evaluation. The shared covariance determines the affine dynamics around this center, and the residual profile adjusts the numerical integration. Therefore, the method does not require a new prompt to closely match one of the calibration templates.

Overall, these results show that sufficient calibration samples are important for accurately estimating the covariance, while category-level similarity to the calibration prompts is not necessary for the observed gains on MJHQ-30K.

Reviewer_NdFYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • AREX splits a pretrained flow-matching velocity field into a moment-matched affine component and a residual. It integrates the affine component exactly and applies multistep quadrature to the residual.
  • The practical method includes an estimated residual magnitude profile and an extension for conditional generation. Experiments on CIFAR-10, ImageNet, and MJHQ-30K show the largest gains at low NFE.
Critical strengths
  • The decomposition has a clear mathematical basis. Showing that the moment-matched Gaussian field is the optimal affine approximation gives a convincing reason to integrate this component exactly.
  • The low-NFE gains are substantial, particularly on CIFAR-10 and SANA. The baseline comparisons, component ablations, and calibration-cost measurements provide fairly thorough support for the method.
Critical weaknesses
  • SANA parameter selection includes samples used in the final evaluation. Although disclosed and also relevant to baseline selection, this prevents a fully independent assessment of the gains.
  • Node position is a high-impact tuning parameter. Appendix E.1 reports FID changes of roughly 17 on SiT and 23 on SANA at NFE 4, making performance without per-backbone selection uncertain.
  • Residual profiling accounts for a substantial part of the improvement. On SiT at NFE 4, AREX-C-AB3 improves from 19.85 without profiling to 8.18 with it; the main results therefore depend on more than exact affine transport.
  • The main global multistep theorem excludes profiling. The separate profiled one-node analysis does not fully cover the practical profiled AB2/AB3 samplers.
  • Dense covariance operations limit scalability. The largest tested dimension is 8,192, and the paper acknowledges that the dense construction becomes impractical around 10510^5 dimensions.
Questions for authors
  • Do the SANA gains persist with disjoint tuning and evaluation prompts?
  • How well do the selected node positions and residual profiles transfer to a different prompt distribution without retuning?
2 comments
AuthorsOpenReview ↗

Thank you for your positive assessment

We thank the reviewer for the careful review and constructive comments.

Disjoint selection and evaluation on SANA (W1, Q1). We agree that the original hyperparameter selection used prompts included in the final MJHQ-30K evaluation. To address this, we retain the first 1,000 MJHQ-30K prompts for hyperparameter selection and evaluate the selected configurations on the remaining 29,000 disjoint prompts. We use the original noise seeds and recompute the reference statistics on these 29,000 prompts.

FIDNFE 4NFE 6NFE 8NFE 10
DPM-Solver++23.7711.859.078.18
STORK-473.4316.227.836.85
DPM-Solver-v310.828.157.297.04
AREX-C-AB28.296.696.496.51
AREX-C-AB310.087.026.686.61

The conclusions of Table 3 remain unchanged: AREX-C-AB2 achieves the lowest FID at every NFE from 4 to 10, and AREX-C-AB3 also outperforms every baseline. At NFE 4, for example, AREX-C-AB2 achieves 8.29 FID compared with 10.82 for DPM-Solver-v3; at NFE 8, the corresponding values are 6.49 and 7.29. The previously selected ρ=0.35\rho=0.35 also remains the best candidate at NFE 4 on the disjoint evaluation set. We will include these results and update the selection protocol in Appendix E.1.

Sensitivity to node position ρ\rho (W2). We agree that ρ\rho has a substantial effect at low NFEs. It is the only continuously tuned sampling hyperparameter of AREX, and we select it per backbone and NFE on a small calibration split, just as the discrete configurations of the competing samplers are selected. For comparison, DPM-Solver-v3 selects among 16 configurations. The sensitivity is particularly pronounced at NFE 4. As reported in Appendix E.1, moving from ρ=0.05\rho=0.05 to ρ=0.35\rho=0.35 increases the held-out FID of SiT AREX-C-AB3 from 21.08 to 38.32. Conversely, on SANA, moving from ρ=0.35\rho=0.35 to ρ=0.05\rho=0.05 increases the held-out FID of AREX-C-AB2 from 53.62 to 76.54. These are small-split selection FIDs rather than the final evaluation FIDs. They show that a single fixed ρ\rho is not equally effective across backbones, which is why we select it separately.

We also studied whether the selected ρ\rho remains effective on disjoint prompts. On SANA, the value ρ=0.35\rho=0.35, selected using the first 1,000 MJHQ-30K prompts, remains the best candidate at NFE 4 when evaluated on the remaining 29,000 prompts. This suggests that the selected value is not specific to the original 1,000 samples, although it does not establish transfer to an entirely different prompt distribution. The sensitivity to ρ\rho is consistent with the tradeoff described in Theorem 1. Moving the evaluation node into the step can reduce temporal interpolation error, but increases the error from evaluating the residual at an affine-only intermediate state. The residual profile characterizes how residual magnitude varies along the trajectory, but does not by itself determine the optimal ρ\rho.

Contribution of residual profiling and theory for the practical sampler (W3, W4). We agree that residual profiling contributes substantially to the reported improvements. Table 14 shows that, at NFE 4, profiling improves AREX-AB2 on CIFAR-10 from 23.67 to 12.20 FID, AREX-C-AB3 on SiT from 19.85 to 8.18, and AREX-C-AB2 on SANA from 13.59 to 8.15.

The benefit also depends on applying profiling to the residual rather than directly to the full velocity field. For example, Table 15 shows that profiling classical AB3 on SiT at NFE 4 worsens FID from 12.35 to 14.59, whereas profiling the corresponding AREX-C-AB3 residual improves FID from 19.85 to 8.18. Thus, the practical gains come from combining exact affine transport, multistep residual integration, and residual profiling, rather than from the affine backbone alone. The original global theorem analyzes unprofiled AREX-AB$p$, while Proposition 5 analyzes profiling only for the one-node rule. We agree that these results did not directly cover the profiled AB2/AB3 samplers used in our main experiments.

We have added an error bound for the practical profile scheme; see Proposition 5 in Subsec. C.8. This result separates the roles of the two analyses. The unprofiled theorem establishes the order-pp result for ρ=0\rho=0 with a matching-order startup. The profiled bound directly controls the finite-step error of the practical sampler through Aϕ\mathcal A_\phi. It does not guarantee that profiling always reduces the error or preserves formal AB$p$ order. For a smooth, strictly positive profile, Γn(t)→I\Gamma_n(t)\to I uniformly as the grid is refined, so the profiled scheme is consistent; a flat profile recovers the unprofiled scheme. The revised theory therefore covers both the underlying affine–residual integrator and the profiled AB2/AB3 implementations used in our experiments.

Scalability of dense covariance operations (W5). We agree that dense covariance eigendecomposition limits the current implementation to moderate latent dimensions. As reported in Appendix E.2, the largest tested dimension is d=8192d=8192 on SANA, where forming and eigendecomposing the covariance takes 0.7 seconds on one GH200. The dense implementation has cubic eigendecomposition cost and quadratic covariance storage, making it impractical at much larger dimensions.

One approach for larger latent spaces is to restrict the covariance to a structured form. In our qualitative FLUX experiment, we used

Σ1≈IT⊗SC,\Sigma_1 \approx I_T\otimes S_C,

where TT is the number of packed latent tokens and SCS_C is a shared feature-space covariance. For FLUX.1-dev at 1024px, C=64C=64 and the full latent dimension is d=262144d=262144. This construction requires eigendecomposition of only a 64×6464\times64 matrix rather than a dense d×dd\times d matrix, while retaining the mean at full token resolution.

The structured approximation changes the affine component but not the underlying learned ODE, since the residual is still computed as r=vθ−vGr=v_\theta-v_G. However, this is a different approximation from the dense-covariance method evaluated in our main experiments, and the FLUX experiment was intended as a qualitative demonstration. We will clarify in Appendix E.2 that the stated scalability limit applies to dense eigendecomposition and distinguish it from the structured approximation.

Transfer to a different prompt distribution (Q2). The SANA experiments already test transfer of the calibrated statistics across prompt distributions. We estimated the covariance and coefficient α\alpha from self-samples of 2,000 synthetic template prompts, with 25 samples per prompt, and estimated the residual profile using 1,024 synthetic prompt trajectories. None of these calibration prompts overlaps with MJHQ-30K. The resulting covariance, α\alpha, and profile were applied to MJHQ-30K without re-estimation.

To examine whether the improvement depends on similarity to the calibration prompts, we also grouped MJHQ-30K categories by their coverage in the synthetic templates before evaluating the results. We evaluated each group using disjoint prompts from the remaining 29,000 MJHQ-30K samples. Each group contains approximately 2,000–3,000 images, so its absolute FID is relatively high; comparisons should be made between samplers within the same group.

FID: AREX-C-AB2 vs. DPM-Solver-v3NFE 4NFE 8
Covered (animals, food, indoor, landscape, vehicles)27.08 vs. 29.2724.49 vs. 25.39
Partly covered (art, plants)27.39 vs. 30.9424.65 vs. 26.10
Not covered (fashion, logo, people)28.73 vs. 32.2425.76 vs. 26.64

AREX-C-AB2 improves over DPM-Solver-v3 in all three groups. At NFE 4, the FID improvements are 2.19, 3.55, and 3.51 for the covered, partly covered, and uncovered groups, respectively. At NFE 8, the corresponding improvements are 0.90, 1.45, and 0.88. The improvement therefore does not systematically decrease with template coverage.

These results are consistent with the structure of AREX-C. The covariance and residual profile are shared across prompts, while the conditional affine center for a new prompt is estimated from its first network evaluation. Consequently, calibration does not require prompts that closely match each evaluation prompt.

The node position ρ\rho, however, was selected separately using the first 1,000 MJHQ-30K prompts. The disjoint evaluation confirms that the selected value remains effective within MJHQ-30K, while the synthetic-to-MJHQ comparison establishes transfer of the covariance, α\alpha, and profile without recalibration. We have not separately tested whether the selected ρ\rho transfers to an entirely different prompt distribution without retuning.

Reviewer_n8bKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AREX, a training-free sampler for pretrained flow matching models. The idea is to decompose the learned velocity field into (i) an affine component induced by a moment-matched Gaussian target (same mean and covariance as the data), which the authors show is the L2-optimal affine approximation to the population velocity field (Prop. 1) and which admits a closed-form matrix-valued propagator (Prop. 3), and (ii) a residual term that is integrated numerically with an Adams–Bashforth multistep rule at one network evaluation per step, optionally reweighted by a pre-estimated residual magnitude profile. A conditional variant (AREX-C) uses the average within-condition covariance and a per-trajectory affine center estimated from the first network evaluation via a scalar calibration coefficient. Experiments on CIFAR-10 (I-CFM), ImageNet-256 (SiT-XL/2), and MJHQ-30K (SANA-0.6B) show consistent FID improvements over seven training-free baselines in the low-NFE regime (NFE ≤ 8), with the advantage vanishing or slightly inverting at NFE ≥ 16–32.

Critical strengths
  1. AREX integrates a matrix-valued affine flow determined by the target covariance eigendirections, over the full trajectory (in contrast to Gaussian teleportation, which uses the Gaussian approximation only for the initial segment). The construction is well motivated by Proposition 1 (L2-optimality)

  2. Strong gains exactly where few-step sampling is hard.

  3. Careful experimental protocol and comprehensive ablations

Critical weaknesses
  1. FLUX appears nowhere else in the paper: no resolution, no covariance estimation protocol, no cost, no baseline comparison, no metric. Worse, App. E.2 states that for d ≳ 10^5 the dense eigendecomposition "is not eligible any more", while FLUX.1-dev at 1024px has latent dimension d ≈ 2.6×10^5. How was this figure produced? As it stands it is marketing rather than evidence, and it silently contradicts the stated scalability boundary.

  2. The held-out split used to select ρ and the multistep order consists of the first 1,000 prompts of MJHQ-30K with the same noise seed as the reported 30k run (disclosed in App. E.1). The Table 3 results should be re-run with selection performed on a truly disjoint prompt set.

  3. In Table 18, AREX-C's FID advantage coexists with consistently lower recall than CAB from NFE 4 to 16 (e.g., NFE 8: .557 vs. .580). Whether the method trades diversity for precision deserves discussion.

  4. The paper motivates AREX primarily through covariance anisotropy, but in Table 16 on SiT replacing pooled Σ1 with the within-condition covariance changes almost nothing (12.58 → 12.54), and the conditional mean does the heavy lifting (→ 8.18); on SANA the conditional mean even hurts at NFE 4 (7.81 → 8.15). The paper should openly discuss which ingredient drives gains on which benchmark, rather than attributing everything to direction-dependent covariance dynamics.

Questions for authors
  1. Please provide the full recipe for Figure 1: resolution, latent dimension, how the covariance and its eigendecomposition were computed at that scale, the offline cost, and a baseline comparison at equal NFE. If this cannot be documented, please replace the figure with a reproducible demo.
  2. How sensitive are S̄_w, the residual profile, and α to the number of self-samples K per condition and the number of calibration conditions |C_cal|? What is the fitted value of α in practice?
  3. On SiT you used training-set latent statistics, while the AREX-C theory targets the terminal distribution of the w=1.5-guided model. Would model self-samples (as on SANA) further improve the SiT numbers?
  4. How do the gains, the selected ρ, and the profile shape vary with the guidance scale w?
  5. AB2 beats AB3 on SANA but loses on SiT; you attribute this to fast residual variation. Would per-step adaptive order selection help, and can the profile predict it?
  6. At NFE ≥ 16 the affine backbone no longer helps (and slightly hurts). Is there a principled way to anneal its contribution with grid refinement?
4 comments
AuthorsOpenReview ↗

Thank you for the careful review

Thank you for the thoughtful comments.

FLUX demo (W1, Q1). We agree that the FLUX demo was insufficiently documented. For FLUX.1-dev at 1024px, the latent dimension is d=262144d=262144. We did not compute a covariance in the full flattened latent space. Instead, we used a structured approximation Σ1≈IT⊗SC\Sigma_1\approx I_T\otimes S_C in the packed latent representation, with feature dimension C=64C=64. This requires eigendecomposition of only a 64×6464\times64 covariance matrix, while the mean is retained at full token resolution. This approximation changes only the affine component used by AREX. The residual is still computed from the original pretrained velocity field as r=vθ−vGr=v_\theta-v_G, so the underlying learned ODE remains unchanged. This experiment was intended as a qualitative demonstration. We will include details in the appendix.

Disjoint selection and evaluation on SANA (W2). We agree that the original selection split overlapped with the MJHQ-30K evaluation set. To address this, we retain the first 1,000 MJHQ-30K prompts for hyperparameter selection and evaluate the selected configurations on the remaining 29,000 disjoint prompts. We use the original noise seeds and recompute the reference statistics on these 29,000 prompts.

FIDNFE 4NFE 6NFE 8NFE 10
DPM-Solver++23.7711.859.078.18
STORK-473.4316.227.836.85
DPM-Solver-v310.828.157.297.04
AREX-C-AB28.296.696.496.51
AREX-C-AB310.087.026.686.61

The conclusion of Table 3 remains unchanged: AREX-C-AB2 achieves the lowest FID at every NFE from 4 to 10, and AREX-C-AB3 also outperforms every baseline. The previously selected ρ=0.35\rho=0.35 also remains the best candidate at NFE 4 on the disjoint evaluation set. We report these results and update the selection protocol in Appendix E.1.

Precision and recall (W3). We agree that AREX-C has lower recall than CAB despite its better FID. In Table 18, at NFE 4, 8, and 16, AREX-C-AB3 achieves recall of 0.502, 0.557, and 0.572, compared with 0.521, 0.580, and 0.599 for CAB. Its precision is higher at the same NFEs: 0.727, 0.802, and 0.814 versus 0.686, 0.781, and 0.799 for CAB. With training-set moments, AREX-C has the highest recall at NFE 4 and 8 and ties CAB at NFE 16.

These results suggest a precision–recall tradeoff on SiT, although the recall differences are relatively small. At NFE 8, recall among all methods except Euler lies between 0.546 and 0.589. At NFE 32, the recall gap between AREX-C-AB3 and CAB decreases to 0.015 (0.584 versus 0.599). We will clarify this tradeoff in the discussion rather than treating the FID improvement as an improvement in every metric.

Contributions of covariance, conditional mean, and profiling (W4). We agree that the contributions differ across benchmarks. To isolate the effect of covariance anisotropy, we replaced the full covariance in the affine component with either its diagonal or its isotropic average (trΣ/d)I(\mathrm{tr} \Sigma/d)I, keeping all other settings fixed.

CovarianceCIFAR-10 (NFE 2/4/8)SiT-XL/2 (NFE 2/4/8)SANA (NFE 4/8)
Full39.2 / 12.2 / 5.484.2 / 8.2 / 3.08.15 / 6.33
Diagonal67.0 / 16.2 / 6.989.7 / 11.2 / 3.88.56 / 6.51
Isotropic76.0 / 24.4 / 7.468.2 / 8.6 / 3.78.13 / 6.55

The full covariance contributes substantially on CIFAR-10. At NFE 4, for example, replacing it with the diagonal or isotropic covariance increases FID from 12.2 to 16.2 or 24.4. On the guided latent models, the contribution is more mixed. On SiT, the isotropic covariance performs better at NFE 2, while the full covariance performs best at NFE 4 and 8. On SANA, the full and isotropic covariances perform similarly at NFE 4, but the full covariance is better at NFE 8.

The conditional mean also contributes differently across benchmarks, as shown in Table 16. On SiT at NFE 4, replacing the pooled covariance with the within-condition covariance has little effect (12.58 to 12.54), while introducing the conditional mean reduces FID to 8.18. On SANA, the within-condition covariance improves FID from 9.49 to 7.81, while introducing the conditional mean increases it to 8.15. At NFE 8, however, the conditional mean improves SANA's FID from 6.44 to 6.33.

Residual profiling is another important contributor. In Table 14, at NFE 4, profiling improves AREX-C-AB3 on SiT from 19.85 to 8.18 FID and AREX-C-AB2 on SANA from 13.59 to 8.15. Applying profiling directly to the full velocity field does not consistently give the same benefit. For example, Table 15 shows that profiling classical AB3 on SiT at NFE 4 worsens FID from 12.35 to 14.59.

Thus, the gains come from different components on different benchmarks. Covariance anisotropy contributes strongly on CIFAR-10, while the conditional mean and residual profiling are especially important on the guided latent models.

Sensitivity of calibrated statistics (Q2). AREX-C uses three calibrated statistics: the average within-condition covariance Sˉw\bar S_w, the coefficient α\alpha, and the residual magnitude profile ϕ\phi. We examined their sensitivity to the number of self-samples KK per condition and the number of calibration conditions ∣Ccal∣|\mathcal C_{\rm cal}|.

For covariance estimation, we re-estimated Sˉw\bar S_w, α\alpha, and the profile using subsets of the calibration data. The effective degrees of freedom (dof) of the covariance estimate are ∣Ccal∣(K−1)|\mathcal C_{\rm cal}|(K-1). We report this relative to the latent dimension dd, which is 4,096 for SiT and 8,192 for SANA. Within each row, the ratios and FIDs follow the order of the listed configurations in Table 19.

The covariance estimate is sensitive to the available degrees of freedom. When the dof is below dd, the estimated covariance is rank-deficient, and sampling quality degrades sharply. Once the dof exceeds approximately 2d2d, FID is within 1.1 of the full-calibration result at NFE 4 and within 0.1 at NFE 8. The configurations used in the paper have approximately 6d6d degrees of freedom.

The fitted α\alpha is considerably more stable:

Fitted α\alphaFewer samples (K=2/5/10K=2/5/10)Fewer conditionsFull calibration
SiT0.963 / 0.956 / 0.9620.951 / 0.933 / 0.953 (100 / 250 / 500 classes)0.955
SANA0.262 / 0.262 / 0.2620.262 / 0.257 / 0.260 / 0.260 (125 / 250 / 500 / 1000 prompts)0.261

We also evaluated the smallest and largest fitted values of α\alpha, keeping all other settings unchanged:

FID, NFE 4 / 8Smallest α\alphaFull calibrationLargest α\alpha
SiT8.71 / 3.00 (α=0.933\alpha=0.933)8.18 / 3.02 (α=0.955\alpha=0.955)7.98 / 3.03 (α=0.963\alpha=0.963)
SANA8.26 / 6.51 (α=0.257\alpha=0.257)8.29 / 6.49 (α=0.261\alpha=0.261)8.30 / 6.50 (α=0.262\alpha=0.262)

On SiT, the NFE 4 FID varies from 7.98 to 8.71, while the NFE 8 results differ by at most 0.02. On SANA, the differences are at most 0.03 at NFE 4 and 0.02 at NFE 8.

Finally, we tested the residual profile with fewer calibration trajectories and conditions, keeping Sˉw\bar S_w and α\alpha fixed. The following ranges summarize the results:

Profile calibrationConfigurationsFID range, NFE 4FID range, NFE 8
SiT, one trajectory per class125–1000 trajectories8.11–8.263.02–3.05
SiT, fixed 2048 trajectories100–1000 classes8.14–8.193.02–3.04
SANA, one trajectory per prompt128–1024 prompts8.15–8.296.49–6.52
SANA, fixed 1024 trajectories128–1024 prompts, 8–1 trajectories each8.22–8.296.49–6.51

Across these settings, FID stays within 0.14 of the full-calibration result at NFE 4 and within 0.03 at NFE 8, even with only 125 trajectories or 100 classes. This is consistent with the smaller number of quantities being estimated: at each time, the profile contains dd root-mean-square values, one per direction, whereas the covariance has d(d+1)/2d(d+1)/2 entries.

Thus, covariance estimation requires substantially more calibration data, while α\alpha and the residual profile are relatively stable across the tested subsets.

Self-sampled moments on SiT (Q3). We compared training-set latent statistics with moments estimated directly from self-samples of the guided SiT model. For self-sampling, we generated 25 samples per ImageNet class at w=1.5w=1.5. For training-set statistics, we used all 1,281,167 ImageNet training images and their horizontal flips. The results for AREX-C-AB3 are:

FIDNFE 2NFE 4NFE 8NFE 16NFE 32
Self-samples84.158.183.022.292.15
Training set84.389.522.972.222.11

Self-sampled moments improve FID by 1.34 at NFE 4. At the other NFEs, the two estimates differ by at most 0.23 FID, with training-set statistics performing slightly better at NFE 8–32.

The NFE 4 improvement is consistent with the AREX-C construction, which uses moments of the guided model's terminal distribution. Self-samples estimate these moments directly, whereas training-set statistics describe the unguided data distribution. However, the benefit is not consistent across all NFEs. The offline costs are similar: 87 minutes for self-sampling and 82 minutes for the training-set approach on one GH200. In the revised paper, we use self-sampled moments as the canonical choice for SiT, consistent with SANA.

Effect of guidance scale (Q4). We evaluated additional guidance scales on both SiT and SANA, re-estimating all calibration statistics from guided self-samples at each scale and selecting the options of every method as in the main experiments. The table compares AREX-C with the best baseline at each NFE:

Model and guidanceNFE 4: AREX-C vs. best baselineNFE 8: AREX-C vs. best baselineSelected ρ\rho
SiT, w=1.25w=1.2511.11 vs. 14.474.03 vs. 5.340.05 / 0.05
SiT, w=2w=27.02 vs. 6.663.37 vs. 3.330.05 / 0.2
SiT, w=4w=410.46 vs. 7.508.71 vs. 8.240.2 / 0.35
SANA, w=3w=36.96 vs. 9.276.51 vs. 6.170.2
SANA, w=6w=612.43 vs. 10.996.92 vs. 7.320.35

As guidance increases, the fitted α\alpha decreases, the selected node moves further into the step, and the advantage of AREX-C shrinks. On SiT, AREX-C is ahead of the best baseline at w=1.25w=1.25 but behind it at w=2w=2 and w=4w=4. On SANA, AREX-C is ahead at NFE 4 for w=3w=3 and at NFE 8 for w=6w=6, and behind in the other two cases.

On SiT with stronger guidance, several few-step samplers reach lower FID than the converged ODE, which has FID 3.62 at w=2w=2 and 12.13 at w=4w=4; for example, the best baseline reaches 3.33 at NFE 8 and 7.50 at NFE 4 (AREX-C: 3.37 and 10.46). FID in these settings therefore no longer measures integration accuracy.

We also measured the residual magnitude profile across guidance scales:

∥ϕ(t)∥\|\phi(t)\|t=0t=00.250.50.750.98Peak (at tt)
SiT, w=1.25w=1.250.418.633.844.112.844.1 (0.74)
SiT, w=1.5w=1.50.820.735.545.213.345.2 (0.74)
SiT, w=2w=22.325.138.647.114.547.1 (0.72)
SiT, w=4w=414.342.749.052.218.252.6 (0.70)
SANA, w=3w=365.348.859.160.115.365.3 (0.01)
SANA, w=4.5w=4.5126.560.165.563.816.8126.5 (0.01)
SANA, w=6w=6188.769.971.367.118.5188.8 (0.02)

The profile shapes differ between the two backbones. On SiT, the residual magnitude increases from near t=0t=0, peaks around t=0.7t=0.7, and then decreases toward t=1t=1. On SANA, it has a sharp peak near t=0t=0 and a relatively flat middle portion.

Increasing guidance mainly increases the residual magnitude near the start of the trajectory. On SiT, from w=1.25w=1.25 to w=4w=4, the magnitude at t=0t=0 increases from 0.4 to 14.3, while the peak increases from 44.1 to 52.6 and remains near t=0.7t=0.7. On SANA, from w=3w=3 to w=6w=6, the initial peak increases from about 65 to 189, while the middle portion increases more moderately. The magnitude near t=1t=1 changes much less on both backbones.

These experiments characterize how AREX-C's performance, selected node position, and residual profile change with guidance. We have not repeated the full baseline comparison at every additional guidance scale, so these results do not establish how the relative gains over all baselines vary with ww.

Per-step adaptive order selection (Q5). Adaptive selection between AB2 and AB3 is possible in principle. Both methods reuse previously computed residual evaluations, so switching between them would not necessarily require additional NFEs. The main challenge is deciding which order to use at each step.

The residual magnitude profile provides information about how residual scale changes along the trajectory, but it does not directly estimate the local interpolation error of AB2 or AB3. A reliable adaptive rule would therefore require an additional error criterion or a calibrated order-selection policy. We have not evaluated such a rule.

At the published guidance scales, the fixed-order results differ by at most 0.21 FID from NFE 8 onward on both SiT and SANA. An adaptive method might improve on both fixed orders, but the magnitude profile alone does not establish when such switching would help. We leave this extension for future work.

Affine backbone at higher NFEs (Q6). We focus on the low-NFE regime, where AREX provides its largest improvements. At higher NFEs, the discretization errors of the competing higher-order samplers become smaller, and their FIDs approach similar values.

On SiT, at NFE 16, the leading four samplers lie within 0.16 FID of one another. At NFE 32, all methods except Euler and the one-node rule lie within 0.37 FID. On CIFAR-10, AREX-AB2 is within 0.09 FID of the best baseline at NFE 32.

A grid-dependent reweighting of the affine component could be considered, but we have not derived or evaluated a principled annealing schedule. Our current results show that the benefit of the affine decomposition is concentrated at low NFEs, while the differences between higher-order solvers diminish as the grid is refined. We will mention adaptive backbone weighting as a possible direction for future work.

Thank you for the comprehensive responses and additional experiments. The disjoint 29,000-prompt SANA evaluation resolves my concern about overlap between hyperparameter selection and evaluation. The full/diagonal/isotropic covariance ablations, the explicit precision–recall discussion, and the calibration, self-sample, and guidance studies also substantially clarify the sources and robustness of the reported gains. In addition, the structured-covariance explanation removes the apparent contradiction between the FLUX demonstration and the stated dense-eigendecomposition scalability limit.

Two parts of my original questions remain incompletely addressed. First, the FLUX demonstration still lacks the requested offline construction/sampling cost and an equal-NFE quantitative baseline. It should therefore be presented strictly as a qualitative result using the IT⊗SCI_T\otimes S_C approximation, rather than as evidence that the dense AREX implementation itself scales to d=262,144d=262{,}144. Second, adaptive AB2/AB3 order selection and a principled annealing schedule for the affine backbone at higher NFEs remain open: the residual profile has not been shown to predict the appropriate order, and no backbone-annealing rule has been derived or evaluated.

Thus, most of my central empirical concerns have been addressed, but the FLUX reproducibility request and these two applicability questions remain open. On balance, I maintain my overall rating of 3. I would view a revision that incorporates the disjoint evaluation and new ablations, while appropriately scoping the FLUX claims, as substantially improved.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reviews and constructive comments. We have conducted additional experiments and extended the theoretical analysis to address their concerns. We have uploaded a revised version of the manuscript, with all changes highlighted in blue. In particular, the following additions and clarifications are incorporated into the revised manuscript:

  • Disjoint SANA evaluation. We reran the MJHQ-30K evaluation with disjoint hyperparameter-selection and evaluation prompts. The main conclusions remain unchanged: AREX-C-AB2 achieves the best FID at every reported NFE from 4 to 10.

  • Expanded component ablations. We added comparisons of full, diagonal, and isotropic covariances to clarify the contribution of anisotropy. We will also discuss the benchmark-dependent roles of the conditional mean and residual profiling, as well as the observed precision–recall tradeoff.

  • Calibration sensitivity and generalization. We ran additional experiments varying the number of calibration conditions and self-samples, the fitted coefficient α\alpha, the residual profile, and the guidance scale. We also evaluated performance across prompt categories with different coverage in the calibration templates.

  • Extended theory and implementation details. We derived a global finite-step error bound for the practical profiled AREX-AB2/AB3 samplers. We will also clarify the structured covariance approximation used in the qualitative FLUX demonstration and distinguish it from the scalability limits of dense covariance eigendecomposition.

Detailed results and explanations are provided in the individual reviewer responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.