- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether the existence of an endpoint-correct generator within a model class is sufficient for conditional flow matching (CFM) to select such a generator. Its main contributions are:
- Constructing an affine family with Gaussian distributions in which a correct generator exists, but every admissible positive-definite input-measurable weight excludes it.
- Characterizing whether reweighting can select a prescribed velocity field under specified information restrictions.
- Deriving the minimum global spectral contrast required for selection.
- Analyzing the asymptotic variance of particular weighted-regression estimators near the selection obstruction.
- Constructing an endpoint-derived moment equation that remains regular in the family under consideration.
Critical strengths
1. rigorous distinction between representability and objective-based selection
- The main construction starts with a model class containing an endpoint-correct generator.
- The result goes beyond showing reference-velocity misspecification.
- It excludes every endpoint-correct generator in the class under the stated weight restrictions.
- Allowing state-dependent positive-definite matrix weights makes the result stronger than a counterexample to uniform weighting.
- The positive endpoint-error bound distinguishes the obstruction from a merely formal discrepancy.
2. quantitative characterization of reweighting feasibility and cost
- The paper characterizes selection of a prescribed candidate in affine parameter classes.
- The minimum spectral contrast explicitly depends on the information available to the weight.
- The analysis shows that state-dependent and time-only weights can incur different costs.
- It distinguishes changes to the population objective from importance sampling that preserves the objective.
- The comparison with an endpoint-derived moment equation clarifies that the variance obstruction concerns a particular estimator protocol, rather than every possible use of the same observations.
Critical weaknesses
1. The neural diagnostics do not sufficiently establish practical significance
- Teacher-defined target. The pretrained experiment replaces the real CIFAR-10 distribution with the teacher's Euler-100 distribution. This design appropriately guarantees a correct starting point, but does not directly represent learning from real image data.
- Narrow model coverage. The image evidence relies on one checkpoint of one architecture. Two confirmation streams do not establish prevalence across teachers, architectures, or model scales.
- Finite update budgets. Results after one and 200 updates demonstrate departure from the initial target, but do not establish discrepancy at a stable solution or convergence.
- Observable drift does not quantify practical degradation. The Inception projections provide valid witnesses of distributional change, but do not quantify overall discrepancy or demonstrate deterioration in image quality.
- The mechanism is not fully isolated. Sampled Adam updates, dropout, and numerical implementation all contribute to the experiment. The results do not establish population-gradient behavior in the neural model.
- Statistical significance differs from practical significance. A narrow confidence interval around a small effect does not establish that the effect materially affects applications.
- The scope is acknowledged, but significance remains limited. The abstract explicitly describes finite-update experiments, and the discussion acknowledges that convergence and prevalence remain open. Nevertheless, I find the neural evidence insufficient to support broader practical implications.
2. The connection between the theoretical constructions and broadly applicable repair remains limited
- Construction-specific geometry. The strongest exclusion result relies on deliberately aligning the residual and fitting direction. It remains unclear how representative this geometry is of ordinary neural model classes.
- A prescribed candidate is assumed. The reweighting characterization addresses selection of an already specified velocity field; it does not address how to identify a correct generator from data.
- Incomplete temporal comparison. Table 2 uses a shared model class, but the temporal follow-up expands CFM while retaining likelihood fitting in the original family. The comparison between objectives in the expanded family remains incomplete.
- Capacity has a substantial effect. Temporal expansion removes most of the base CFM KL. This makes it important to clarify the relative contributions of class restriction and objective selection to the empirical discrepancy.
- Repair exploits known structure. The structured experiments use coordinates and features known to both learners. This is a valid controlled setting, but does not establish effectiveness when the structure must be learned.
- Diagnostics rely on experimental ground truth. Teacher-derived KL and its decomposition are useful for post-hoc evaluation, but do not constitute a deployable diagnostic when the target distribution is unknown.
- Scalability is not established. The computational savings from two endpoint corrections are evaluated on two 2D tasks. The 100D structured repairs also exploit special structure; corresponding evidence for neural-scale repair is absent.
- Neural repair remains unconvincing. The exploratory companion improves some moment metrics but fails the structural-improvement criteria.
- The risk budget is empirical. The result under a 10% regression-risk increase does not provide a population-risk-cap guarantee.
- Implications for other methods require clarification. The paper does not establish how far the obstruction transfers to other paths, couplings, or objectives. This limits its applicability; it is not a requirement to prove failure across every generative framework.
Questions for authors
- Significance of the pretrained discrepancy: Beyond establishing distributional change, what do the current results show about the magnitude and practical consequences of the discrepancy? Is there existing evidence about its evolution under longer training or at a stable solution?
- Numerical and implementation controls: Which controls were performed specifically for the pretrained experiment? How large are the discrepancies attributable to precision, target generation/evaluation, and teacher-risk reconstruction relative to the reported effects?
- Novelty of the characterization: What is the essential new contribution relative to the metric-angle and constrained condition-number results discussed in Appendix A.4? Which part provides insight beyond the specific constructions?
- Capacity and objective: In the temporal follow-up, which conclusions can be attributed to objective selection when the methods are not fitted in the same expanded family? Which existing comparator helps separate capacity effects from objective effects?
- Deployability of repair: Which steps dominate computational cost as dimension and parameter count increase? Which assumptions about coordinates, features, or density tractability are essential?
- Scope of the weighting theory: How do the theorem's assumptions map onto practical time-weighting schemes? Is there a concrete setting in which the characterization yields a testable prediction?
6 comments
Response to Reviewer 8fpe (2/4)
Question 2. Controls for the pretrained experiment
Appendix F.2 reports the following. Teacher and student are evaluated on identical fresh draws, with 16,384 triples for risk and 10,000 common sources for the endpoint. Target, training, risk and endpoint streams use separate seeds, and a pilot fixed the update counts and feature directions before the two confirmation streams. Dropout is active only in training.
The teacher's per-triple risks were reconstructed from the recorded evaluator and seeds. Combining each reconstructed difference with its baseline reproduces the retained student arrays byte for byte in all eight combinations of stream and checkpoint, and the appendix notes that this does not prove identity of every intermediate tensor. Separate CPU code reproduced the summaries, intervals and bootstrap results from the stored vectors, without rerunning GPU generation or feature extraction.
Pairing estimates the uncertainty of the difference between teacher and student, conditional on the fitted models. The nominal 99% intervals of Table 3 resolve that difference under the stated interval model. Their half-widths are 15% and 20% of the one-update risk change and 7% and 11% of the one-update projection change. They do not bound systematic precision error or variation across training runs.
Four further controls for the pretrained experiment give the following exploratory results, with per-coordinate risk changes in units of as in Table 3. The witness intervals are nominal conditional 99% intervals for the fixed observable and are not additional primary confirmations.
- Zero learning rate, 1 update: risk change 0.000; witness change 0.00000 [0.00000, 0.00000].
- TF32 disabled, 1 update: risk change -1.906; witness change 0.07517 [0.06838, 0.08196].
- Dropout disabled, 1 update: risk change -3.339; witness change -0.04612 [-0.05265, -0.03960].
- Midpoint, 50 steps / 100 NFE, 200 updates: risk change -51.249; witness change 0.59968 [0.56772, 0.63164].
The zero-learning-rate control checks null behavior. Unchanged weights produce exactly zero paired risk and witness differences in this run. Departure also occurs with TF32 disabled. That run is not a float64 comparison and gives no systematic precision bound. Departure also occurs with dropout disabled, where the fixed observable moves in the opposite direction. That run also uses different training and evaluation seeds, so it does not isolate dropout as the cause of the sign reversal. The midpoint run defines its target with the midpoint teacher and uses the same deployment for the refit. It is a second matched construction of teacher and deployment and not an evaluation against the original Euler-100 target. The one-update and 200-update witness directions were fixed separately in the pilot.
Question 3. Novelty relative to Appendix A.4
The classical results cover two special cases. In the deterministic one-direction case, Theorem 2 reduces to the metric-angle threshold of the generalized Wielandt inequality. For finitely many information atoms, the problem is an instance of convex condition-number design and is solved numerically. Three things are new. The first is a closed form for measurable weights under an information restriction, , with attainment below the boundary and exclusion at without any bound on the condition number. The second is the role of information. More information can only lower the required contrast, and in the squeeze family the difference is one of order, against . The third is the use. The classical results do not ask whether regression selects a generator with the correct endpoint, which Theorem 1 answers for a whole correct set. Theorem 2 applies to any identifiable affine class satisfying its stated assumptions, with the reference target determined by the chosen coupling and path. The resulting lesson is that capacity to generate the target does not guarantee correct regression selection. The information available to the weight determines whether a prescribed correct field is selectable and at what contrast.
Response to Reviewer 8fpe (3/4)
Question 4. Capacity and objective
Your reading of Table 5 is correct. It cannot show that likelihood would beat CFM when both are optimized over the expanded family, and the paper says so. Three submitted comparisons do separate capacity from objective. Table 2 compares the two objectives in the same 6,725-parameter family, with 57.06 nats for CFM and 0.105 for likelihood. The original likelihood fit, at 0.104 nats, is a member of the expanded family because constant coefficients are allowed, while CFM selects a member at 6.49 nats with lower regression risk. Tables 6 and 7 fix the latent and contraction profiles of the temporal fits and vary only the drive, which carries 107,456 of the 107,600 parameters. There the regression-selected drive gives 6.515 nats, a drive fitted to endpoint samples gives 1.960 nats under the empirical 10% risk budget, and an oracle drive attains 0.227 nats, which shows substantial remaining approximation capacity in the same subfamily. Temporal capacity thus removes 88.6% of the CFM KL, and a change of criterion within the expanded family removes most of the remainder. This oracle uses teacher information, extreme coefficients and very high regression risk; it is a capacity diagnostic, not a practical repair. These are finite-data fits, and the membership statement is not a likelihood optimization over the expanded family. Our reply to Reviewer CEeF (item 4) discusses time weighting and finer temporal classes.
On theoretical reach, Appendix E already treats infinitely many correct generators and radial endpoints that need not be Gaussian. We have subsequently proved exclusion of every correct generator in a class allowing unrestricted spatial dependence and unrestricted time dependence on half the interval, with the halves linked by an explicit temporal constraint. This extension uses independent Gaussian endpoints with the same law and positive scalar input weights. The temporal constraint prevents representation of the reference velocity. We will present this extension and a complementary positive result, with their assumptions and proofs, in the consolidated follow-up.
Question 5. Deployability of the repairs
The corrections of Section 6.3 need target samples, a differentiable sampler and a feature class for the endpoint objective. Each correction computes the Jacobian of the feature residuals with respect to all parameters by forward sensitivities and then solves a dense least-squares problem. Both steps grow with the parameter count, and a neural-scale version would need reverse-mode differentiation through the sampler and matrix-free solves. We have not built or timed one, and Section 6.3 says that its timings do not establish neural-scale costs. The drive fits of Section 6.2 are linear least squares. They need the known coordinates, the known features and the triangular structure that makes the conditional mean linear in the drive. A tractable density is needed for the likelihood fits and for the KL evaluation, and the sample-only drive fits do not use it. The endpoint equation of Theorem 4 needs linear fields and identifies the endpoint only within the squeeze class.
Response to Reviewer 8fpe (1/4)
Thank you very much for the detailed review. We now report additional pretrained image metrics, longer-training observations and implementation controls. We also identify the existing comparisons that hold representation fixed and clarify the contribution beyond classical metric design. Convergence and broad neural applicability remain open.
Question 1. Magnitude and evolution of the pretrained discrepancy
The result is this. From a start that generates the target exactly, sampled CFM updates reach a model with lower fresh regression risk and a different generated law, in two independent streams. Regression risk therefore ranks an incorrect model ahead of a correct one. That is an objective mismatch at finite displacement. It does not settle convergence, prevalence or a material loss of image quality.
On magnitude, after one update the fresh risk falls by 3.3×10−5 and 4.0×10−5 per coordinate, and the fixed feature projection moves by 0.088 and 0.078. After 200 updates the risk falls by 5.8×10−4 and 5.5×10−4 (Table 3). The projection is a witness that the law changed and not a calibrated distance, and the directions for the two counts were fixed separately. Appendix F.2 states that the one-update changes are not evidence of a practically important loss of image quality.
On evolution, the submitted pretrained confirmations cover one and 200 updates; the separate exploratory run below extends to 2,000. Since submission, six 240,000-update runs in the synthetic neural setting retain lower held-out risk and detected endpoint discrepancies at all final raw and parameter-averaged checkpoints under fixed Heun-8 deployment. These are synthetic results, not longer CIFAR training, and establish neither stationarity nor convergence. We will provide the complete protocols and results in the consolidated follow-up.
Secondary image metrics use 10,000 generated images per model. FID uses CIFAR-10 training reference statistics. Teacher-reference FD is the same Frechet distance computed against teacher samples.
Each pair below lists confirmation 1, then confirmation 2.
The teacher-reference values for the teacher compare two independent teacher samples. They calibrate finite-sample variation and are neither a floor to subtract nor an equivalence threshold. All values are point estimates without confidence intervals. The raw 200-update models have worse recorded FID, and EMA strongly attenuates the change. The EMA starts at the teacher with decay 0.9999, and after 200 updates the teacher still has nominal weight 98.02% in the parameter average. This is not a mixture of generated distributions, and the short-horizon attenuation does not establish a general repair.
A separate exploratory run, outside the two confirmation streams, continued for 2,000 updates. Its real-data FID was 6.5337 for the teacher, 11.0391 after 200 updates, 10.6642 after 1,000 and 10.7398 after 2,000. The values at 2,000 updates follow.
Paired risk changes are per coordinate, in units of 10−5. Brackets give nominal conditional 99% intervals for the fixed-witness change.
The calibration between two teacher samples in this run is 4.9220. The EMA uses the same initialization and decay, and the teacher still has nominal weight 81.87% in the parameter average at 2,000 updates. The EMA witness is a secondary evaluation of the same fixed observable and not an additional primary confirmation. This run shows persistence over the measured horizon and supports no claim of stationarity or convergence.