acceptodds
Withdrawn from ICLR 2027

From Feed-Forward to Flow: Unifying Reconstruction and Generation Is Easier Than You Think

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstruct where the images provide evidence, and generate where they do not: recent success of spatial world models such as Atlas (World Labs Team, 2026) highlights the value of unifying reconstruction and generation in one model. Yet the two have long lived in separate paradigms with distinctive failure modes: feed-forward reconstruction averages ambiguity into blur, while conditional generation invents plausible but scene-inconsistent detail. In this work, we present a unified flow-based formulation for reconstruction and generation, where a shared clean-target predictor performs direct reconstruction at its single-step endpoint and unfolds conditional generation through multi-step flow. A controlled toy study reveals the mechanism: with a single step, the predictor collapses to the conditional mean just like feed-forward methods, favoring consistency over diversity. With multi-step inference, the fidelity of generated details grows with context richness: closer observations reduce ambiguity and yield better-matched details. We further instantiate the formulation in appearance and geometry 3D tasks. JiT-LVSM improves perceptual and distributional quality in novel view synthesis, while JUSt3R retains competitive single-step geometry prediction with additional multi-step inference capabilities that reduces veil and flying-pixel artifacts, producing cleaner surface structure with greater test-time compute. Together, they show that reconstruction and generation can share both a formulation and a backbone, with their behavior governed by denoising configuration—making unification surprisingly simple.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
feed forward models, 3d reconstruction, novel view synthesis, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EU79OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores a unified flow-based approach that allows a single model to perform reconstruction with one inference step and conditional generation through multiple steps. The authors investigate this idea using toy examples and two 3D tasks, namely novel view synthesis and geometry prediction. The experiments suggest that multi-step inference can improve perceptual image quality and certain geometric boundary metrics, although these gains sometimes come at the cost of pointwise prediction accuracy.

Critical strengths

1. The paper addresses a relevant problem.

Understanding how reconstruction and generation can complement each other is an interesting direction, particularly given the practical tension between making accurate predictions and producing visually plausible, scene-consistent outputs.

2. The evaluation spans different tasks and output spaces.

The toy experiments help illustrate the proposed mechanism in a controlled setting, while the novel view synthesis and geometry experiments provide a broader picture of how the approach behaves on realistic 3D prediction tasks.

3. The evaluation goes beyond standard reconstruction accuracy.

In addition to PSNR, the authors report perceptual metrics such as LPIPS and FID, as well as measures of geometric boundary quality and artifacts. This makes it easier to understand where iterative inference helps and what is potentially lost in terms of reconstruction accuracy.

Critical weaknesses

1. Limited novelty of the theoretical contribution.

The paper builds on the observation that target-independent noise provides no additional information about the target once the context is given. As a result, the optimal one-step predictor is the same as in standard regression, reducing to the conditional mean under squared error. With multiple steps, state-dependent predictions can instead produce conditional samples. These observations are fairly natural from the perspective of statistical decision theory and conditional flow models. Although the paper cites relevant prior work, it remains unclear what the new technical insight is beyond existing research on conditional generation, diffusion-based reconstruction, and the perception–distortion tradeoff. Applying these principles to two 3D tasks is interesting, but does not by itself seem sufficient to support the broader claims of a unified framework made in the title and abstract.

2. The theoretical guarantees do not fully carry over to the actual models.

The idealized flow transport result in the appendix relies on an optimal conditional mean field under squared error, suitable ODE assumptions, and exact integration. In practice, both the image and geometry experiments use finite-capacity networks, a limited number of sampling steps, and additional training losses. The authors also acknowledge that, under general task-specific losses, the optimal clean-target prediction need not coincide with the optimal squared-error flow field. The geometry sampler further introduces a normalization projection. Therefore, while the theory offers a plausible explanation for why iterative generation might help, it does not establish that the improvements of JUSt3R and JiT-LVSM arise from the same underlying mechanism.

3. The image experiments do not isolate the effect of iterative sampling.

LVSM is trained as a one-step predictor, whereas JiT-LVSM uses flow matching together with additional image-space losses. The appendix also describes configurations involving pixel MSE and perceptual losses. Thus, despite sharing the same backbone, the two models differ in more than just the number of inference steps. The results are also somewhat mixed. JiT-LVSM has roughly 1.7–1.8 dB lower PSNR under sparse context, while its PSNR advantage under richer context is only about 0.6 dB. Improvements in LPIPS and FID are encouraging, but they do not necessarily imply better scene fidelity. A sharper or more realistic-looking image may still contain plausible details that are inconsistent with the actual scene, as the authors themselves acknowledge in the sparse-context setting. A more controlled comparison is needed to determine how much of the improvement comes specifically from iterative sampling rather than the changed training objective.

4. The geometry experiments have a similar attribution issue.

The paper presents the one-step endpoint of JUSt3R as a counterpart to DUSt3R's direct prediction, but their training objectives are not exactly matched. As noted in the appendix, endpoint equivalence requires matching the full loss, whereas the complete flow training objective is not simply equivalent to DUSt3R's geometry regression loss. The authors acknowledge this limitation, but its implications are not sufficiently addressed in the experimental analysis. Consequently, comparable or better one-step performance does not necessarily demonstrate that flow training preserves the reconstruction ability of direct regression. Likewise, the multi-step gains cannot be cleanly attributed to iterative generation without separating the effects of the training objective and the sampling procedure.

Questions for authors

See weaknesses.

Reviewer_GBc7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This submission argues that feed-forward reconstruction and flow-based conditional generation are tightly related, and proposes a flow-based formulation that unifies both. A core insight is that feed-forward reconstruction and one-step flow inference that starts from pure noise have the same optimal action (since the noise is providing no signal), which is to average across all possible solutions when trained with MSE loss (this is why feed-forward estimators produce blurry results in under-specified regions). The main reason why multi-step flow gives sharper results is that although the model is still predicting an average at each step, the set of possible answers shrinks at each step as the model's estimate evolves from pure noise (and so over many steps the model gives a sharp albeit potentially incorrect solution). As the paper states, "a sequence of conditional means need not produce the context-only mean" (lines 135-136). The paper shows a toy experiment that illustrates these points (ie: one-step flow inference behaves near-identically to a feed-forward reconstructor in matching the analytic average, and the same flow model checkpoint given 15 steps commits to a single mode).

The practical implication of this observation, beyond the fact that a single model can act either as a direct predictor or generator, is that, according to the authors, multi-step denoising can be viewed as a form of test-time computation. To test the formulation, the authors implement flow-based variants of LVSM and DUSt3R (JiT-LVSM and JUSt3R) that keep the original backbone and add noisy target and flow-time conditioning. In sparse-context settings (where input views don't fully cover the target view), 10-step JiT-LVSM gives better LPIPS and FID at the cost of almost 2dB PSNR, and in rich-context settings that tradeoff vanishes (0.6dB higher PSNR while remaining best across LPIPS and FID). In the case of DUSt3R/JUSt3R, pointmap prediction is generally a less ambiguous problem where most ambiguity happens near depth discontinuities. One-step JUSt3R generally matches or outperforms the authors' matched reimplementation of DUSt3R (trained on the same data as JUSt3R) while 10-step JUSt3R reduces floating points near depth discontinuities and generally provides cleaner edges at a small cost to AbsRel and δ1.25.

Critical strengths

I agree that feed-forward reconstruction and generation are closely related, and the idea of training model checkpoints that can do both makes sense to me. I like the toy experiment the paper uses to illustrate the relation between feed-forward reconstruction and flow matching and Figure 1 in particular - it does a good job of capturing the core observations of the paper. In terms of the experiments, I found the example of how multi-step inference can help reduce floating points near depth discontinuities for point map prediction to be neat, albeit suggested by prior work such as PointDiT.

Critical weaknesses

Although I agree with the premise that feed-forward reconstruction and generation are closely related, I am unconvinced that this submission makes contributions significant enough to merit acceptance. In terms of insights, most of what the paper argues has been explored in prior work, some of which has not been cited. InDI [1] (not cited) trains a network to iteratively improve image quality in small steps to address the same "regression to the mean" problem this paper describes - in their words they similarly argue "a small-step restoration largely avoids the regression-to-the-mean effect because the set of plausible ‘slightly-less-bad’ images is relatively small," which is essentially the mechanism this submission relies on in Sec 2.2 ("a sequence of conditional means need not produce the context-only mean," lines 135-136). They also show a similar toy example, and observe that the number of steps controls the distortion vs perception tradeoff. Garcia et al. [2] (not cited) point out that a single step from pure noise predicts the conditional mean - a blurry image for text-to-image tasks but already a well-defined estimate for image-conditioned depth prediction - which is essentially the posterior width argument applied to geometry that this submission makes. Pixel-Perfect Depth [3] (not cited) also claims that flying pixels are due to feed-forward reconstruction averaging foreground and background depth at discontinuities and removes them with diffusion transformers. PointDiT [4] (cited tangentially as the source of the BF1 metric) presents findings that are very similar to JUSt3R (one-step is competitive, more steps sharpen boundaries). PointDiT is debatably concurrent work under ICLR's policy but the others are not, and it is critical for the authors to position their submission relative to these works and what is new for me to reconsider my assessment.

Regarding the rest of the paper's contributions, what remains is applying the flow matching formulation to LVSM and DUSt3R which feels incremental. Describing a more general and detailed recipe and analysis for how to train a single model that is both a strong feed-forward reconstructor and generator would make for a significant contribution - the closest the paper comes to this is with the JUSt3R experiments, although many important details are missing, such as the training convergence implications of training a unified model vs a dedicated feedforward reconstructor, inference time, the timestep distribution used to train the model, its guidance scale, etc. The JiT-LVSM experiments are even more sparse (no analysis of single-step performance).

On a relatively minor note, the paper contains sequences of prose that seem written by an LLM that make things harder to understand than they need to be, such as "Posterior width sets the price of commitment," in Sec 3.4 and "Context constrains that detail; distortion alone does not tell us whether it is useful," in the conclusion. More examples:

  • "intermediate states propagate the model's evolving hypothesis" and "compose that hypothesis" (lines 137-138)
  • "More steps commit to the data manifold" (line 202)
  • "depth" used to describe both for scene depth and the number of denoising steps, e.g. "Context and depth play complementary roles" (line 147) and "the choice of depth per input or spatial region" (line 481)

The authors acknowledge that AI was used to improve the readability of the manuscript which is totally fine - I'd just advise them to prompt it to generate wording that is as clear and concise as possible without jargon.

References:

  • [1] Delbracio & Milanfar. Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration. TMLR, 2023.
  • [2] Martin Garcia et al. Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think. WACV, 2025.
  • [3] Xu et al. Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers. NeurIPS, 2025.
  • [4] Xu et al. PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation. ICML, 2026.
  • [5] Yin et al. One-step Diffusion with Distribution Matching Distillation. CVPR, 2024
Questions for authors
  • Could you more clearly position this submission relative to prior work, and articulate its novelty and contributions?
  • Could you provide an analysis of the one-step JiT-LVSM results in particular, and a more comprehensive sweep of both methods in general?
  • As an example of how this would be helpful - the authors argue that AbsRel/δ1.25 cost of 10-step JUSt3R comes from the perception-distortion trade-off, but it could also be caused from error accumulation across steps (as Garcia et al. claim for Marigold). PointDiT does not see this tradeoff over 1-4 steps, so a sweep would help disentangle the two.
  • Could you provide details on training convergence and inference speed of the unified methods?
  • Could you provide the timestep distribution, solver, and guidance scale used for JUSt3R?
  • How do your observations and the trained checkpoints compare to a distilled one-step/few-step generator such DMD?
Reviewer_xDmPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method that treats feed-forward reconstruction as the one-step case of a flow model. As a result, a single checkpoint can perform reconstruction in one step and generation in multiple steps. The approach is evaluated on two tasks, novel view synthesis and pointmap prediction and achieves results competitive with LVSM and DUSt3R.

Critical strengths
  1. The formulation is simple and well motivated. Framing feed-forward reconstruction as the one-step case of flow matching explains why regression tends to blur ambiguous regions, and the toy experiment with an analytic posterior illustrates this point convincingly.
  2. The conversion is minimal and general. Both LVSM and DUSt3R retain their original architectures and only add a noisy target input and time conditioning. Consequently, a single checkpoint can trade pixel accuracy for sharpness at test time, making the idea easy to adopt in other feed-forward models.
Critical weaknesses
  1. The improvements are marginal and mostly trade-offs. JiT-LVSM improves perceptual metrics but lowers PSNR under sparse context and is nearly tied with LVSM under rich context, and multi-step JUSt3R gives cleaner structure but a worse depth error than the retrained DUSt3R. It is unclear whether the method improves over feed-forward models or only moves along the distortion–perception trade-off.
  2. The unification claim in the title is not evaluated. The model uses one global step count for the whole output, with no reconstruction where observations exist and generation where they do not, and it is not compared with generative models that also unify both, such as Gen3R or Stable Virtual Camera.
Questions for authors
  1. What is the one-step result of JiT-LVSM, and how does an LVSM trained with a noise target input perform? This would show how much of the gain comes from multi-step generation rather than training.
  2. What is the variance of the results over training seeds and noise samples, especially for the structural metrics of JUSt3R?
  3. Is there a step count, or a schedule, at which the method improves both pixel and perceptual metrics over the feed-forward baselines, rather than trading one for the other?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.