acceptodds
Withdrawn from ICLR 2027

Stream3D: Sequential Multi-View 3D Generation via Evidential Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

View-conditioned 3D generators such as SAM3D, TRELLIS and Hunyuan3D produce high-quality object 3D representations from a single view, but real-world visual observation often arrives as long monocular streams. Each frame provides a partial observation; views accumulated across the stream provide complementary evidence for reconstruction. To address this problem, we propose Stream3D, the first training-free streaming mechanism that turns a frozen view-conditioned 3D generator into a streaming generator with constant cross-chunk evidential memory. Stream3D achieves this by maintaining a compact evidential memory, which selectively caches the most informative historical frames based on a proposed evidence score mechanism. As the stream progresses, the memory dynamically updates to retain a fixed number of informative frames, preventing the evidence cache from growing linearly with sequence length. This sustains generation quality over long sequences and keeps the underlying generator completely unchanged without retraining, architectural modifications, or auxiliary losses. Evaluated on both realistic and synthetic streaming benchmarks, Stream3D outperforms latent-transport baselines, including KV-cache reuse and flow-based feature editing, across both photometric and geometric metrics. More details can be found at: [Link](https://anonymous-submission-20.github.io/streaming3D2.github.io/)

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Generation, Streaming, Multi-Condition Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d8EwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stream3D, a training-free method that lets frozen single-view 3D generators such as SAM 3D and TRELLIS.2 take long monocular video streams as input. For each chunk, every new frame is probed with one denoising step. The generator's cross-attention maps are then used to compute a per-voxel-token evidence score, and each token keeps its top-D frames. Tokens vote for the top-K frames, which are used for multi-view generation, where per-view velocities are fused with token-wise evidence weights. The cache of frame indices has a fixed size, independent of stream length. Experiments on GSO and NAVI show improvements over single-view generators, multi-view fusion baselines, and other view-selection strategies.

Critical strengths

1.Using the generator's own cross-attention to decide which frame is useful for which part of the object is a simple and neat idea, and it requires no training.

2.The method is plug-and-play on top of existing 3D generators and gives clear improvements over single-view SAM 3D.

3.The ablations are fairly clean. Separating view selection from fusion, and comparing against visibility- and VLM-based selection, shows that both parts contribute.

Critical weaknesses
  1. The "streaming" framing is oversold. Each chunk regenerates the object from scratch with the selected frames, and the only thing carried over is a table of frame indices, while the frames themselves are still stored. In practice, this is attention-based view selection plus weighted multi-view fusion, repeated every chunk.

  2. Trained multi-view generators are missing from the comparison. Most baselines are training-free fusion methods, and EscherNet is quite outdated. The authors should compare with methods that are trained to take multiple views natively, such as ReconViaGen and Hunyuan3D-2mv. ReconViaGen already shows that trained multi-view conditioning is much better than TRELLIS multidiffusion.

  3. Generality and the long-stream benefit are not convincing. On TRELLIS.2, the gain over plain multidiffusion is small (CD 0.093 → 0.087), so most of the improvement seems specific to SAM3D. Geometry also stops improving after about 40 frames, so the long-stream setting is not really tested.

Questions for authors

1.Please compare against ReconViaGen and Hunyuan3D-2mv under the same frame budget. For Hunyuan3D-2mv, frames closest to its required viewpoints could be selected from the stream.

2.Why does Stream3D(TRELLIS.2) underperform TRELLIS+M.D. on GSO? Does the evidence score need to be retuned for each backbone?

3.What does the streaming formulation add beyond selecting the best K frames from the whole video at once? A comparison with offline selection over the full stream would help clarify this.

Reviewer_9ccpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Stream3D, a training-free approach that enables pretrained 3D generation models to process sequentially arriving views. Rather than retaining all observations, it estimates the relevance of each input view to individual 3D tokens through cross-attention and maintains a compact memory of useful visual information. A limited number of views are then retrieved and combined based on their estimated importance, allowing the model to progressively refine its 3D predictions while keeping the original generator unchanged.

Critical strengths

1. The overall design is straightforward and easy to follow.

By retaining relevant past observations separately for each 3D token, the approach allows different regions of an object to benefit from different viewpoints. Moreover, the bounded memory size keeps storage costs manageable as more images are processed.

2. The experimental evaluation provides reasonable support for the proposed components.

The paper benchmarks its view retrieval strategy against several alternatives, including visibility heuristics, VLM-guided selection, and random sampling. It also studies how reconstruction performance changes with different fusion strategies, cache sizes, and numbers of retrieved views. These comparisons help justify the main design decisions.

Critical weaknesses

1. The problem formulation and baseline selection are not sufficiently convincing.

Although the work is presented as streaming 3D generation, the actual setup seems much closer to conventional multi-view object reconstruction. The model takes multiple images as input and is evaluated against ground-truth 3D geometry, with reconstruction accuracy serving as the main objective. It is therefore not clear what makes this a distinct generation task. More importantly, the evaluation omits relevant multi-view reconstruction methods, which seem like natural baselines for this setting. Including these comparisons would provide a clearer picture of where the proposed approach stands relative to existing techniques.

2. The effectiveness of the attention-derived evidence metric lacks direct validation.

The method measures the usefulness of a view using a combination of attention weights over image patches and attention entropy. However, the connection between these quantities and the quality of geometric evidence is not well established. Strong or concentrated attention does not necessarily mean that the corresponding observation is accurate or helpful for reconstruction. This is particularly concerning for views containing occluded regions, repetitive patterns, or distracting visual content. The experiments demonstrate the overall performance of the method but do not isolate whether the proposed metric can actually distinguish useful observations from less useful ones. It would be valuable to investigate how the scores relate to reconstruction errors or the improvement contributed by individual views, rather than relying solely on the final reconstruction results.

3. The evaluation seems do not directly establish temporal consistency.

Most reported results evaluate only the final reconstruction, while the checkpoint analysis measures accuracy against ground truth at individual horizons. Neither directly measures whether successive outputs remain consistent. This matters because the method regenerates the object at each chunk with a potentially different conditioning set; reusing the same random seed does not by itself prevent changes in geometry or appearance. In addition, independently aligning each output to ground truth with Sim(3) and ICP removes global pose and scale differences from the evaluation. The paper should distinguish useful corrections introduced by new observations from unwanted changes in already well-observed regions, and compare this behavior with the streaming baselines.

4. The scalability claim is not established for the full pipeline.

The fixed memory bound applies only to the stored evidence scores and frame indices; historical images and their metadata are retrieved from external storage. More importantly, Sec. 3.3 and Algorithm 1 state that observed views are probed again on the current sparse structure to select the appearance-stage conditioning views. It is unclear whether this requires revisiting all previously observed frames at every chunk. If so, the cost of each update grows with the stream length despite the fixed-size evidence cache. If only a bounded candidate set is revisited, its construction and update rule need to be specified. Runtime measurements on a single 100-frame setting do not resolve this issue. Since long-stream processing is a central motivation, the paper should establish how the complete pipeline scales, including historical-view probing and data retrieval.

Questions for authors

See weaknesses.

Reviewer_YLS4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to adapt frozen, view-conditioned object-level 3D generators (SAM 3D, TRELLIS.2) to streaming input. Applying the generators independently to each frame can produce inconsistent reconstructions, while latent-state transport across chunks (KV-cache reuse, FlowEdit-style edits) can introduce misalignment and memory growth. Stream3D is a training-free procedure: a one-step cross-attention probe computes a mass–concentration evidence score for each incoming view and 3D query token; each token retains its top-D frames in a fixed-size Adaptive Evidential Memory; token votes select a top-K conditioning bundle for each chunk; and per-view flow velocities are fused using evidence-derived weights. The paper reports experiments on GSO-30 and NAVI subsets, with comparisons against single-view, multi-view-diffusion, and streaming baselines, and ablations on probe layers, memory depth D, bundle size K, weighting variants, seeds, pose conditioning, and runtime/memory accounting.

Critical strengths
  1. A relevant problem and a simple training-free design. The paper addresses the practical setting of adapting a frozen single-view 3D generator to streaming input with bounded cross-chunk state. Using the generator's own cross-attention maps to guide view retention avoids additional training and is compatible with the two backbones evaluated. The token-level memory allows different 3D regions to rely on different historical views, which provides a useful distinction from global keyframe-selection approaches. The bounded-cache and chunk-partition-invariance properties follow from the construction.

  2. The evaluation includes several useful controlled comparisons. The paper evaluates on two benchmarks and includes streaming baselines (KV-cache, FlowEdit) as well as selection baselines (random / visibility / VLM) that help separate view selection from fusion. In the controlled view-selection comparison, MV-SAM3D+Stream3D obtains CD 0.053, compared with 0.059 for VLM selection and 0.064 for random selection, while full Stream3D reaches 0.048. These comparisons provide evidence for the contributions of both evidential selection and evidence-weighted fusion. The seed and pose sensitivity analyses add further information about robustness.

Critical weaknesses
  1. The bounded memory creates an information bottleneck that is not analyzed as a function of stream length. Although newly arrived views are scored and can replace older entries, each token retains only its top-D candidates (D=5), and each generation pass consumes only K=8 views. Thus the method does not accumulate all available evidence; useful but lower-ranked views can be discarded, and the marginal benefit of additional frames may saturate. The long-horizon curves are consistent with this behavior—geometry nearly stabilizes after roughly 32–40 frames—but the paper does not separate memory saturation from viewpoint-coverage saturation or the generator's own performance ceiling. The D and K ablations report final-horizon performance rather than quality-versus-stream-length curves, memory turnover, or an all-frame/large-budget upper bound, leaving unclear how much information is lost as T grows.

  2. The comparison with reconstruction methods is too limited for the long-horizon setting. At roughly 100 object-centric views, the task closely resembles multi-view reconstruction rather than few-view generation. The experiments include StreamVGGT and EscherNet+NeuS on GSO, but other recent streaming/feed-forward reconstruction approaches, such as Spann3R, SLAM3R, CUT3R, and Stream3R, are not evaluated. In addition, there is no comparison with a strong per-scene optimization baseline, such as 3D Gaussian Splatting or an explicitly all-frame NeuS/NeRF pipeline, using the same observed frames and pose assumptions. Without these all-frame reconstruction references, it is difficult to determine whether bounded evidential memory retains a meaningful advantage over simply reconstructing from the accumulated observations.

  3. The real-video evaluation does not systematically test robustness to the preprocessing required in practice. The four DL3DV-10K real-video streams are useful qualitative examples with occlusion and irregular viewpoints, but SAM 3 is first used to segment and track each target, and the reported inputs are therefore already object-isolated. There are no quantitative results on these streams, and no controlled analysis of imperfect masks, tracking failures, or occlusion-induced depth/pose errors. As a result, the paper does not show whether the evidence score and memory remain reliable when upstream segmentation and geometry estimation are noisy, which is central to the claimed phone-video and robot-observation scenarios.

Questions for authors
  1. As stream length increases, how frequently do new views enter the top-D memory and how frequently does the selected top-K bundle change? Can you report memory turnover/coverage and quality-versus-T curves for several D and K values?

  2. Can you add stronger streaming/feed-forward reconstruction baselines, such as Spann3R, SLAM3R, CUT3R, Stream3R, and suitable VGGT-based variants, under matched input frames, pose/depth estimates, and evaluation horizons?

  3. Can you compare against an explicitly all-frame per-scene reconstruction baseline (e.g., 3D Gaussian Splatting, NeuS, or NeRF) at approximately 100 frames?

  4. For the four real DL3DV-10K streams, can you provide quantitative results where ground truth permits, and clarify the evaluation protocol rather than relying only on qualitative real-video examples? Can you report SAM 3 segmentation/tracking quality on these streams and test the full pipeline with imperfect or perturbed masks, failed tracks, and occlusion-corrupted depth/pose estimates? Are erroneous views rejected or retained by the memory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.