acceptodds
Withdrawn from ICLR 2027

SimpleProc: Fully Procedural Synthetic Data from Simple Rules for Multi-View Stereo

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating procedural synthetic data for multi-view stereo (MVS) usually requires writing complex rules to match the realism of curated datasets. We demonstrate that we can generate effective training data using SimpleProc: a new, fully procedural generator driven by a very small set of rules based on Non-Uniform Rational Basis Splines (NURBS), as well as simple displacement and texture patterns. At a modest scale of 8,000 images, our approach achieves superior results compared to manually curated images at the same scale sourced from games and real-world objects. When scaled to 352,000 images, our approach achieves similar, and in some benchmarks, even better results than the current state-of-the-art trained on over 692,000 manually curated images.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Procedural Generation, Synthetic Data, Multiview-Stereo

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RZonOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a synthetic data generation pipeline SimpleProc to generate effective training data for downstream MVS tasks such as depth estimation in fully procedural manner. The procedural generator is driven by lofting random NURBS profile curves ("starfish" or "reptile" style) along a 3D random-walk stem curve. It adds brick, wave, or noise displacement, applies textures made from thresholded procedural patterns combined with Boolean operations, and samples simple material parameters. Each scene contains up to 8 large and many small objects, an optional room box, optional ground scatter, area lights, and 8 cameras on a spherical band, rendered with Blender EEVEE. The experiments show that, with the same scale of data under limited budget, the MVS model trained by generated data is outperforming the one trained with curated data. When considering an unlimited budget scenario, model trained by SimpleProc is comparable to the curated baseline when ScanNet is excluded, with about half the images. However, the result also reveals that the model powered by the generation pipeline struggles to scale to sparse visibility indoor scene.

Critical strengths
  • At an equal number of training images, SimpleProc data outperforms the curated mixture on four of five benchmarks and MegaSynth [1] on all five, with aggregate margins well beyond the reported standard deviation.
  • The generator is fully procedural and needs no artist-made assets or image textures.

[1] CVPR 2025, Jiang, MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data.

Critical weaknesses

Pipeline design is not connected to the research questions. The introduction asks which rules MVS data needs and argues that realism is unnecessary (L073–075). However, Sec. 4 lists implementation details without explaining what the NURBS shapes and procedural textures contribute to MVS training. Several rules are instead justified by realism: clustered placement, the room box, and ground scatter (L301–306). I suggest that the authors define what MVS training data requires and map each rule to it, which would make the "small set of rules" claim testable.

Ablations could more clearly isolate the contribution of each rule. First, all shapes are lofted NURBS surfaces (L191), yet the only stated motivation is that they contain "both smooth and sharp features" (L227–228), and the only baseline is cubes, spheres, and cylinders (L672–675). Explaining why NURBS suits MVS training, and comparing it with a representation of similar complexity, such as the implicit surfaces in Shape Evolution [2], would show whether the gain comes from NURBS or from richer geometry. Second, the abstract names displacement and texture as core rules (L015–016), but these rules are ablated only in Appendix A; moving these results into the main text would let readers assess which rules matter.

Only one architecture is tested. All results use MVSAnywhere [3], which combines monocular and multi-view cues, so the claim that the generator "is by no means specialized to MVSAnywhere" (L151–152) is untested. At least one purely multi-view model without monocular depth cues (e.g., CasMVSNet [4] or MVSFormer++ [5]) trained on SimpleProc and on curated data is needed.

[2] CVPR 2018, Yang, Shape from Shading through Shape Evolution.

[3] CVPR 2025, Izquierdo, MVSAnywhere: Zero-Shot Multi-View Stereo.

[4] CVPR 2020, Gu, Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching.

[5] ICLR 2024, Cao, MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo.

Questions for authors
  • How does performance change as each rule is added to a minimal NURBS generator? Since the best configuration may shift with dataset size (L656–657), was it re-validated at 352K?
  • How is MVSAnywhere [3] initialized? If Depth Anything V2 [6] weights are used, does SimpleProc still match curated data with a randomly initialized encoder?
  • Why does each image still take 262 s (L315) when the rasterization-based EEVEE engine is chosen for efficiency (L308), and what is the rendering resolution? How much performance is lost with cheaper rendering, such as coarser NURBS tessellation or simpler shading? The reported compute also seems inconsistent: 352,000 images x 262 s ≈ 25,600 GPU-hours, which already equals the stated total (L364–365), yet that total is said to include variant configurations during generator development. What is the rendering cost of the final 352K dataset alone, and how much compute went into tuning the generator on the validation split?

[6] NeurIPS 2024, Yang, Depth Anything V2.

Reviewer_2EywOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents SimpleProc, a fully procedural synthetic data generator for training multi-view stereo (MVS) models. The main idea is that effective MVS training data does not necessarily require realistic 3D objects or scenes. Instead, the authors generate random 3D shapes through NURBS-based lofting, apply geometric displacements and procedural textures, and arrange these objects into scenes with randomized cameras and lighting. The entire pipeline relies on a relatively small set of procedural rules without using manually created 3D assets or texture libraries.

Critical strengths

The paper addresses an important and practical problem. Generating large-scale 3D training data is expensive and often requires manually created assets or curated datasets. Exploring whether simple procedural rules can generate effective training data for MVS is a worthwhile direction. If successful, this could reduce the dependence on manually created 3D assets and make training data easier to scale.

The experimental results demonstrate the effectiveness of the generated data. With only 8K training images, SimpleProc achieves better average performance than the curated-data baseline and MegaSynth. At a larger scale, the results are reasonably competitive with MVSAnywhere trained on substantially more curated images, although there are still performance gaps on some benchmarks. The authors also provide ablation studies on different procedural design choices, which help explain what contributes to the performance.

Critical weaknesses
  1. Limited novelty and missing important related work. The central idea of this paper is very similar to LRM-Zero (NeurIPS 2024), which is surprisingly not cited or discussed. LRM-Zero already demonstrated that 3D reconstruction models can be trained entirely on procedurally synthesized objects without realistic semantic content, while achieving competitive reconstruction quality on real-world data. Although SimpleProc focuses on MVS instead of sparse-view 3D reconstruction, and uses NURBS-based geometry instead of combinations of primitives, I don't think simply applying the same idea to a different reconstruction task establishes sufficient novelty. The main hypothesis that realistic 3D assets are not necessary for learning 3D reconstruction has already been explored and validated by LRM-Zero. The authors should clearly distinguish their contribution from this prior work.

  2. The benefits of the proposed procedural design are not sufficiently demonstrated. Compared with LRM-Zero, the main technical differences are the NURBS-based shape generation and fully procedural textures. However, the paper does not demonstrate whether these design choices offer meaningful advantages over existing procedural generation approaches. For example, the authors compare NURBS shapes with basic primitives, but not with more complex primitive-based geometry using Boolean operations and geometric augmentations, as in Zeroverse. Similarly, the paper does not investigate whether fully procedural textures outperform textures sampled from an existing library. I would expect more direct comparisons to justify these design choices.

Furthermore, while the results with 8K images are promising, the larger-scale results are less convincing. With 352K images, SimpleProc still performs worse than the curated-data baseline on the overall five-benchmark average (79.99 vs. 81.80 in threshold accuracy). The authors emphasize the slightly better performance after excluding ScanNet, but I don't think excluding a benchmark where the method performs poorly is sufficient to support a general performance advantage. The evaluation is also limited to one MVS architecture, so it remains unclear how well the proposed data generalizes to other reconstruction models.

Overall, I don't see SimpleProc providing much additional benefit beyond demonstrating the same idea on another reconstruction task. To me, a more significant contribution would be to show that procedural training data can address challenging cases that previous methods struggle with, such as glass, water, transparent or highly reflective materials, hair, and other complex thin structures. These would be more meaningful extensions of the idea rather than simply changing the procedural shape generation rules.

I think the paper addresses a useful problem, but the main idea is not sufficiently novel, and the experiments do not clearly establish what new capabilities or advantages SimpleProc provides over previous procedural data generation methods.

Questions for authors

None.

Reviewer_e7LxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SimpleProc, a Blender-based procedural generator that produces MVS training data from lofted NURBS shapes, noise-based displacement and textures, and randomized object, light, and camera placement. The authors train MVSAnywhere only on this data and evaluate on the five real-world RMVD datasets, each split into a validation half (used to design the generator) and a test half. At a fixed budget of 8,000 images, SimpleProc outperforms equally sized subsets of the curated eight-dataset mixture and of MegaSynth on average. At 352,000 images, it approaches the released MVSAnywhere checkpoint on several benchmarks but is substantially worse on ScanNet and lower on both five-dataset averages.

Critical strengths
  1. A useful data-centric question with a compact, reproducible recipe. Code and sample data are provided.
  2. A convincing fixed-budget result. With architecture, budget, and schedule held fixed (Table 1), SimpleProc improves average τ from 73.20 to 77.13 and average rel from 3.52 to 3.27, well beyond the run-to-run variation.
  3. Real-world, multi-domain evaluation. Newly trained models are averaged over three runs; the released full-scale baseline is the stated exception.
  4. Informative geometry and camera ablations, notably lofted NURBS shapes versus primitives (71.81 → 76.77 validation τ).
Critical weaknesses

W1. Parity and transfer beyond the development benchmarks are not established

Generator settings are selected on the validation halves of all five RMVD datasets. No MVS training on target-domain images is reported, but this is target-domain feedback during design, which limits the evidence for transfer to unseen domains. The selection is also asymmetric in Table 1: SimpleProc and the MegaSynth view threshold are tuned on the validation halves, while the curated subset keeps its original dataset proportions. The split unit is unspecified; an image-level split could place views of the same scenes in both halves.

The full-scale results (Table 2) show a tradeoff rather than parity:

Evaluation subsetMetricCurated baselineSimpleProc
Test, all five datasetsrel ↓ / τ ↑2.66 / 81.802.92 / 79.99
Test, excluding ScanNetrel ↓ / τ ↑2.39 / 86.252.38 / 86.62
Both splits, excluding ScanNetrel ↓ / τ ↑2.49 / 85.752.58 / 85.14

The authors designate the test split as primary (ll. 368–369). On it, the ex-ScanNet advantage is small (+0.37 τ) and reverses on the combined splits. Assuming equal halves, the implied validation-half ex-ScanNet τ is about 83.7 for SimpleProc versus 85.3 for the baseline. SimpleProc is therefore relatively weaker on the half it was tuned on, which argues against overfitting in the usual sense, but the swing between halves (≈2 τ points) exceeds the test advantage, and the baseline has no variance estimate. The ex-ScanNet result should not be read as superiority, and "confirming" the absence of validation overfitting is too strong. SimpleProc trails by 10.57 τ points on ScanNet; the proposed explanations are plausible but untested, and the paper does not establish how annotation errors affect the relative ranking. The five-dataset average should be the headline, and "state-of-the-art performance" (contribution 1) and "as effective as curated data" (Fig. 1) should be qualified.

Would help: the split unit, direct validation-half results for Table 2, and at least one real-world benchmark never used during generator development, as Yan et al. [2] do with held-out sets for stereo.

W2. Initialization is unspecified

MVSAnywhere initializes its reference-image encoder from Depth Anything V2 and its cue combiner from DINOv2 [1]. The Depth Anything V2 teacher was trained on synthetic data that includes four of the eight curated datasets in the baseline mixture (Hypersim, TartanAir, BlendedMVS, VKITTI 2) [5]. The manuscript does not state which initializations SimpleProc retains. If it retains them, the result shows that procedural data can replace curated data in the MVS training stage, conditional on priors learned partly from curated data, rather than that procedural data suffices on its own. That would still be useful, but the scope should be stated. Would help: the initialization of each component and, ideally, both data sources compared under a common initialization without depth pretraining.

W3. The large-scale comparison is not matched

The full-scale baseline is the released checkpoint, trained for 145k steps on two GPUs (6–8 images per GPU, two-batch gradient accumulation) [1], while SimpleProc receives three new runs of 1.6M steps at batch size 6. Table 2 therefore does not isolate the data effect. The composition of the 352,000 images is also unclear: 352,000 × 262 s ≈ 25,600 GPU-hours, the figure given "including variant configurations during generator development" (ll. 363–366). Please state whether the Table 2 models are trained on all of these images and, if so, what share comes from non-final configurations. Would help: retraining the baseline under an identical protocol, or a compute-matched comparison. Because the 8k and 352k experiments also use different schedules, a data-scaling curve under a fixed schedule would be a useful optional extension.

W4. Cross-architecture generality is untested

The generator is designed, tuned, and evaluated only with MVSAnywhere. The paper motivates transfer by architectural similarity (ll. 154–163), but cross-architecture transfer, and hence the statement that the generator is "by no means specialized to MVSAnywhere," is not empirically demonstrated. Evaluating the unchanged generator with a second MVS architecture would most strengthen the paper. Since the paper asks whether many rules are needed, a comparison with a richer procedural generator (e.g., the Infinigen-based generator of [2]) at matched image count, and separately at matched generation compute, would also be informative. For the fixed-budget comparison, please report the curated subset's per-dataset scene and image allocation, and how often the angular-threshold MegaSynth tuples are actually covisible.

W5. Ablation conclusions are stronger than the evidence

Defaults differ between ablation blocks (acknowledged, ll. 653–655). Each block is internally controlled, but interactions and attribution within the final configuration remain unclear. Selecting the best validation mean is normal practice, but several choices are described as beneficial on differences not clearly resolved by the reported variability (e.g., hybrid versus uniform small-object placement, 76.23 ± 0.28 vs 76.13 ± 0.16). Some text also contradicts the tables: the room box is said to improve all benchmarks, but DTU decreases (88.94 → 88.83); more lights are said to improve performance, but 5–40 lights scores below 5–20 (75.60 vs 76.01). A leave-one-component-out study from the final configuration would show more reliably which choices matter.

Suggestion: the appearance and lighting ablations cover uniform color, noise textures, and noise textures with Boolean combinations, plus the number of overhead area lights. Given the ScanNet gap and the stated limitation that sensor noise and lens effects are not modeled (ll. 472–474), testing camera-effect augmentation (noise, blur, exposure) and light type and color would be informative follow-ups; Mayer et al. found that simulating real camera flaws improves disparity and optical-flow performance on images from such cameras [4]. Diagnostics comparing the procedural data, the curated mixture, and the evaluation sets (depth and baseline distributions, covisibility, textureless-region coverage), with error stratified by region type, could also help investigate the ScanNet gap and indicate which generator choices transfer.

W6. Related work needs more precise positioning

Several relevant precedents are not discussed. Yan, Raistrick, and Deng [2] vary the parameters of an Infinigen-based procedural generator for zero-shot stereo, compare against curated mixtures, and find that realistic backgrounds help. LRM-Zero [3] trains sparse-view reconstruction solely on procedurally composed primitives (with sampled, non-procedural textures), with results competitive with, though still below, Objaverse training. Mayer et al. [4] found that photorealism is not necessary for effective disparity and optical-flow training. AutoFlow [7] learns the hyperparameters of a procedural optical-flow generator by optimizing performance on a target dataset, which bears directly on benchmark-informed generator design (W1). MegaSynth reports synthetic-only training [6], so describing it only as complementary to real data is incomplete, although its synthetic-only model trails real-data training (21.50 vs 23.89 PSNR, Table 4 of [6]).

Minor comments

  • Specify rendering resolution, scene scale and depth ranges, light intensities and colors, and the optimizer, learning rate, and augmentation used for training (initialization: see W2). State whether the full dataset will be released.
  • Generation cost (262 s per image; ~580 GPU-hours for 8,000 images) sits uneasily with "minimal cost" (l. 59); distinguish reduced human effort from computational efficiency.
  • Section 2.2 says BlendedMVS-only insufficiency was "proved" by MVSAnywhere, which does not appear to report a BlendedMVS-only training run of its model (the nearest evidence is an MVSFormer++ baseline trained on DTU+BlendedMVG, a different architecture). Cite the specific experiment or soften.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.