acceptodds
Withdrawn from ICLR 2027

Learning to Code the World: Harness Evolution for Compositional 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing editable and compositional 3D scenes from a real image requires jointly reasoning about geometry, object structure, and spatial relationships. We introduce SceneEvolve, a framework that represents 3D scenes as executable code and improves reconstruction by evolving the harness around a small open-source agent. SceneEvolve combines SceneAct, a compositional semantic action interface built on parameterized primitives, hierarchical composition, and semantic spatial relations, with SceneHarness-RSI, which leverages a stronger teacher agent to iteratively evolve the harness using feedback from development-set reconstructions. Throughout harness evolution, the model parameters, action semantics, and execution backend remain fixed. The resulting harness provides a reusable reconstruction procedure that enables the small open-source model to reconstruct unseen scenes without parameter updates or teacher assistance. Across two in-domain datasets and one out-of-domain dataset, SceneEvolve with the comparatively compact Qwen3.8-27B model improves all four evaluated reconstruction metrics while reducing mean inference time by 47.0% relative to the base model. More notably, despite relying on a substantially smaller model, SceneEvolve achieves 32.3% higher reconstruction quality than the large proprietary GPT-5.6-Sol model, while reducing mean equivalent deployment cost by 59.8%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Compositional 3D Reconstruction, Agentic Coding, Harness Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2AkdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper notes the recent breakthrough in 3D scene creation brought about by frontier models like GPT-6 Astra and aimed to bring similar capabilities to smaller open source models that cost a lot less (e.g. Qwen)

The two proposed contributions are 1) prompt-based definition of interfaces to help the model interact with Blender 2) a method to evolve the harness by using a stronger frontier model like GPT-6 Astra.

Critical strengths

The ambition and research goal of the paper is quite interesting and could be impactful if it is successful. If we can use a frontier model to create a generalization harness improvement for the task of 3D reconstruction based on image, the saving in compute cost would be quite substantial.

To this end, the paper does show a consistent improvement on Qwen performance. And show that Qwen + their method out performs GPT-6 Sol on image based metrics while being cheaper.

Critical weaknesses

Even if harness evolution is possible to improve the model quality (which other paper has already shown), this improvement is not useful unless the improved model reached sufficient quality in its output so that the 3D reconstruction is useful.

This paper does not define any downstream task to evaluate the quality of the 3D reconstruction. This means there is no benchmark on the applicability of the method quantitatively. Qualitatively, the results are also not convincing. Although it beats Sol in image-based metrics, the result are quite far away from a realistic reconstruction of the input image. Although Astra cost 4-5 times more, the quality gap is so large that I would always prefer to spend this money on Astra as both Sol and Qwen outputs does not look usable.

In summary, as a method paper, nothing in the method is very novel. it is the first application of these concepts to the 3D reconstruction task. As an application paper, the results are not impressive. The created 3D reconstruction is not sufficiently good so that the cost saving becomes a factor. If it can achieve Astra-level reconstruction at a fraction of the cost, then the method has meaningful applications.

Questions for authors

Suggestion for future work is to apply this method to stronger baseline like Sol. Maybe with a strong enough harness, Sol can rival Astra in quality while still being cheaper. Qwen is too weak, and it is not clear that improvements shown is transfererable to other stronger model. (or generally applicable when stronger models is released)

Reviewer_A7rsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SceneEvolve proposes a framework for single-image compositional 3D reconstruction using a frozen coding agent. It combines SceneAct, a semantic interface with parameterized primitives, hierarchical composition, and spatial relations, with SceneHarness-RSI, which uses a stronger teacher to evolve a reusable reconstruction harness. Experiments on ScanNet, CA-1M, and ARKitScenes report improved reconstruction metrics and lower inference time relative to the base Qwen model. Component ablations support both design choices.

Critical strengths

1.The improvements over the frozen base model are consistent. Table 1 reports improvements across all four reconstruction metrics on all three datasets, with mean DINOv2 similarity increasing from approximately 0.198 to 0.392. Transfer to ARKitScenes without target-dataset feedback strengthens the evidence beyond development-set optimization.

2.The component ablation supports both design choices. Table 2 shows that removing either SceneAct or SceneHarness-RSI reduces reconstruction performance. With SceneAct fixed, harness evolution also reduces inference time substantially, supporting the practical value of optimizing the reusable workflow.

Critical weaknesses

1.The additional contribution of SceneHarness-RSI is insufficiently characterized. The provided manuscript lacks initial and final harnesses or representative accepted modifications. While Table 2 supports the effectiveness of evolution, it does not explain which procedural changes produce the gains, limiting assessment of the contribution beyond existing reflective optimization and independent reproducibility.

2.The structural and editing claims are insufficiently validated. The image-based metrics and Figure 6 support improved visual matching, but do not directly verify 3D consistency after editing. In particular, §3.2 describes automatic updates of dependent elements, yet no editing example or targeted validation demonstrates this behavior.

Questions for authors

Q1, What concretely changes between the initial and final harnesses? Please provide representative accepted modifications, explain which changes account for the principal gains, and clarify how the fixed SceneAct semantics are preserved.

Q2, Under matched teacher and evaluation budgets, how does SceneHarness-RSI compare with a single teacher-guided harness revision or an established reflective prompt optimizer with SceneAct fixed? Is there existing evidence that the broader harness search provides benefits beyond these alternatives?

Q3, How are camera intrinsics, extrinsics, and image alignment handled, and are the same inputs and procedures used across methods? Is there existing geometric, novel-view, or editing evidence supporting the structural and editability claims beyond reference-view image similarity?

Reviewer_ek77OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SceneEvolve, a framework for converting images into editable and compositional 3D scenes in Blender. SceneEvolve is composed of two components: SceneAct and SceneHarness-RSI. SceneAct provides an interface for semantic construction and composition of simple scene primitives. SceneHarness-RSI is a self-improvement loop where a student agent repeatedly generates scenes using SceneAct, and a teacher modifies the harness to improve runtime and cost. Overall, SceneEvolve is able to significantly improve the reconstruction quality and cost of Qwen3.8 compared to existing models.

Critical strengths
  • The quantitative results are quite strong. The authors demonstrate a significant improvement in reconstruction fidelity and also reduction in cost/number of tokens required to reconstruct the scene.

  • The authors ablate their changes and demonstrate that both SceneAct and SceneHarness-RSI improve performance.

Critical weaknesses
  • The SceneAct portion of the work is not very novel. Various similar DSLs for inverse graphics, reconstruction, and 3D scene generation have been proposed that compose simple primitives with semantic functions (Aguina-Kang et al., 2024, LayoutVLM, Infinigen Indoors).

  • The SceneHarness-RSI portion of the work is more novel, but is extremely opaque given the paper’s current contents. The authors do not report or give any examples of changes that the agent actually proposes to improve the harness. As such, it is hard to get a sense of why performance improves/why costs go down. Is the agent stitching together some sort of compositional abstractions for common objects? Without seeing the exact proposal, it's not clear whether some sort of implicit shortcutting or reward hacking is going on.

  • The teacher used in the iteration is GPT-6 Astra. Therefore, the framing of the work as self-improving seems quite misleading. In fact, based on my understanding, this is something closer to distillation of Astra’s knowledge of 3D reconstruction into a set of textual skills.

  • Costs of evolution process are not reported - how many new scenes would need to be generated before evolution is “worth it” from a cost perspective.

Questions for authors
  • Can you provide concrete examples of proposals made by the teacher to the harness?

  • Can you include the full cost of inference required to perform evolution (both in the student and teacher cost)?

  • Can you provide a trivial performance baseline where the teacher model Astra reconstructs the train scenes using SceneAct, and these examples are provided in-context to the student?

  • Does the number of “train scenes” matter? Do we observe something like a scaling curve as the number of scenes matter?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.