- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper notes the recent breakthrough in 3D scene creation brought about by frontier models like GPT-6 Astra and aimed to bring similar capabilities to smaller open source models that cost a lot less (e.g. Qwen)
The two proposed contributions are 1) prompt-based definition of interfaces to help the model interact with Blender 2) a method to evolve the harness by using a stronger frontier model like GPT-6 Astra.
Critical strengths
The ambition and research goal of the paper is quite interesting and could be impactful if it is successful. If we can use a frontier model to create a generalization harness improvement for the task of 3D reconstruction based on image, the saving in compute cost would be quite substantial.
To this end, the paper does show a consistent improvement on Qwen performance. And show that Qwen + their method out performs GPT-6 Sol on image based metrics while being cheaper.
Critical weaknesses
Even if harness evolution is possible to improve the model quality (which other paper has already shown), this improvement is not useful unless the improved model reached sufficient quality in its output so that the 3D reconstruction is useful.
This paper does not define any downstream task to evaluate the quality of the 3D reconstruction. This means there is no benchmark on the applicability of the method quantitatively. Qualitatively, the results are also not convincing. Although it beats Sol in image-based metrics, the result are quite far away from a realistic reconstruction of the input image. Although Astra cost 4-5 times more, the quality gap is so large that I would always prefer to spend this money on Astra as both Sol and Qwen outputs does not look usable.
In summary, as a method paper, nothing in the method is very novel. it is the first application of these concepts to the 3D reconstruction task. As an application paper, the results are not impressive. The created 3D reconstruction is not sufficiently good so that the cost saving becomes a factor. If it can achieve Astra-level reconstruction at a fraction of the cost, then the method has meaningful applications.
Questions for authors
Suggestion for future work is to apply this method to stronger baseline like Sol. Maybe with a strong enough harness, Sol can rival Astra in quality while still being cheaper. Qwen is too weak, and it is not clear that improvements shown is transfererable to other stronger model. (or generally applicable when stronger models is released)