acceptodds
Withdrawn from ICLR 2027

Struct2Real: A Systematic Framework for Accurate and Efficient Structure-Grounded Object Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in image generation have enabled the creation of high-quality visual content with impressive semantic fidelity. However, generating object images under fine-grained structural constraints, particularly preserving topology and spatial layout, remains an open challenge. We propose Struct2Real, a structure-grounded generation framework that enables photorealistic object image synthesis under explicit structural control while supporting flexible viewpoint and structural editing, consisting of twofold. 1) we develop a structure modeling system that enables users to create a 3D structural representation named StructMap, which is composed of various geometric primitives and allows precise, intuitive and editable structural specification. 2) We develop the Structure-Aware Generation Agent (SAGA), a skill-oriented agent that formulates structure-grounded image synthesis as a dynamic skill orchestration process and enables photorealistic object image generation under structural constraints encoded in StructMap. SAGA further incorporates a realism memory module that retains visual attributes throughout generation, enabling viewpoint and structural editing with consistent visual attributes. Extensive experiments demonstrate that Struct2Real achieves strong performance in structure-grounded object image generation while ensuring low user effort required for this task, highlighting the practicality and effectiveness of our method. Please refer to more details in the Appendix and supplementary material.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Structure-grounded Image Generation, 3D Structural Representation, Viewpoint and Structural Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KXGKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Struct2Real, a system capable of generating object-centric images conditioned on the 2D rendering of a sketchy 3D mesh created with simple primitives, and further supports viewpoint and structural editing. The authors claim 2 main contributions: 1) a structure modeling system that enables users to create a 3D structural representation named StructMap, 2) a Structure-Aware Generation Agent (SAGA), an agent that turns the StructMap into images with skills to enhance details and realism. The authors compared the proposed system to other condition variants such as text, line art, and sketches, showing better results.

Critical strengths
  1. The idea of using the rendered image of 3D geometric shapes (the so-called StructMap) as conditions makes sense, and the generation quality indeed looks better than the other condition candidates presented in this paper. While the advantage over lineart is not that clear from the numbers and visualizations, I agree that creating the StructMap is a simpler task for ordinary people.
  2. The results in the editing experiments shown in fig 5 are pretty promising, achieving better consistency and ease of control compared to baselines. This system is more suitable for multi-round interaction between a human artist and the image generation system, compared to representations such as linearts.
  3. The paper is well written and easy to follow. The authors also provide sufficient implementation details (e.g., the prompts) for reproducibility.
Critical weaknesses

I think the system presented in the paper is interesting, but the missing details specified below make it hard to make a fair judgment on the soundness of the claims made in this paper, including both the generation quality and the claimed ease of use.

  1. The authors list the structure modeling system as a core contribution, but it is unclear that the system has advantages over existing software such as Blender. From my understanding, the main selling point of the proposed system is some pre-built primitives (e.g., a set of 4 rectangles that can be used as chair legs). The problem with this is that the pre-defined degrees of freedom are likely not enough when the user starts to create complex objects, where they have to fall back to basic primitives. To make the point stand, the authors should make a systematic study of asset creation time using the proposed systems, compared to software such as Blender and CAD, across objects of different complexity. A good example of a complex object is the desk in cell (e) in fig 5.
  2. The main evaluation dataset used in this paper is constructed by the authors, and many details are unclear. For example, basic statistics such as the total number of scenes and category distributions; is there a reference image given when the human constructs the structure priors? If so, the authors should show the reference image so we can examine the faithfulness of the reconstruction. If not, how are the SSIM and LPIPS computed?
  3. I'm not an expert on image generation, so my confidence is low on this. But from what I see in the related works section, the methods discussed there are mostly from 2023 and 2024, which are pretty old for such a fast-evolving research area. The same holds for the baselines compared; I'm not sure if ControlNet is still a relevant baseline as of 2026. The authors should clarify the reason for not comparing with more recent baselines.
  4. More details are needed for the part involving humans. How many human artists are hired, and what is their level of expertise in creating the different forms of art? This is important to ensure the creation time and difficulty in tab1 are meaningful. For the human study on MOS-R and MOS-A, how many human subjects are involved?
Questions for authors

N/A

Reviewer_MC8uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Struct2Real, a framework for photorealistic object image generation under explicit structural control (topology and spatial layout), with support for viewpoint and structural editing. It has two parts: (1) StructMap, a 3D structural representation composed of parameterized geometric primitives, built with an interactive interface; and (2) SAGA, an agentic framework on top of an image generator that iteratively evaluates the generated image and invokes one of three skills (structure preservation, detail enhancement, appearance refinement), plus a realism memory module that keeps appearance consistent across viewpoint and structural edits. The authors build a 3000-sample dataset with aligned text / lineart / scribble / StructMap conditions and show improved structural fidelity and realism compared to text-, lineart- and scribble-conditioned baselines, along with creation time and difficulty measurements for each condition.

Critical strengths

1. The paper addresses a real need: a structural representation that is controllable (easy to edit and to render from any viewpoint) while not being too hard to create. Prior primitive-based control such as Blocks2World and LooseControl works at the scene level; this paper focuses on object-level, part-aware structure (part count, connectivity, relative pose), which is a useful distinction.

2. The method preserves the input structure to a good extent, and the authors have put good effort into improving this preservation (the structure preservation skill with multi-view/normal/position maps, and the realism memory for appearance-consistent editing).

3. The 3000-sample dataset with aligned text, lineart, scribble and StructMap conditions describing the same structure is useful in its own right, and enables cross-condition comparison.

4. The viewpoint and structural editing results are convincing. Edits follow the modified StructMap while keeping appearance consistent, and the realism memory ablation (Table 3) supports this, with appearance consistency improving from 3.84 to 4.62.

Critical weaknesses

1. Limited novelty of StructMap.

Primitive-based control for image generation has already been explored in works like Blocks2World, LooseControl, and Generative Blocks World (which is not cited). The authors argue that StructMap is different because it focuses on object-level, part-aware structural control. I see the distinction, but the underlying representation, a set of parameterized geometric primitives rendered into an image, is still quite similar to these earlier works. I am not convinced that StructMap itself offers enough novelty.

2. StructMap is not necessarily easy to create.

Table 1-Right shows that creating a StructMap takes 6.69 minutes on average (difficulty 2.68), compared with 3.26 minutes (2.26) for scribbles and 2.79 minutes (2.09) for text. So it takes roughly twice as long as scribbles and is also rated as more difficult. Its clear advantage in creation time is only over lineart (25.76 minutes).

I understand that text and scribbles give weaker structural fidelity, so StructMap is not meant to be the fastest option; the actual claim is a better trade-off between user effort and structural fidelity. However, the paper does not show this trade-off directly. It reports effort and fidelity separately, and the "low user effort" claim is mainly supported by the comparison with lineart. Showing structural fidelity as a function of creation effort (e.g., more detailed scribbles, or StructMaps with fewer primitives) would make this claim more convincing.

Also, the creators used the authors' own interface, which includes 40 higher-order templates (Appendix A.8). I wonder whether new users with no experience using this interface would achieve similar creation times.

3. Limited expressiveness of StructMap.

I am also concerned about how well StructMap can represent more complicated objects. The primitive library mainly consists of simple shapes like cuboids, cylinders, tori, and spheres. This works well for objects with regular geometry, but what about irregular shapes, complex curved surfaces, or objects with many small components?

For example, the non-rigid examples in Appendix A.12 required additional deformation in Blender, outside the proposed interface. The failure cases also involve small components and cluttered layouts. I would like to see how well StructMap works beyond simple or moderately complex objects.

4. Novelty of SAGA and the need for three separate skills.

I have a similar concern about SAGA. Iterative generate-evaluate-refine pipelines are already well established, including Idea2Img and SLD, which the authors cite.

The three skills make sense, and the ablation study shows that they help. However, I am not sure why we need to separate these objectives into three individual skills. Could a single, well-designed refinement prompt containing the important instructions from all three achieve similar results?

Table 2 removes one skill at a time and compares the full method with single-shot generation without SAGA. This mixes several factors: the benefit of iteration itself, the extra conditions (multi-view images, normal maps, position maps, and structural descriptions), and the benefit of splitting the process into separate skills.

A useful baseline would be an iterative refinement pipeline with the same conditions and iteration budget, but using one combined refinement prompt. A fixed sequence of the three skills without dynamic routing would also help.

Even if all three skills improve the results, I still struggle to see what new research insight comes from this particular decomposition.

5. How much of the improvement comes from StructMap, and how much comes from SAGA?

In Table 1, lineart with GPT Image 2 already performs quite close to Struct2Real. MOS-R is 4.58 versus 4.75, MOS-A is 4.51 versus 4.71, and FID is 38.74 versus 38.51. Lineart with ControlNet++ even achieves a slightly higher SSIM (0.706 versus 0.694).

Importantly, this compares single-shot GPT Image 2 with GPT Image 2 using the full SAGA pipeline.

Interestingly, in Table 2, StructMap with GPT Image 2 but without SAGA gets MOS-A of 3.86 and MOS-R of 4.02. These are lower than scribble with GPT Image 2 (4.05/4.19) and lineart with GPT Image 2 (4.51/4.58) in Table 1. Of course, these results come from different subsets, so this is only an observation, not a direct comparison.

Still, it makes me wonder whether most of the improvement actually comes from SAGA rather than StructMap.

The authors already apply SAGA to lineart in Appendix A.5.2 and report comparable structural consistency, but only qualitatively. A quantitative comparison of SAGA across all four conditioning types would help separate the contributions of the representation and the generation pipeline.

This would not necessarily make StructMap less useful, since creating detailed lineart is still more time-consuming according to their study.

Overall assessment. My current rating reflects the concerns above. I am open to changing it based on the authors' rebuttal and the other reviews; the responses to W4 and W5 will weigh most in my final assessment.

Questions for authors

1. What is the novelty of SAGA compared to other iterative image generation methods (e.g., Idea2Img, SLD)? What is the benefit of 3 separate skills compared to one skill that contains the important content of all three, under the same iteration budget and the same conditions?

2. Since lineart with single-shot GPT Image 2 is already close to Struct2Real in Table 1, what happens if a similar iterative refinement loop (evaluate the output against the input condition and regenerate, as in SAGA but adapted to lineart, scribble and text) is applied to the other conditions? Can you report quantitative results (FID, SSIM, LPIPS, MOS) for this setting, i.e., a quantitative version of A.5.2? This would show whether the gain comes from StructMap or from the iterative refinement.

3. Can the StructMap representation be improved to represent more complex objects (e.g., non-rigid or organic shapes) with less user effort?

4. Can StructMaps be obtained automatically instead of being built from scratch? For example: (i) use GPT Image to turn the user's scribble into a realistic image, (ii) ask a multimodal model to generate Blender code that approximates the object with a small number of geometric primitives (possibly specifying roughly how many), (iii) let the user edit this scene or render it from another viewpoint, and (iv) give GPT Image the original image together with the edited structural rendering to generate the final image. Using code-generated 3D scenes as a structural condition has also been explored recently, e.g., Code Video Model (Code Video Model Team, tech report, https://code-video-model.github.io/), where an LLM writes a program (Three.js) that lays out the scene, objects and camera, and the rendered proxy video conditions a pretrained video model. This pipeline may introduce errors (e.g., for hidden parts), but given the paper's emphasis on reducing user effort, how would it compare to the manual StructMap interface in creation time, structural fidelity, and editing accuracy? The Hunyuan3D experiments in Appendix A.6 are related, but they do not test this pipeline.

Reviewer_37HJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to address an important problem: how to get a structured 3D representation easily and transform it into photorealistic images. It further leverages a powerful agent-based framework to ensure that the generated images preserve the topology and spatial layout of the input structure.

Critical strengths
  1. This paper studies a very interesting problem: how to render a structure-controllable 3D representation into photorealistic images. With the emerging capabilities of VLMs in 3D-as-code, I believe this is a very interesting and important topic. It raises the question of whether VLMs can leverage these tools to transform structured code representations into photorealistic 3D worlds.

  2. The feedback loop proposed in this paper to ensure the correctness of topology and spatial layout seems to be a solid approach.

  3. The final performance looks promising, and the qualitative results and user study appear convincing.

Critical weaknesses
  1. The challenges raised in this paper are important, but I do not think the authors have fully addressed them. The authors claim that StructMap provides a user-friendly interface, but the evidence from the user study and quantitative evaluations is not sufficient to fully support this claim. For example, could the authors leverage the powerful capabilities of current LLMs to generate an initial StructMap and further reduce the time and effort required from users? As an ML paper, I still believe that how to generate a StructMap is a more important challenge.

  2. The authors claim their second contribution is SAGA, which uses a feedback loop to ensure structural correctness. However, this pipeline seems more like an engineering effort rather than a novel methodological contribution, as similar feedback-loop approaches have already been studied in previous work.

  3. Figure 4 is quite confusing. I believe the feedback loop should be illustrated with a real example showing the actual process, rather than simply presenting a conceptual diagram.

  4. I think the comparisons in this paper are somewhat unfair because StructMap requires human effort, while other methods do not necessarily require such effort. Moreover, considering the current capabilities of ChatGPT-Astra, I suspect that using Python code to generate 3D objects in Blender could achieve comparable results with structured representations, potentially without requiring any human effort.

Questions for authors
  1. Since StructMap is a 3D representation, can Struct2Real guarantee 3D consistency across images rendered from different viewpoints?

  2. What are the intended application scenarios of Struct2Real? In what practical settings would this framework be particularly useful?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.