acceptodds
Under review as a conference paper at ICLR 2027

PoseStage: Unifying External Pose and Internal Structure Control in Multi-Object Scenes

Abstract

Controllable image generation can place objects, specify their orientation, or articulate a human body, but in scenes with several open-category objects these controls are rarely available at the same time. We present PoseStage, a unified framework that controls both the external pose and the internal structure of multiple objects from two 2D geometric maps. PoseStage supports three novel pose-and-structure tasks within one framework: generation, reference-guided generation, and editing. A model trained only on standard conditional generation handles the latter two at inference time, without task-specific training, by extending its input sequence. To prevent the appearance and text of different instances from mixing, we apply a training-free instance-routing mask at inference that removes direct attention between instances and reduces attribute confusion. Training a single model to control both aspects requires images annotated with external pose and internal structure at the same time, which existing datasets do not provide. We therefore build a hybrid dataset of real-world and synthetic images that provides aligned annotations for external pose and internal structure, and a benchmark that evaluates both controls, on which PoseStage outperforms existing methods.

Primary area:
generative models
Keywords:
Diffusion Models, Controllable Image Generation, 9-DoF Pose, Internal Structure

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iqmAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a simple method for controllable image generation targeting on joint control of external object compositions and structure/pose of individual objects. External composition is represented via 3D bounding boxes color coded with coordinate map and internal object poses are represented as 2D skeletons. Both of these representations are rendered as 2D images and are used to condition the text-to-image generative model. The conditioning is performed by concatenating the tokens of these 2D conditioning image tokens along with the noise image tokens of the flow model. Additionally, to disambiguate between multiple boxes and their text, an attention masking operation is for the object specific text and reference tokens attend only to the object region during generation. A combination of synthetic and real proprietary dataset is used to train this model.

Critical strengths

[Simple approach]:

The proposed method is simple and generalizes for additional downstream tasks such as image editing and reference driven generation without training. The method leverages established in-context conditioning mechanism of DiT where the conditioning can be directly concatenated to condition the image generation on different control signals. Adding pose control along with the scene composition control is new seems to be working well with in-context conditioning.

[Results]:

The qualitative and quantitative results present in the paper are promising. The generated scenes look more realistic as compared to the baseline approaches and follows the conditioning layouts well.

Critical weaknesses

[Novelty concern]:

  • Presented approach is incremental on top of the existing works SceneDesigner, SeeThrough3D, BoxCtrl as they have already proposed 3D bounding box based control for image generation and editing. The major distinction is adding the skeleton rendering as an additional conditioning. The conditioning mechanism is straightforward where the tokens of new skeleton map is additionally appended to the bounding box conditioning map. Additionally the attention mask operation to disambiguate objects is already present in the SeeThrough3D work. Though adding skeleton conditioning improves the results qualitatively and qualitatively, I believe the contribution is marginal. Forming a new representation that compactly represent both the internal and external structure in a common space can be considered as a good contribution over existing methods but this paper lacks in that respect.

[Representation]:

  • The proposed representation of bounding boxes and 3D skeletons share some common information. For example, skeletons already capture extent of the object and 3D orientation as well. And for rigid objects 3D bounding boxes with orientation based color coding is sufficient to capture the objects internal structure as it is fixed. Why do we have two separate conditioning mechanism? an ideal would be creating a new representation that encodes both the information efficiently.

  • During inference, a user has to draw both the bounding boxes and skeleton to perform conditioning image generation. Drawing bounding boxes is relatively easier but drawing accurate skeleton is challenging to draw manually for most of the users. How does the proposed method performs if only bounding box conditioning is passed during inference?

[Experiments]:

  • The paper has not compared with state-of-the-art methods for bounding box based image generation SeeThrough3D[1] and editing method BoxCtrl[2]. A direction comparison for these methods is important to understand the contribution of the proposed method.

  • Why not use the attention routing during training also? As the dataset consists of object masks and 3D bounding boxes, it makes intuitive sense to use it during the training stage also. An ablation experiment should be provided to understand this design choice.

  • The method a combination of real and synthetic datasets an ablation on real vs synthetic dataset should be provided to understand the contribution of each of the components.

[Presentation]:

The paper is easy to read but there are some concerns about presentation particularly about missing important details, as discussed below:

  • Dataset details: The dataset generation process is not clearly presented, the paper claims use of 'internal image/asset collection', and just report the total number of images in the appendix. As dataset is one of the core component of the proposed method, detailed analysis about the dataset should be provided for understanding the data distribution - types of images, distribution of categories, and a sufficient number of representative images from the dataset.

  • Evaluation benchmarks: All the three evaluation benchmarks are not detailed with the number of samples and distribution of images. Disclosing evaluation set or evaluating on public benchmarks is important to validate the correctness of the comparisons presented.

  • Section 3.1 - The geometric conditioning for the external map and the internal maps should we explained in detail about their construction as it the important part of the proposed model pipeline.

  • L218: The training objective is not desire in details about the loss function and how the time steps were sampling for training. Though these details are provided in appendix, presenting them in the main paper is crutial.

[1] SeeThrough3D: Occlusion aware 3d control in text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

[2] BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing, In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH), 2026

Questions for authors

[Clarification]:

  • L169: Disjoint masks {M_i} . What happens if there is some occlusion in the box as it is common for real world scenes where objects don’t fill the complete box tightly such as bicycle, hence the objects may not occlude in the imager but their boxes can occlude upto some extent.

  • L219: Image editing without the mask can be ambiguous: Imaging two humans in the input scene and the user defined different conditioning for each of the humans, how will the model be able to accurately assign the human to its correct conditioning form.

  • L278: What does geometry mean extracted from SAM 3 means? Is this segmentation masks?

  • L279: What is reconstruction rate? Aren’t you doing generation, how to measure the reconstruction?

6 comments
AuthorsOpenReview ↗

Response to Reviewer iqmA (1/3)

Thank you for the thoughtful questions on novelty, representation design, training strategy, and evaluation. In response to these points, we added new comparisons and further clarified the method design and evaluation protocol.

W1a: Novelty at the Capability Level

Thank you for the in-depth discussion of novelty and representation. To clarify our view, the contribution of this work at the capability level lies not only in applying box control or attention isolation, but also in enabling joint control of external pose and internal structure for open categories and multiple instances, and in showing that the same model design can be reused across reference-free generation, reference-guided generation, and editing.

W1b: Novelty at the Representation Level

At the representation level, the internal map in the paper is not a standard 2D skeleton. Instead, it is a control map obtained by projecting a 3D geometric graph and encoding colors with canonical coordinates. This 3D-aware representation provides richer spatial information than a 2D skeleton.

Regarding your questions about why we do not use a more compact unified representation and about sequence concatenation, Table R1 provides ablations that test the practical value of separating the two conditions before fusion:

  • Why separate encoding followed by fusion: We render the box and skeleton into a single image with 0.5 transparency (Table R1). The results indicate that this simple unified representation introduces information loss from occlusion and color overlap.
  • Clarification and comparison for sequence concatenation: Our method is not simple concatenation of two sequences; instead, we first concatenate along channels and then fuse through a learned layer. We tested direct concatenation along the sequence dimension (Table R1). The results indicate that direct concatenation doubles the number of tokens and increases computation, whereas our learned fusion achieves a better balance while using half as many control tokens.

W2: Complementarity of the Two Condition Representations

In response to your question about using two separate condition representations, we agree that the internal structure map and the external box do share some spatial information (for example, the color encoding in the internal structure map also implicitly contains pose information). However, the two are functionally complementary: the external box provides an explicit global anchor for absolute position and scale, whereas the internal structure map provides a more detailed description of internal shape. As shown in the geometric-condition ablation in Table 5 of the original submission, using both conditions together yields better geometry metrics than using either condition alone (external only or internal only).

W3a: Cost of Using Skeletons

Regarding the cost of using skeletons, the framework is flexible in practice: users can decide how many joints to create depending on the desired level of control precision. In addition, to further lower the interaction burden, we provide preset templates for common poses, which users can edit directly.

W3b: Support for Box-Only Inference

To further clarify the question of box-only inference, the original model in the paper always receives both the external map and the internal map during training and therefore is not directly exposed to inputs with only the external map. It thus cannot directly support box-only inference. To answer this question, we additionally trained a model with dropout on the internal condition at probability 0.5 and evaluated it under both external-only input and full dual-condition input (Table R8). The results indicate that, under external-only input, this dropout model achieves higher geometric accuracy than the external-only model in Table 5 of the original paper, which was both trained and tested with only the external condition. Under full dual-condition input, its performance is again close to that of the full model. This suggests that simple condition-dropout training can make the model reasonably compatible with box-only inference while preserving most of the benefit of dual-condition input.

Table R8: Support for box-only inference

TrainingInference InputRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)PickScore (up)
Dropout (p=0.5)External only100.042.4256.10.15630.25400.04590.65920.292
Dropout (p=0.5)External + Internal98.929.0071.70.11970.20390.02460.86820.267
External onlyExternal only100.046.4751.90.14770.27600.04910.65020.337
Full modelExternal + Internal100.024.7774.60.10250.19410.02720.87320.314
AuthorsOpenReview ↗

Response to Reviewer iqmA (2/3)

W4: Comparison with SeeThrough3D and BoxCtrl

We added additional comparisons with relevant baselines. For SeeThrough3D, we compare on the shared generation task; for BoxCtrl, we add a comparison on its related rigid editing task.

Table R4: Additional comparison with SeeThrough3D (reference-free generation)

MethodRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)PickScore (up)
SeeThrough3D98.930.4367.10.13190.26310.04460.66420.532
Ours100.024.7774.60.10250.19410.02720.87320.314

Table R5: Additional comparison with BoxCtrl (rigid editing)

MethodRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)PSNR (up)SSIM (up)DINO (up)
BoxCtrl100.024.3078.70.13660.198024.580.8340.798
Ours100.018.1580.60.14640.194621.870.7600.826

W5: Ablation of Training-Time Routing

We did not introduce routing during training mainly because we want a single model to support multi-object generation, single-object generation, and image editing within one unified framework, while these tasks do not require the same degree of instance-level isolation. We therefore designed routing as a training-free inference-time mechanism that can be enabled or disabled flexibly depending on the task. As shown in Section 5.2 (Table 6), this design already improves instance binding in multi-object scenes without requiring additional training.

W6: Contribution of Real and Synthetic Data

To disentangle the effect of data source from that of data size and training budget, Table R2 compares real-only, synthetic-only, and mixed training under matched data size and matched budget. The results indicate that the real-only model produces more natural image appearance but has weaker geometric precision, while the synthetic-only model achieves higher geometric accuracy but shows a stronger synthetic style. Figure 7 in the revised supplementary material (Appendix A.7) also illustrates this difference more directly. By contrast, the mixed model achieves a better balance between image quality and geometric accuracy.

W7: Dataset Details

The initial submission was limited by internal disclosure review, so the dataset statistics were not sufficiently detailed. We have now added detailed category distributions for the training data in Section 1 of the General Response.

W8: Benchmark Scale and Verifiability

In Section 1 of the General Response, we added detailed benchmark sizes and asset provenance information. To improve verifiability, we plan to release the test prompts, control maps, and representative evaluation examples after the necessary anonymization and compliance review.

W9: Additional Details on Geometric-Map Construction

Thank you for the suggestion. In the revised main paper, we have clarified the coordinate-color encoding, the construction of the 3D geometric graph, and the projection-based rendering process.

W10: Additional Details on Loss and Sampling

We have also clarified key training and sampling settings in the revised main paper, including the weighted flow-matching MSE loss and the discrete time indices of the uniform-sampling scheduler.

AuthorsOpenReview ↗

Response to Reviewer iqmA (3/3)

Q1: Details of Disjoint Masks

The disjoint masks here are not pixel-accurate segmentations. What we mean is the following: after downsampling the projected regions to the latent grid, if different objects overlap, we assign the overlapping locations to one instance according to user input order, producing mutually exclusive token-assignment masks. The purpose is to provide an instance-level partition for instance routing and reduce cross-instance leakage from text or reference-image conditions as much as possible. We agree with the concern that this partition is not perfect in heavily occluded scenes. Still, it is sufficient to support the routing mechanism in practice and improves instance binding in multi-object generation.

Q2: Identity Correspondence Without Masks

In the current editing setup, identity correspondence mainly relies on the global appearance context from the source image, the spatial position and pose in the target control map, and explicit object descriptions in the editing instruction. As a result, when different objects can be clearly distinguished by semantic descriptions, the model can usually perform the correct edit. We also agree with the reviewer's concern that, when a scene contains multiple visually similar people and each needs a different instance-level condition, the correspondence becomes more challenging without explicit masks or routing. This more difficult multi-object editing setting is a meaningful direction for future work and calls for finer-grained instance-level control mechanisms.

Q3: Clarification of the Division of Labor in the SAM 3 Geometry Module

SAM 3 performs 2D instance segmentation, and SAM 3D then uses the segmentation results for 3D reconstruction.

Q4: Clarification of Rec

Rec (Reconstruction Rate) refers to the success rate of reconstruction, i.e., the fraction of cases in which the evaluation pipeline successfully recovers a 3D bounding box for the target instance. If the generated image misses the target instance, or if the instance is too blurry for SAM to identify, reconstruction fails.

Reviewer_bR45OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PoseStage controls the external pose and internal structure of objects using two geometric maps: a bounding-box coordinate map for 9-DoF pose and a rendered geometric graph for internal structure. It encodes the maps separately and combines them through a learned fusion layer to condition a FLUX.2 backbone. For multi-object generation, an inference-time attention mask limits access to other objects' text and reference tokens while allowing spatial tokens to communicate across the scene. The model is trained on conditional generation and uses the backbone's reference-image interface for reference-guided generation and editing, without additional task-specific training. The paper also constructs 13,862 real and synthetic training pairs and evaluates the three tasks on separate benchmarks. The reported results show improved geometric control, and the ablations examine the effects of the two maps and instance routing.

Critical strengths
  1. PoseStage allows users to specify both where objects appear and how their internal structure is arranged within one framework. On multi-object reference-free generation, Table 1 reports an angular error of 26.93°, compared with SceneDesigner's 57.53°, and a structural F1 of 0.868, compared with 0.521. Figures 3–4 also show examples of generation and editing under combined pose and structure constraints. These results support the practical value of controlling both aspects together.

  2. Table 5 compares separately trained models and shows that adding the external map to the internal map reduces angular error from 31.14° to 24.77° and translation error from 0.1294 to 0.1025. This supports the use of an explicit external-pose condition even though the internal map already contains projected pose information. Table 6 shows that enabling routing at inference improves reference binding accuracy from 82.6% to 88.1%, providing evidence that the mask helps assign each reference appearance to the intended object without additional training.

Critical weaknesses
  1. Insufficient evidence for the fusion design and hybrid-data benefits:

Table 5 shows that using both control maps is beneficial, but does not establish whether learning the fusion weights improves performance over keeping the initial equal-weight average fixed. A comparison using the same backbone, training data, and training budget would clarify the value of the learned fusion layer. Likewise, the paper attributes natural appearance diversity to real images and geometric precision to synthetic data, but does not compare models trained on real data only, synthetic data only, and their combination while accounting for dataset size and training budget. Thus, the claimed benefits of the fusion design and the complementary data sources remain insufficiently validated. Separately, general-purpose baselines receive textual descriptions of target geometry; where their interfaces permit, providing the control maps directly would help assess whether the reported gap partly reflects information lost in textual conversion.

  1. The evaluation protocol leaves uncertainty about coverage and geometric accuracy:

The paper states that evaluation sources are disjoint from training, but does not report benchmark sizes, category and object-count distributions, or how overlap between individual assets is excluded. These details are needed to assess the coverage of the multi-object and open-category claims. Geometric scores are obtained through SAM 3/SAM 3D, which is also used to annotate the real training images; independent checks would help determine how reconstruction errors affect the reported results. Appendix A.7 additionally aligns reconstructed depths using target depths, eliminating center-depth error when only one instance is reconstructed, and evaluates structure through reprojected 2D skeletons. The resulting scores measure aligned or projected agreement, but do not fully establish absolute depth or 3D internal-structure accuracy. The 20-participant human study provides complementary evidence, yet its sample count, scoring procedure, and uncertainty estimates are not reported in sufficient detail.

Questions for authors
  1. Table 5 compares one control map with two, but does not compare learned fusion with fixed fusion. How does the model perform when the fusion layer is kept at its initial equal-weight average, using the same backbone, data, and training budget?

  2. Does training on both sources improve image quality and geometric accuracy over training on either source alone? Please provide comparisons that account for dataset size and training budget, so that the benefit of combining the sources can be distinguished from the benefit of using more data.

  3. The paper describes providing general-purpose editors with textual descriptions of the target geometry. Where their interfaces support multiple images, have the authors also supplied both control maps with explanations of their encoding? Such a comparison would help determine whether part of the performance gap comes from converting geometric information into text. If this input is not supported, please clarify the interface limitation.

  4. How many images and instances are in each benchmark, which categories and object counts are represented, and how is asset overlap with training excluded? Please report the single-object and multi-object results separately for Table 2 and clarify whether the tests include more than three objects or substantial occlusion.

  5. Have the SAM 3/SAM 3D measurements been checked against independent annotations or known geometry to quantify reconstruction error? Given the use of target-depth alignment and projected skeletons, what additional evidence, if any, supports correct relative depths between objects or correct 3D internal structure?

  6. How many samples were evaluated, what rating scale was used, how were outputs presented and randomized, and what is the uncertainty in the reported scores? Please also explain the statement that human and automatic rankings agree on every axis: Table 1 gives SceneDesigner a higher multi-object PickScore than PoseStage, whereas Table 7 gives PoseStage a higher human image-quality score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bR45 (1/2)

Thank you for the detailed and constructive comments. In response to these questions, we added new comparisons, more evaluation details, and clarifications of several statements. We reply point by point below.

W1a: Benefit of Learned Fusion over Fixed Averaging

To address whether learned fusion outperforms fixed averaging, we added a fixed equal-weight averaging baseline under matched training conditions, as reported in Table R1. These results further support that learned fusion yields a better balance between geometric accuracy and image quality.

W1b: Ruling Out Other Factors in the Mixed-Data Setting

To disentangle the effect of data source from that of data size and training budget, Table R2 adds ablations with real-only and synthetic-only training under matched data size and matched training budget.

W1c: Gap for General-Purpose Editors Under Control-Map Inputs

We further evaluated three strong general-purpose image editors: FLUX.2-klein, Qwen-Image-Edit, and GPT-Image-2. For each model, we used a unified prompt template that explicitly states that the external map specifies object orientation, position, and scale, while the internal map provides pose information, and we also supplied the original text prompt so that the model could generate directly from these conditions. Under this unified input setting, none of the three models was consistently able to use the two control maps stably and accurately: FLUX.2-klein and Qwen-Image-Edit occasionally rendered the box or skeleton directly into the image; GPT-Image-2 did not make that specific mistake, but its generated external pose and internal structure still showed clear deviations from the input conditions. Taken together with the quantitative results in Table R6, these findings suggest that current general-purpose image editors still have clear limitations in directly using our geometric control maps to produce high-fidelity results, making dedicated adaptation to this input format worthwhile.

Table R6: Comparison with general-purpose models under geometric control-map inputs (reference-free generation)

MethodRec. (%) (up)Ang.Err (deg, down)[email protected] (%) (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)PickScore (up)
FLUX.2-klein-9B81.855.5244.40.16250.28140.03690.75619.919
GPT-Image-290.947.3452.50.27180.44660.07630.51021.417
Qwen-Image-Edit-251188.643.3850.90.30150.32620.04520.71720.146
Ours100.024.7774.60.10250.19410.02720.87320.314

W2a: Independent Reconstruction Validation

In response to your concern about evaluation validity, as detailed in Table R3, we conducted an independent validation on real images from Objectron, which provides additional support for the effectiveness and reliability of the current evaluation protocol.

W2b: Relative Depth Design

We focus on relative depth because, in practical scene composition, users may not know exact metric distances either. Accordingly, we ask the model to control relative depth relations between objects, rather than absolute distances in meters. The shared scale alignment used in evaluation does not alter depth ratios or front/back ordering across instances; incorrect relative depth still produces evaluation error. This protocol is therefore consistent with the relative-depth control target of the paper.

W2c: 3D Internal-Structure Metrics

We acknowledge that metrics based on reprojected 2D skeletons (Chamfer/F1) cannot fully capture complete 3D structure. They nevertheless provide a useful quantitative reference for visible-structure faithfulness, and their conclusions are broadly consistent with the human evaluation in Appendix A.9, Table 10.

AuthorsOpenReview ↗

Response to Reviewer bR45 (2/2)

Q1: Comparing Learned Fusion with Fixed Averaging

For the baseline comparison discussed above, the results in Table R1 indicate that our learned fusion scheme provides a better balance between geometric accuracy and image quality than fixed averaging (Table R1), further supporting the benefit of learned fusion.

Q2: Complementary Benefits of Mixed Data

The results in Table R2 indicate that the real-only model generates more natural image appearance but has weaker geometric precision, whereas the synthetic-only model achieves higher geometric accuracy but exhibits a stronger synthetic style. We also added more direct qualitative comparisons in Figure 7 of the revised supplementary material (Appendix A.7). By contrast, the mixed model achieves a better balance between image quality and geometric accuracy, suggesting that the two data sources are complementary for this task.

Q3: Providing Control Maps to General-Purpose Editors

As discussed in W1c above, in our experiments general-purpose editors still showed interpretation errors and information loss during conversion when receiving the control maps.

Q4a: Benchmark Details

In Section 1 of the General Response, we added detailed benchmark sizes and category statistics. All pose conditions and prompts in the test set were manually constructed in our editor and then reviewed. This part did not use or refer to pose conditions from the training set.

Q4b: Results in Table 2

Thank you for the suggestion. In the revised Table 2, we now report single-object and multi-object results separately.

SplitMethodRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)CLIP-I (up)DINO (up)
SingleMS-Diffusion100.074.2515.90.12540.42340.08100.4440.6750.435
SingleLaMIC96.859.6736.10.12480.41490.07240.4900.6080.344
SinglePoseStage (Ours)100.014.4385.70.02950.17950.02110.8970.7730.552
MultiMS-Diffusion99.587.8714.00.81690.73360.11510.4200.6140.246
MultiLaMIC97.086.7215.40.82870.75230.16980.3240.5350.106
MultiPoseStage (Ours)100.024.0779.10.12080.18960.02750.8800.6990.393

Q4c: Three-Object Cases and Heavy Occlusion

The multi-object subset contains up to three objects, and it includes a small number of cases with severe occlusion.

Q5: Reliability of Reconstruction Error

As shown in Table R3, the reconstruction success rate and the external geometric errors all fall within a reasonable range, providing additional support for the relative reliability of our geometry measurement pipeline in real scenes.

Q6a: Details of the User Study

For the 20-participant user study in Appendix A.9, a total of 120 images were scored. Ratings used a 1-5 scale. Outputs were presented one case at a time, without method names, and the display order was randomized to reduce subjective bias as much as possible. For reporting, we normalize the ratings to [0,1], and additionally report mean and standard deviation (Mean +/- SD), as shown in Table R7.

Table R7: User-study statistics (Mean +/- SD)

MethodOrientation (up)Pos. & Scale (up)Internal Struct. (up)Image Quality (up)
SceneDesigner0.77 +/- 0.150.81 +/- 0.120.43 +/- 0.220.75 +/- 0.14
SKDream0.74 +/- 0.160.39 +/- 0.210.80 +/- 0.130.31 +/- 0.19
InstanceDiffusion0.45 +/- 0.240.67 +/- 0.180.41 +/- 0.250.57 +/- 0.20
PoseStage (Ours)0.88 +/- 0.090.93 +/- 0.060.91 +/- 0.080.82 +/- 0.11

Q6b: Consistency Between Human Evaluation and Automatic Rankings

Thank you for pointing this out. On the image-quality axis, human evaluation and PickScore do show some discrepancy. The phrase "absolute agreement" in the initial submission was therefore too strong, and we have corrected that statement in the revised paper.

Dear reviewer bR45, thank you for your constructive comments. We just wanted to gently follow up to see if our responses have addressed your concerns, or if there are any remaining points you would like us to clarify.

Reviewer_XySfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines external pose and internal structure control for multiple objects using two rendered geometric maps. The paper adds a learned fusion layer to FLUX.2 and uses an inference-time attention mask to bind each object to its text and optional appearance reference. The model is trained on a hybrid real–synthetic dataset for conditional generation, then reused for reference-guided generation and editing without additional task-specific training. Validation covers these three settings through geometric, appearance, and preservation metrics, ablations, and a human study. The reported results show strong geometric adherence, with some tradeoffs in image quality and background preservation.

Critical strengths
  1. combination of controls are very interesting and very useful. Controlling placement, orientation, articulation, and appearance together is a meaningful capability for scene creation. The qualitative examples and quantitative results are also convincing that the method clearly is able to effectively controls different components.

  2. The ablations are adequate and convincing. Using both maps improves geometric adherence, and inference-time routing improves reference binding without additional training. The same ablation might be useful for other control papers as well.

Critical weaknesses

The comparisons leave the source of improvement unclear. PoseStage has a newer backbone and richer geometric inputs than several baselines. Without a baseline using the same backbone, data, and both maps, I cannot tell how much the proposed architecture contributes. SeeThrough3D and DreamRenderer are also close enough that stronger experimental comparison would better attributes the place of the critical design.

Questions for authors
  1. Were any baselines adapted to the same training data and backbone? Is there a simple dual-map conditioning baseline that would isolate the contribution of the fusion and routing design?
  2. How sensitive are the conclusions to the reconstruction pipeline and alignment? Do you have independent geometric checks or results before alignment?
  3. What are the benchmark sizes, object-count distributions, and category/asset separation from training? How much of the evaluation includes substantial inter-object occlusion?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XySf

Thank you for the detailed questions. To address your concerns about the source of gains, comparisons with related baselines, and evaluation reliability, we added new experiments and statistics, and we respond point by point below.

W1a: Attribution of the Gains

Thank you for raising the question of where the architectural gains come from. As shown in the added representation-and-fusion ablation in Table R1, we compare four schemes under the same backbone, training data, and training budget. The results indicate that, compared with single-map joint rendering (Table R1) and dual-map sequence concatenation (Table R1), our learned fusion strategy reduces the information loss introduced by transparent compositing while maintaining high computational efficiency with half the number of control tokens.

W1b: Comparison with Relevant Baselines

Thank you for the suggestion. To address this point more directly, we added comparisons with relevant baselines. We did not include DreamRenderer in this round of experiments because its setup differs substantially from ours and it requires depth maps as additional inputs, which are not available in our benchmark. As an alternative, we compare against SeeThrough3D on the shared generation task and add a comparison with BoxCtrl on the related rigid editing task.

Table R4: Additional comparison with SeeThrough3D (reference-free generation)

MethodRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)PickScore (up)
SeeThrough3D98.930.4367.10.13190.26310.04460.66420.532
Ours100.024.7774.60.10250.19410.02720.87320.314

Table R5: Additional comparison with BoxCtrl (rigid editing)

MethodRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)PSNR (up)SSIM (up)DINO (up)
BoxCtrl100.024.3078.70.13660.198024.580.8340.798
Ours100.018.1580.60.14640.194621.870.7600.826

Q1a: Simple Dual-Map Baseline

In response to your suggestion to add a simple dual-map baseline, Table R1 reports a fixed equal-weight averaging baseline under the same training conditions. The results indicate that our learned fusion scheme yields a better balance between geometric accuracy and image quality than this baseline.

Q1b: Contribution of Routing

Regarding the independent contribution of routing, as shown in the ablation in Section 5.2 (Table 6) of the original submission, we compare results with and without routing in multi-object settings, including both reference-free and reference-guided generation. The results show that enabling routing substantially improves binding accuracy and appearance preservation without harming geometric accuracy, thereby reducing attribute confusion across instances.

Q2: Sensitivity of the Conclusions to the Reconstructor and Alignment

The human evaluation reported in Section 5.3 and Appendix A.9 (Table 10) does not use any reconstruction results or depth alignment; it relies entirely on human judgment of visible geometric faithfulness. The overall trend in human evaluation is also broadly consistent with the automatic metrics. In addition, as shown by the independent validation in Table R3, we verified on real Objectron data that reconstruction succeeds consistently and the resulting errors remain within a reasonable range, providing independent support for the reliability of the geometry measurement pipeline.

Q3: Benchmark Statistics and Occlusion

We have added detailed benchmark sizes, category distributions, object-count distributions for the training and evaluation sets, and the asset-separation mechanism used to prevent overlap in Section 1 of the General Response. In addition, our multi-object evaluation set contains up to three objects per scene, with no cases beyond three objects, and includes significant inter-instance occlusion.

Dear reviewer XySf, thank you again for your valuable feedback. We just wanted to briefly check in and see if our clarifications resolved your questions, or if there is anything else you would like to discuss further.

Thank you for the detailed response! I believe it addresses all the concerns I have for this paper.

Thank you for your positive feedback and support! We are glad that our response addressed all your concerns

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank all reviewers for their careful reading and constructive feedback. In response to these comments, we have added dataset statistics, ablation studies, additional baseline comparisons, and an independent validation of our geometric evaluation pipeline. We summarize these additions below.

1. Dataset and Benchmark Details

We agree that more detailed dataset statistics are important for assessing the coverage and reliability of the benchmark. In the initial submission, we could not provide sufficiently detailed statistics because the internal dataset details had to go through strict internal review. We now provide additional aggregate information to better clarify the coverage of multi-object and complex-scene cases.

Dataset statistics:

  • Training set: 13,862 pairs in total, including 3,522 real pairs and 10,340 synthetic pairs.
  • Category coverage: Our dataset covers humans, cartoon characters, robots, monsters, birds, marine animals, reptiles, vehicles, and other categories. More detailed category statistics and visualizations are provided in Appendix A.6 and Figure 6 of the revised supplementary material.
  • Evaluation set distribution (images / instances):
    • Reference-free generation: Single (63 / 63), Multi (86 / 201; 57 two-object cases and 29 three-object cases)
    • Reference-guided generation: Single (63 / 63), Multi (86 / 201; 57 two-object cases and 29 three-object cases)
    • Rigid editing: Single (30 / 30), Multi (27 / 64; 17 two-object cases and 10 three-object cases)
    • Arbitrary editing: Single (30 / 30), Multi (26 / 62; 16 two-object cases and 10 three-object cases)

All conditions in the evaluation set were manually constructed and verified in our editor and were not based on the training dataset. We also manually checked the evaluation assets to ensure no overlap with the training assets.

2. Representation and Fusion Ablation Under Matched Training Conditions

To address reviewer concerns about the practical benefit of the geometric representation and fusion strategy, we added a four-way comparison under matched conditions (same backbone, training data, and training budget), including a simple dual-map baseline:

  • Fixed Average: keep a fixed equal-weight average c = 0.5 (L_ext + L_int) throughout and fuse before the Transformer.
  • Joint Rendering: render the external pose and internal structure into a single control map with 0.5 opacity.
  • Sequence Concat: directly concatenate [L_ext; L_int] along the sequence dimension and let Transformer attention integrate them (doubling the number of control tokens).
  • Ours (learned fusion): first concatenate along channels, then project through a learned layer and fuse before the Transformer.

The results indicate that our learned fusion scheme achieves a good balance between geometric accuracy and image quality (Table R1). Compared with joint rendering, it preserves the two control maps more faithfully and thus reduces the information loss introduced by alpha blending. Compared with direct sequence concatenation, it attains comparable or better performance while using only half as many control tokens, leading to higher computational efficiency.

Table R1: Four-way ablation on representation and fusion (reference-free generation)

SchemeToken CountRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)PickScore (up)
Fixed AverageN98.131.6269.10.10620.19890.02430.87320.162
Joint RenderingN97.727.3872.90.10410.20400.02390.88520.372
Sequence Concat2N98.128.9372.60.10500.20270.02270.88920.310
OursN100.024.7774.60.10250.19410.02720.87320.314
AuthorsOpenReview ↗

General Response (2/2)

3. Data-Source Ablation Under Matched Data Size and Budget

To disentangle the effect of data source from that of data size and training budget, we compared training with real-only, synthetic-only, and mixed data under matched data size and matched training budget. The results indicate that the real-only model produces more natural image appearance but has substantially weaker geometric precision than the other two settings. The synthetic-only model achieves higher geometric accuracy but shows a stronger synthetic style in the generated results. We added qualitative examples in Figure 7 of the revised supplementary material (Appendix A.7) to make this difference more explicit. By contrast, the mixed model achieves a better balance between image quality and geometric accuracy, which further suggests that real and synthetic data are complementary for this task.

Table R2: Data-source ablation

Data SourceData SizeRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)Chamfer (down)[email protected] (up)PickScore (up)
Real-only3.5k98.131.5666.00.11870.22660.02800.83320.493
Synthetic-only3.5k98.925.1675.50.08190.18870.02030.90919.454
Mix3.5k100.024.9875.00.10150.18790.02400.87620.219

4. Independent Reconstruction Validation and Reliability of Geometric Evaluation

To further validate the reliability of our reconstruction-based geometric evaluation pipeline, we conducted an independent real-image validation on the Objectron dataset. Specifically:

  • Sampling: We sampled 50 videos from Objectron and randomly selected one frame from each video.
  • Evaluation pipeline: We used only the 2D ground-truth bounding boxes provided by Objectron as prompts to SAM 3, and then passed the results to SAM 3D for 3D reconstruction.
  • Alignment and metrics: We used the official Objectron annotations as ground truth, transformed both predictions and ground truth into the camera coordinate system, and applied the same focal-length correction and scale-alignment protocol as in our generated-image evaluation. As shown in Table R3, the reconstruction success rate and external geometric errors fall within a reasonable range, suggesting that the geometry measurement pipeline we rely on is reasonably reliable in real scenes.

Table R3: Independent reconstruction validation on Objectron

DataRec. (up)Ang.Err (down)[email protected] (up)Trans.Err (down)Scale.Err (down)
Objectron100.015.6087.90.08650.1973

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.