acceptodds
Under review as a conference paper at ICLR 2027

Depth-to-RGB: Repurposing a Frozen Depth Estimator for Geometry-Guided Compositing

Abstract

Reference-based object compositing inserts or replaces an object using a background image, a reference image, and a 2D compositing mask. These inputs guide appearance and placement but leave the completed scene's geometry implicit, which can distort object structure or alter the surroundings. Our Depth-to-RGB (D2R) framework predicts composite depth for a scene not yet observed in the RGB inputs. It learns reference-conditioned corrections to a frozen depth estimator using encoder features of paired completed scenes as targets. The unchanged decoder maps the corrected representation to the intended scene's depth, which a separately trained renderer holds fixed during RGB synthesis. Under matched architecture and training, encoder-feature supervision reduces OOD Stage-1 AbsRel by 31.4% relative to decoded-depth supervision. We also introduce AnyInsertion++ with paired in-distribution and category-disjoint splits to evaluate generalization beyond compositing training categories. The complete D2R framework leads 12 open-source and 3 closed-source baselines in estimator-derived geometry and photometric quality on both paired splits. On category-disjoint data, D2R reduces AbsRel by 43.7% and improves PSNR by 2.4 dB over the matched RGB baseline. Across three unpaired benchmarks, D2R leads both identity metrics and reduces mean CLIP reference cosine distance by 55% relative to the strongest baseline. We will publicly release code and AnyInsertion++ upon publication.

Primary area:
generative models
Keywords:
Reference-Based Object Compositing, Composite Depth Prediction, Geometry-Aware Image Generation, Depth-Conditioned Diffusion Models, Reference-Guided Image Editing, Image Inpainting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UEjZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Depth-to-RGB (D2R) for reference-based object compositing. D2R predicts completed-scene relative depth from a background image, reference image, and compositing mask using encoder-feature supervision, then uses this depth as a fixed condition for RGB synthesis. The paper also introduces AnyInsertion++ to evaluate generalization beyond compositing training categories.

Critical strengths
  1. Explicit geometric conditioning. D2R predicts the completed scene’s depth before RGB synthesis and uses it as a fixed geometric condition, explicitly addressing geometry that is typically left implicit in reference-based compositing.
  2. Extensive evaluation. Broad comparisons and ablation studies support the method’s effectiveness and generalization beyond compositing training categories.
Critical weaknesses
  1. It remains unclear whether D2R retains its geometric advantages with coarse masks rather than precise silhouettes, as users often provide only rough object outlines in practice.
  2. The paper lacks evaluation assessing foreground–background integration. Depth consistency and identity similarity do not sufficiently capture plausible contact, occlusion, lighting, or boundary transitions, leaving the perceptual quality of scene integration insufficiently validated.
Questions for authors
  1. Evaluate robustness to mask precision using silhouettes, dilated masks, and bounding boxes on the same paired cases.
  2. Provide a user study comparing spatial plausibility and appearance harmonization against baselines.
  3. Evaluate fine-grained spatial relationships in multi-object scenes, including contact and occlusion, while preserving non-target objects.
  4. Compare with the matched RGB-only baseline on geometrically simple scenes with challenging illumination to assess the benefit of depth guidance for harmonization.
6 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer UEjZ

We deeply appreciate Reviewer UEjZ's thoughtful review and the recognition of the explicit geometric conditioning and the extensive evaluation that supports the generalization beyond compositing training categories. In what follows, we address each of your comments, grouping related weaknesses and questions together. All changes are marked in blue in the revised manuscript.

# [W1/Q1] Robustness to Mask Precision: Silhouettes, Dilated Masks, and Boxes

With coarse masks, D2R has the highest identity scores of all 16 methods on three unpaired benchmarks, which provide only bounding boxes (Tab. 4). On the same paired cases, D2R also retains its geometric advantage: with the same bounding box, its OOD AbsRel (↓) is 52% lower than that of the matched RGB-only baseline (0.115 vs. 0.237; Fig. C4 (a)). For the paired evaluation, we change only the input mask on the 139 OOD cases of AnyInsertion++ and keep the background, the reference, and the paired target fixed (new App. C.3.5).

  • Coarse Masks on Unpaired Benchmarks. Three unpaired benchmarks in Tab. 4 (TF-ICON, AIComposer, and ComplexCompo) provide only box-shaped target masks, i.e., rough object outlines. With these boxes, D2R has the highest CLIP-I and DINO-I on all three (CLIP-I 0.902, 0.915, and 0.895 vs. 0.760, 0.816, and 0.778 for the matched RGB-only baseline). Figs. D3/D4/D5 show the composite results of D2R compared to state-of-the-art approaches.
  • Silhouettes vs. Bounding Boxes on the Same Paired Cases (Table R1). Replacing the precise silhouette with its bounding box degrades both methods: the AbsRel of D2R rises from 0.067 to 0.115, and that of the baseline ("RGB Only") from 0.133 to 0.237. With the same box, D2R is lower on 121 of the 139 cases (87%).
  • Dilated Masks (Table R1). At 3% dilation, inside the mask augmentation range of training (4%), the AbsRel of D2R stays below its value with the box (0.102 vs. 0.115). At 10% dilation, it grows to 0.205. In both cases, D2R is lower than "RGB Only" (0.170 and 0.290).
  • Mask Augmentation in Training. D2R trains with dataset masks (50%), eroded or dilated masks (25%), and bounding boxes (25%); the revised App. B.3.1 describes it.

Table R1 (values of Fig. C4). Mask Sensitivity on the OOD Split of AnyInsertion++ (AbsRel ↓).

Input MaskRGB OnlyDepth-to-RGB (D2R; Ours)
Mask (i.e., Silhouette)0.1330.067
Dilation 3%0.1700.102
Dilation 10%0.2900.205
Bounding Box0.2370.115

# [W2/Q2] Evaluation of Foreground–Background Integration and User Study

At the boundary transition between the object and the background, D2R has the lowest depth error of all 16 methods on both splits (Tab. C11). Contact and occlusion are evaluated under [Q3], lighting under [Q4], and the perceptual quality of scene integration in the user study below.

  • [W2] Boundary Transitions (Table R2). A full-image score is dominated by regions that barely change. Therefore, we also evaluate the boundary band, about 10 pixels to each side of the mask boundary (App. C.4.2 and Fig. C5). In the band, the AbsRel of D2R is 17% lower than that of the matched RGB-only baseline on the ID split (0.137 to 0.114) and 21% lower on the OOD split (0.189 to 0.149). These results measure geometric agreement; the perceptual judgment is left to the user study.
  • [Q2] User Study (App. C.4.5 and Fig. C6). Each screen shows the results of seven methods with hidden names in shuffled order. Every result is judged separately on spatial plausibility and on appearance harmonization. The cases are selected from the ID and OOD splits of AnyInsertion++ to cover diverse groups, including scenes with multiple objects, contact, and occlusion, and no case is repeated. The user study is in progress, and the final results are not yet available. Our target is 20 or more participants, and we aim to share the results by November 14 (AoE): the list of the evaluated cases, a plot of the selection rate of each method for each criterion, and the number of valid participants.

Table R2 (i.e., Tab. C11; AbsRel ↓ of 8 of 16 methods). Depth Error on the Full Image and Inside the Boundary Band on AnyInsertion++. ID = In-Distribution (287 cases), OOD = Category-Disjoint (139 cases).

MethodID Full ImageID Boundary BandOOD Full ImageOOD Boundary Band
MimicBrush (NeurIPS'24)0.0660.1640.0900.189
Insert-Anything* (AAAI'26)0.0620.1370.1190.189
SHINE (ICLR'26)0.1620.3810.2180.426
HiFi-Inpaint (CVPR'26)0.1040.2340.1330.261
DIRECT (ICML'26)0.0880.2690.1370.292
GPT-Image-2 (Tested 2026-08)0.2060.3620.3000.400
Nano Banana 2 (Tested 2026-08)0.1750.3970.2370.356
D2R (Predicted Depth)0.0480.1140.0670.149

(Continued in Next Response)

AuthorsOpenReview ↗

[2/2] Response to Reviewer UEjZ

(Continued from Previous Response)

# [Q3] Fine-Grained Spatial Relationships in Multi-Object Scenes (Contact and Occlusion) and Non-Target Objects

On scenes with multiple objects, contact, and occlusion, D2R has the lowest AbsRel and the highest PSNR of all 16 methods, and its AbsRel is 46% to 67% lower than that of the matched RGB-only baseline (Tab. C10). From the 139 OOD cases, we select three subgroups, which can overlap and together cover 66 cases: Multi-Object (two or more non-target objects next to the inserted object), Contact (the inserted object touches a non-target object other than its support or wearer), and Occlusion (a non-target object partly occludes it). The group labels use the target image and the target mask only (App. C.4.2 and rows five to seven and nine of Fig. D2).

  • Full-Image Results (Table R3). Against the matched RGB-only baseline (Insert-Anything*), the AbsRel of D2R is 46% lower on Multi-Object (0.149 to 0.080), 67% lower on Contact (0.224 to 0.073), and 61% lower on Occlusion (0.168 to 0.065). The baseline is worse in these subgroups than on the full split (0.119), whereas D2R stays close to its full-split error (0.067).
  • Boundary Band of Contact and Occlusion. Contact and occlusion occur at the boundary of the object. There, the AbsRel of D2R is 21% lower on Multi-Object (0.257 to 0.202), 30% lower on Contact (0.201 to 0.140), and 31% lower on Occlusion (0.188 to 0.129).
  • Preservation of Non-Target Objects. An inpainting backbone is conditioned on the input background and should reproduce it up to the encoding and decoding error of the VAE. On the background, which holds the non-target objects, D2R has the lowest AbsRel of all methods in all three subgroups (0.080, 0.074, and 0.065 vs. 0.158, 0.245, and 0.178 for the RGB-only baseline). The RGB-only baseline on the same backbone sometimes redraws this region (Fig. D7).
  • Human Judgment. These scores measure geometric agreement and do not label an individual contact or occlusion as correct. This perceptual judgment is left to the user study, which includes such scenes (App. C.4.5).

Table R3 (i.e., Tab. C10; 8 of 16 methods). Multi-Object Scenes, Contact, and Occlusion on the Category-Disjoint Split.

MethodMulti-Object(52)Contact(27)Occlusion(25)
AbsRel ↓PSNR ↑AbsRel ↓PSNR ↑AbsRel ↓PSNR ↑
MimicBrush (NeurIPS'24)0.10519.20.11019.90.08921.4
Insert-Anything* (AAAI'26)0.14920.10.22419.40.16820.7
SHINE (ICLR'26)0.21216.60.23517.10.22617.7
HiFi-Inpaint (CVPR'26)0.10018.80.11619.10.12019.3
DIRECT (ICML'26)0.15718.70.13719.10.11319.9
GPT-Image-2 (Tested 2026-08)0.37712.70.37712.80.31615.4
Nano Banana 2 (Tested 2026-08)0.29514.50.27314.40.23516.8
D2R (Predicted Depth)0.08022.80.07323.90.06524.9

# [Q4] Matched RGB-Only Baseline on Geometrically Simple Scenes with Challenging Illumination

Fig. D7 adds the requested matched comparison: in a low-light scene, "RGB Only" redraws the surroundings and keeps a bright exposure, whereas D2R preserves them and follows the local light. In a low-light room lit through one window, we fix the background and the bright reference and place the same bounding box on a dark surface (table) and on a bright surface (floor) (see App. D).

  • Observation (Fig. D7). "RGB Only" redraws the surroundings of the object (background hallucination): the table and the magazine on it change, and the object grows beyond the box. It also keeps a bright exposure on the dark table, which leaves a visible seam. With the composite depth, D2R keeps the object inside the box, preserves the non-target objects, and takes the exposure and color tone of each location.
  • Scope and Future Work. This example covers two placements in one scene, and depth guidance does not replace harmonization (App. C.5). Existing harmonization benchmarks (e.g., iHarmony4) provide an object that is already placed with plausible geometry, so they evaluate appearance only. Once a benchmark that covers both geometry and harmonization is available, we expect it to allow an accurate evaluation of illumination-aware compositing. We leave extending the depth condition with illumination to future work.

We hope these additional evaluations address your concerns. We will post the user study results in this thread once they are complete, and we welcome any further questions or requests in the meantime.

AuthorsOpenReview ↗

Follow-up to Reviewer UEjZ: User Study Results [Q2] and Revision Summary

We would like to kindly follow up on our response. The user study that you proposed [Q2] is now complete, and we share its results below. We acknowledge that depth and identity scores cannot capture whether people perceive an inserted object as part of the scene, and that perception determines whether a compositing method is useful in practice. After the results, we list where each point of your review is addressed in the revised paper.

# [Q2/W2] User Study on Spatial Plausibility and Appearance Harmonization

With 51 participants and 30 cases, D2R receives the highest selection rate on both criteria: 91.4% for spatial plausibility and 87.5% for appearance harmonization, compared with 80.5% and 79.2% for the previous state-of-the-art, matched RGB-only baseline Insert-Anything* (Table R4 and Fig. C7).

  • [Protocol] (App. C.4.5 and Fig. C6). Each case shows the results of seven methods with anonymized names in a randomized order for each participant. Since multiple outputs can be valid composites for a given scene, we use a multi-select setup: a participant marks every result that is spatially plausible and, separately, every result whose appearance blends with the scene. The selection rate of a method is the proportion of responses in which it is selected (1,530 answers per criterion).
  • [Cases] (Tab. C14). Since 30 cases already take a participant about 20 minutes, we evaluate a diverse subset instead of the full benchmarks, which keeps the study short and helps recruit more participants: 12 OOD and 9 ID cases from AnyInsertion++ (18 object categories) and 9 cases of three unpaired benchmarks. They include scenes with multiple objects, contact, and occlusion, and no case is repeated.
  • [Spatial Plausibility]. D2R (91.4%) and Insert-Anything* (80.5%) are marked far more often than other methods (9.3% to 26.0%).
  • [Appearance Harmonization]. D2R is marked in 87.5% of all responses, followed by Insert-Anything* (79.2%) and the two proprietary models (58.6% and 51.8%).

Table R4 (i.e., Fig. C7). Selection Rate (%) of Each Method in the Multi-Select User Study (51 Participants, 30 Cases), Sorted by Appearance Harmonization.

MethodSpatial Plausibility ↑Appearance Harmonization ↑
D2R (Predicted Depth)91.487.5
Insert-Anything* (AAAI'26)80.579.2
Nano Banana 2 (Tested 2026-08)17.558.6
GPT-Image-2 (Tested 2026-08)9.351.8
DIRECT (ICML'26)26.030.1
HiFi-Inpaint (CVPR'26)26.015.4
SHINE (ICLR'26)15.14.3

Also, we add the plot of the selection rates (Fig. C7) and the list of the 30 evaluated cases (Tab. C14), and we replace the placeholder sentence for the user study with the number of participants and the selection rates.


With this study, every evaluation requested in your review is in the revised manuscript: mask precision ([W1/Q1]; Fig. C4), boundary transitions ([W2]; Tab. C11), contact and occlusion ([Q3]; Tab. C10), challenging illumination ([Q4]; Fig. D7), and the user study ([Q2]; Fig. C7). Your review raised an important question of how a composite should be evaluated beyond quantitative depth and identity scores, and these results are our answer. We would be grateful for your view on whether they address your concerns, and on what a paper in this area should report so that later work can be compared fairly.

One study does not settle how scene integration should be evaluated. We hope that the two criteria you proposed, together with our curated dataset (i.e., AnyInsertion++) and the codebase that we will release, give future work a great starting point. If you see a comparison or a breakdown that would make this evaluation more useful to the research community, please let us know. We will do our best to add it within the discussion period.

Thank you again for the suggestion and for your time.

Reviewer_gK4BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a novel approach to compositing: given a background image and reference object appearance, generate a composited image what that object on that background in a natural, high-quality way. The proposed solution is to factorize the problem into two steps: first, generate the depth property of the composited image, and only in a second step render that into the appearance. The authors discuss how to implement the former through a minimal injector-layers approach on a pre-trained RGB-to-depth predictor. The latter is trained as a LoRA over a pre-trained generator. The resulting numbers and images

Critical strengths
  • the factorization is not intuitive to me (compared to joint Depth+RGB generation), but the authors show with experiments that the feedforward depth estimation with corrective injection layers is more efficient/accurate than a diffusion model predicting the same
  • numerical results are good and clearly improve over the baseline
  • visual results too, although I would strongly prefer more of them in the main paper (moving some tables to the appendix).
Critical weaknesses
  • related work is too succinct, and several important papers (e.g. 3DIS) are not even discussed in the main paper. They should at least be briefly discussed in the main paper.
  • I want a more direct discussion of the benefits/drawbacks compared to joint generation (and not just relegate everything to the appendix...)
  • in general, the paper is weak on motivation, and on clearly communication said argumentation/communication.
  • the factorization in integrals if functions (p(D|X)) are only ever dirac delta distributions is nonsensical.
  • I couldn't spot actual model details in the main manuscript about the base model for Stage 2 (depth-to-rgb renderer)
  • footnote 1 should just have been a proper reference instead.
  • the shown masks in the visualizations are extremely specific; the authors say that they want to avoid the user already having to pose the object in the scene, but e.g. the dog mask clearly does exactly that.
Questions for authors
  • I'm very interested in a more controlled baseline/ablation comparison between joint and sequential Depth/RGB predictions; the main argument (e.g. in image C3) that the depth is still blurry in earlier noisy timesteps of the joint process does not seem like such a huge issue to me, because it directly addresses one of the authors-identified weaknesses of stage 1 failures propagating towards stage 2. After all, any depth output is thrown away anyway, and all the user cares about is the final RGB.
  • what happens to the method if the masks are drastically simplified, e.g. simply axis-aligned bounding boxes of the current masks?
  • one thing not clearly discussed by the authors is the loss in a deterministic stage 1: there is no longer an option to e.g. sample multiple poses
5 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer gK4B

We are grateful to Reviewer gK4B for the detailed and constructive feedback, and we appreciate your positive assessment of both the efficient and accurate feed-forward injector and the numerical and visual results that clearly improve over the baseline. Below, we respond to your comments point-by-point, with related weaknesses and questions grouped under one subtitle. All changes are marked in blue in the revised manuscript.

# [Q1/W2] Joint (Depth & RGB) vs. Sequential (Depth-to-RGB) Prediction on the Final RGB

Thank you for pointing out that the user cares about the final RGB in the end; we compare the two designs on it. Measured on the final RGB, D2R is more accurate than joint denoising inside the object, where the geometry has to be created: its OOD AbsRel is 37% lower (0.081 to 0.051; Tab. C7). We evaluate the depth error of the final generated RGB separately on the background, whose geometry is given, and inside the object (App. C.3.4). The revised Sec. 5 states the benefits and drawbacks below directly in the main paper.

  • [Benefits of D2R] Background vs. Object (Table R1; Figs. C2 and C3). The two designs differ in where the geometry of the object comes from. Joint denoising receives only the background depth and has to create the geometry of the object while it generates RGB. On the background, whose geometry it is given, it lowers the OOD AbsRel of the RGB-only baseline by 40% (0.125 to 0.075). Inside the object, whose geometry it has to create, it brings no gain (0.081 vs. 0.073; the difference is not significant), and the object can leave the user's mask (Fig. C2) or lose its shape (Fig. C3). By contrast, D2R fixes the composite depth first and generates only RGB on top of it. D2R is clearly more accurate inside the object: its AbsRel is 30% lower than the baseline (0.073 to 0.051) and 37% lower than joint denoising (0.081 to 0.051). Inside the object, it has a lower error than joint denoising on 73% of the OOD cases (101 of 139) and on 75% of the rigid cases (89 of 119). Thus, the composite of D2R follows the region specified by the user more accurately. The additional cost is one Stage-1 pass of 56.5 ms, about 0.25% of the RGB generation time on FLUX.1-Fill (22.34 s; Tab. B4).
  • [Drawbacks of D2R] Repair of Stage-1 Failures by Joint Denoising (App. C.3.4). Stage-1 depth errors persist in the final RGB image, and each input yields one geometry. Therefore, joint denoising can help when the Stage-1 depth is wrong: D2R keeps the wrong depth fixed, whereas joint denoising can still change the geometry during sampling. However, such repairs are rare: even among the cases with the highest Stage-1 error, D2R remains more accurate on 74% (34 of 46).
  • Scope of the Comparison. The two variants share the backbone, the canvas, and the number of training iterations. They differ in the depth they receive (background depth vs. Stage-1 composite depth) and in their objectives, so the comparison does not isolate the generation order alone. Isolating it requires a variant that gives both models the same Stage-1 composite depth, which needs a new training run (App. C.3.4).

Table R1 (i.e., Tab. C7). Joint Denoising vs. Sequential Depth-to-RGB. Rows (a), (d), and (e) of Tab. 6 on the ID and OOD splits, evaluated on the foreground (inside the object mask) and on the background.

Split of AnyInsertion++MethodForeground AbsRel ↓Background AbsRel ↓
OOD (139)(a) Matched RGB Baseline0.0730.125
OOD (139)(d) Joint Denoising0.0810.075
OOD (139)(e) D2R (Ours)0.0510.068
ID (287)(a) Matched RGB Baseline0.0730.057
ID (287)(d) Joint Denoising0.0770.057
ID (287)(e) D2R (Ours)0.0530.043

# [Q3] Loss in a Deterministic Stage 1 (Sampling Multiple Poses)

Stage 1 returns one depth for a given background, reference, and mask, so changing the Stage-2 seed does not sample another pose, whereas joint (or stochastic) depth generation can. This loss matters most for non-rigid objects, which admit several poses inside one mask; for rigid objects, the fixed geometry preserves the structure. However, this loss does not appear as a loss of accuracy on the paired benchmark. On non-rigid objects, D2R lowers the OOD AbsRel of the matched RGB-only baseline by 47% (0.133 to 0.071; Tab. C3). Its gap to rendering with the depth of the paired target is also similar on non-rigid and rigid objects (0.071 vs. 0.060 and 0.067 vs. 0.059). The revision states this cost after Eq. (1), in Sec. 5, and in App. C.5. Predicting multiple plausible composite depths is a possible extension that requires additional modeling and evaluation.

(Continued in Next Response)

AuthorsOpenReview ↗

[2/2] Response to Reviewer gK4B

(Continued from Previous Response)

# [Q2/W7] Simplified Masks: Axis-Aligned Bounding Boxes

With bounding boxes as masks, D2R has the highest identity scores of all 16 methods on three unpaired benchmarks (Tab. 4). On the paired cases with each mask replaced by its box, D2R is also more accurate than the matched RGB-only baseline: its OOD AbsRel (↓) is 52% lower (0.115 vs. 0.237; Fig. C4(a)). Replacing the mask (i.e., silhouette) with its box lowers the accuracy of both methods, since a precise mask conveys much of the pose (W7). The revision states this in the limitations (Sec. 5 and App. C.5).

  • Bounding Boxes on Unpaired Benchmarks. In Tab. 4, three unpaired benchmarks (TF-ICON, AIComposer, and ComplexCompo) provide only box-shaped target masks. With these boxes, D2R has the highest CLIP-I and DINO-I on all three (CLIP-I 0.902, 0.915, and 0.895 vs. 0.760, 0.816, and 0.778 for the matched RGB-only baseline). Figs. D3/D4/D5 show the composites, in which D2R places the object inside a box that does not prescribe its pose.
  • Mask/Silhouette vs. Bounding Box on Paired Cases. On the 139 OOD cases of AnyInsertion++, we replace the mask (i.e., silhouette) with its bounding box and keep everything else fixed (new App. C.3.5). The AbsRel of D2R rises from 0.067 to 0.115, and that of the baseline ("RGB Only") from 0.133 to 0.237.
  • Erosion and Dilation. At 3% erosion and dilation, inside the mask augmentation range of training (4%), the AbsRel of D2R stays below its value with the box (0.071 and 0.102 vs. 0.115). At 10% dilation, it grows to 0.205. In all conditions, D2R has a lower AbsRel than "RGB Only" (Fig. C4 (b)).
  • Mask Augmentation in Training. The revised App. B.3.1 describes it: D2R trains with dataset masks (50%), eroded or dilated masks (25%), and bounding boxes (25%).

# [W1/W3/S3] Presentation

Thank you for these suggestions on the presentation. The revised manuscript discusses the geometry-first and joint RGB-depth methods in Sec. 2 and states the benefits and drawbacks of predicting depth first in Sec. 5. We would like to complete the remaining changes in the camera-ready version, because the main paper keeps the numbering of the submission during the discussion, so that the reviews can be checked against it.

  • [W1] Related Work (Important Papers Discussed in Sec. 2). The revised Sec. 2 discusses Compose-and-Conquer and 3DIS under "Geometry Before RGB Synthesis", and JointDiT and Modality Forcing under "Geometry During Sampling". Unlike these methods, D2R first predicts depth from a background, a reference, and a mask alone.
  • [W3] Motivation.
    • Problem: the inputs of compositing (a background, a reference, and a mask) do not contain the geometry of the result. One-shot RGB models have to infer it while they generate, and they miss the target geometry, alter the object identity, or modify the surrounding content (Fig. 1).
    • Solution: D2R predicts this geometry first and renders RGB on top of it.
    • vs. RGB-only generation: with the backbone and data matched, the OOD AbsRel falls from 0.119 to 0.067, and PSNR rises from 21.6 to 24.0 dB (Tab. 3).
    • vs. Joint Generation: inside the object, the OOD AbsRel of D2R is 37% lower (0.081 to 0.051; Tab. C7). With joint denoising, the object can leave the mask or lose its shape (Figs. C2/C3).
  • [S3] More Visual Results in the Main Paper. In the camera-ready version, we will move the ablation tables to the appendix and bring representative qualitative results from App. C/D into the main paper. We will compile them to preserve the diversity across benchmarks, and the new comparisons of the revision (Figs. C2/C3, C4, and D7) are candidates as well.

# [W4/W5/W6] Readability

All three points are fixed in the revised manuscript.

  • [W4] Integral Notation in Eq. (1). Thank you for this precise remark. The integral is removed. Eq. (1) writes the two stages as they are computed, D^=Φϕ(X)\hat{D}=\Phi_\phi(X) and I∼pθ(I∣D^,X)I\sim p_\theta(I\mid\hat{D},X), and the next sentence states that Stage 1 is deterministic and only Stage 2 samples.
  • [W5] Details of Stage 2. The method section includes the base setup for Stage 2: FLUX.1-Fill with Redux (Sec. 3.2 and Fig. 2). In the submission, we described the implementation details in Sec. 4 (lines 254 to 260), not in the method section. Because of the limited page length, the full setup is in the appendix (e.g., training and inference details in App. B.3, computational cost in App. B.4 and Tab. B4, and training and sampling settings in App. C.3.2).
  • [W6] Footnote 1. We replaced it with citations of FLUX.1-Fill and FLUX.1-Redux (line 255).

We hope the comparison with joint generation, the results with mask variants, and the revisions (marked in blue) in the manuscript address your concerns. If any point remains unclear, we would be happy to provide further clarification.

Reviewer_gK4BOpenReview ↗

Concerns addressed

Dear authors,

These answers have adequately addressed my remaining concerns. The visualizations and results around bounding boxes specifically have been very helpful.

Reviewer_DBjNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Depth-to-RGB (D2R) studies reference-based object compositing: inserting or replacing a reference object in a target scene given a background image, a reference image, and a target mask. The authors argue that these inputs do not explicitly specify the completed scene's geometry, and propose first predicting composite depth—the depth of the scene after compositing—and then generating RGB under this fixed depth condition. Stage 1 freezes the depth estimator's encoder and decoder, learns injectors to correct background features, and uses encoder features from the target composite as supervision to predict scene depth including the inserted object. Stage 2 arranges the background, reference, and depth images on a conditioning canvas and adapts an RGB editing model with LoRA, adding auxiliary depth-denoising and RGB–depth consistency supervision. The default configuration uses Depth Anything V2 and FLUX.1-Fill. Experiments on the in-distribution and category-disjoint splits of AnyInsertion++, as well as three unpaired compositing benchmarks, show improvements in geometry, image quality, and reference identity preservation.

Critical strengths
  1. The problem is clearly defined, and using geometry to guide object insertion is reasonably motivated. Compared with directly using depth from the background or reference image, predicting the completed scene's composite depth and using it to guide insertion makes intuitive sense.
Critical weaknesses
  1. I am not yet fully convinced that the additional composite-depth prediction stage is necessary. My intuition is that directly training an editing model to take a background, mask, and reference image and produce a plausible composite is already a viable approach. The paper demonstrates improvements over a matched RGB baseline, but these results do not yet make it clear to me whether the gains require this additional stage or how much of the advantage would remain with a stronger editing model.

    I would like to see comparisons with newer, stronger Qwen or FLUX editing models, while controlling the inputs, training data, and training budget as much as possible. If stronger direct-editing baselines can already handle object structure and placement well, the practical value of adding a separate depth-prediction stage needs to be made clearer. Based on the current results, I still have reservations about the incremental value over existing editing models and the necessity of the two-stage design.

Questions for authors
  1. Could you use stronger direct-editing baselines to further demonstrate why predicting composite depth first is worthwhile? With the same inputs and comparable training settings, I would like to see what consistent, clear benefits this stage provides over direct editing.
6 comments
AuthorsOpenReview ↗

[1/1] Response to Reviewer DBjN

We sincerely thank Reviewer DBjN for the careful review and for recognizing the clear problem formulation and the intuition of guiding insertion with the depth of the completed scene. Below, we address your concern. All changes are marked in blue in the revised manuscript.

# [W1/Q1] Controlled Comparison with State-of-the-art Editing Models (e.g., Qwen and FLUX)

Yes, predicting the composite depth first remains worthwhile on stronger editing models: with matched inputs, training data, and training settings, it lowers the OOD AbsRel by 41% on Qwen-Image-2.1-Turbo (0.129 to 0.076) and by 54% on FLUX.3-Klein (0.154 to 0.071; Tab. C6). We fine-tune each backbone for compositing in two ways, with RGB supervision only ("+ RGB Only") and with the complete D2R framework ("+ Depth-to-RGB"), and compare the two setups within each backbone (see App. C.3.3, Tab. C6, and Fig. C1).

  • Consistent, Clear Benefit (Table R1). On the OOD split of AnyInsertion++ (139 cases), all five metrics improve on all three backbones. PSNR rises by 2.4 dB on Qwen-Image-2.1-Turbo (20.4 to 22.8), by 3.8 dB on FLUX.3-Klein (19.3 to 23.1), and by 2.4 dB on FLUX.1-Fill (21.6 to 24.0), where AbsRel falls by 44% (0.119 to 0.067).
  • Controlled Inputs, Data, and Training Budget. The two rows share the backbone, inputs, training data, and training settings, including the number of iterations and the batch size. The only additional training of D2R is Stage 1, which is trained once (about 17.5 h; see App. C.2.1) and reused by every backbone. The two rows differ only in the Stage-1 depth condition and the depth objectives, so their difference measures the gain of the complete D2R framework. Supplying the depth without the depth objectives reaches only 0.108 on FLUX.1-Fill (row (b) of Tab. 7).
  • Remaining Gap of Stronger Direct-Editing Baselines. We fine-tune the two most recent open-source editors (released 2026-10; Qwen-Image-2.1-Turbo and FLUX.3-Klein), which are also the most accurate open-source editors on this benchmark (OOD AbsRel 0.167 and 0.271 vs. 0.699 for FLUX.1-Kontext-dev and 0.887 for Qwen-Image-Edit-2511; Tab. C6). RGB-only fine-tuning lowers their AbsRel to 0.129 and 0.154, which remains well above D2R on the same backbone (0.076 and 0.071).
  • Practical Value of the Additional Depth Stage. Stage 1 adds one feed-forward pass of 56.5 ms, about 0.25% of the RGB generation time on FLUX.1-Fill (56.5 ms vs. 22.34 s; Tab. B4). In return, the OOD AbsRel falls by 42% for rigid objects (0.116 to 0.067) and by 47% for non-rigid objects (0.133 to 0.071; Tab. C3).
  • Extension of D2R to Other Editing Tasks. Our evidence covers reference-based compositing, but the gain on editing backbones suggests that the idea may extend to other editing tasks: an edit that keeps the structure of the input (e.g., colorization or relighting) can read the depth from the input, whereas an edit that changes it (e.g., adding, removing, or moving an object) needs the depth of the result predicted first. We have not verified this direction and state it as future work (see App. C.5).

Table R1 (i.e., Tab. C6; 8 of 16 rows). RGB-Only vs. Depth-to-RGB with Editing and Inpainting Models on the OOD Split.

MethodParamsAbsRel ↓δ1\delta_1 ↑RMSE ↓PSNR ↑LPIPS ↓
Qwen-Image-2.1-Turbo (Released 2026-10)7B0.1670.8590.07418.70.270
Qwen-Image-2.1-Turbo + RGB Only7B0.1290.9030.05720.40.198
Qwen-Image-2.1-Turbo + Depth-to-RGB (D2R; Ours)7B0.0760.9610.04422.80.152
FLUX.3-Klein (Released 2026-10)6B0.2710.7770.09716.30.351
FLUX.3-Klein + RGB Only6B0.1540.8780.06919.30.251
FLUX.3-Klein + Depth-to-RGB (D2R; Ours)6B0.0710.9680.04223.10.138
FLUX.1-Fill + RGB Only (i.e., Insert-Anything*)12B0.1190.9140.04821.60.176
FLUX.1-Fill + Depth-to-RGB (D2R; Ours)12B0.0670.9700.03024.00.117

We hope this controlled comparison clarifies the value of predicting the composite depth first. We would be glad to answer any further questions or to provide additional analysis.

AuthorsOpenReview ↗

Follow-up to Reviewer DBjN: New User Study Results and Practical Value of Stage 1 [W1/Q1]

Following up on our previous response, we share new user study results and a unified summary of Stage 1's practical value below. You raised an essential question of what practical value a separate depth-prediction stage (e.g., for compositing into an empty background lacking prior object geometry) adds over direct editing. We consider this the central question for our manuscript, because it also determines what future work can build upon D2R (Depth-to-RGB). Below, we summarize the new and revised evidence on this question, and we would value your view on whether it answers it. Unless noted, the numbers are AbsRel (↓) on the OOD split of AnyInsertion++ (139 cases).

For just 0.25% (56.5 ms) of additional latency over RGB diffusion (GPU: A100/PRO 6000), D2R (Depth-to-RGB; Ours) lowers the depth error of RGB-only fine-tuning by 41% to 54% on three backbones and by 67% in scenes with contact, and people mark its results as spatially plausible more often (D2R: 91.4% vs. Prev. SOTA: 80.5%; Table R2).

Table R2. RGB Only vs. Depth-to-RGB: Cost, Depth/Geometry Error, and Human Judgment.

EvidenceRGB Only (Direct Baseline)D2R (Ours)Location in Our Paper
Latency on FLUX.1-Fill ↓22.34 s22.40 s (+0.06 s / +0.25%)Tab. B4
AbsRel ↓, Qwen-Image-2.1-Turbo0.1290.076 (-41%)Tab. C6
AbsRel ↓, FLUX.3-Klein0.1540.071 (-54%)Tab. C6
AbsRel ↓, FLUX.1-Fill0.1190.067 (-44%)Tab. C6
AbsRel ↓, FLUX.1-Fill, cases with contact (27)0.2240.073 (-67%)Tab. C10
AbsRel ↓, FLUX.1-Fill, cases with occlusion (25)0.1680.065 (-61%)Tab. C10
User study: Spatial plausibility ↑80.5%91.4% (+10.9%)Fig. C7
User study: Appearance harmonization ↑79.2%87.5% (+8.3%)Fig. C7

# Minimal Computational Overhead (Sec. 5; Tab. B4)

Stage 1 adds one feed-forward pass of 56.5 ms to an RGB generation of 22.34 s on FLUX.1-Fill, i.e., about 0.25% latency. It takes 5.21 GB of peak inference memory and 272.4M trainable parameters, is trained once, and is reused by every backbone.

# Benefit on Both Editing and Inpainting Models (Tab. C6)

With the same inputs, training data, number of iterations, and batch size, D2R lowers the AbsRel of RGB-only fine-tuning by 41% on Qwen-Image-2.1-Turbo, by 54% on FLUX.3-Klein, and by 44% on FLUX.1-Fill. PSNR, which does not depend on a depth estimator, rises by 2.4 dB, 3.8 dB, and 2.4 dB.

  • Editing. On the two most recent open-source editors (released 2026-10), RGB-only fine-tuning lowers the AbsRel from 0.167 and 0.271 to 0.129 and 0.154, which remains well above D2R on the same backbone.
  • Inpainting. FLUX.1-Fill gives the strongest RGB-only baseline of the three (0.119). Against it, D2R is lower on 103 of the 139 cases (Tab. C8), and the gain is largest where the object interacts with the scene: 67% with contact and 61% with occlusion (Tab. C10).
  • Depth Evaluators. D2R ranks first among the sixteen methods under all five depth estimators on both splits (Tabs. C12 and C13).

# Human Preference (Multi-Select User Study; App. C.4.5)

With 51 participants and 30 cases, D2R is selected as spatially plausible in 91.4% of the answers and the matched RGB-only (previous state-of-the-art) baseline in 80.5%; for appearance harmonization, the rates are 87.5% and 79.2% (Fig. C7). Both methods use FLUX.1-Fill and differ only in the Stage-1 depth condition and the depth objectives. This confirms that even after fine-tuning on the same data, RGB-only editing still falls short of producing spatially plausible composites in human judgment, whereas D2R achieves the highest selection rate among all seven methods on both criteria.

We appreciate this constructive discussion, which directly tests whether predicting composite depth first is a design worth adopting for future methods. To enable the community to verify and extend these findings, we will release both our curated dataset (i.e., AnyInsertion++) and the complete codebase. If any questions remain regarding the necessity of Stage 1, or if there is a specific experiment that would be decisive for your assessment, we would be glad to conduct it while the discussion remains open.

Thank you once more for your thoughtful review and time.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer DBjN: Our Response on Stage 1 and New User Study [W1/Q1]

We appreciate your careful review, particularly your question about the practical value of a separate depth-prediction stage (e.g., when the background does not provide the inserted object's geometry) over direct editing [W1/Q1]. This question helped us make the evidence for our design more concrete.

Before the discussion ends, we kindly ask you to review our response and follow-up on this point. The revised manuscript includes controlled comparisons with stronger Qwen and FLUX backbones (released 2026-10): D2R lowers AbsRel by 41-54% across three backbones on the OOD split of AnyInsertion++ compared with matched RGB-only fine-tuning (Tab. C6).

On FLUX.1-Fill, Stage 1 adds 56.5 ms, about 0.25% of the RGB generation time (Tab. B4), while D2R lowers OOD AbsRel by 67% in contact scenes compared with the matched RGB-only baseline (Tab. C10). In our new multi-select user study, D2R also achieves a higher spatial-plausibility selection rate than the matched RGB-only baseline Insert-Anything* (91.4% vs. 80.5%; Fig. C7).

A short reply letting us know whether this evidence answers your question, or what remains unresolved, would be very valuable to us. We will respond promptly to any remaining concern. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chairs and Reviewers,

We sincerely appreciate the time and effort that the Area Chairs and all reviewers have devoted to our submission, as well as the insightful comments that helped us improve it. We are pleased that the reviews acknowledge the idea of predicting the depth of the completed scene and using it as an explicit geometric condition (Reviewers DBjN and UEjZ), the clearly defined problem (Reviewer DBjN), the efficient and accurate feed-forward injector (Reviewer gK4B), and the extensive evaluation with numerical and visual results that clearly improve over the baseline (Reviewers gK4B and UEjZ).

Following the reviews, we revised the manuscript with new experiments and clarifications. The main paper keeps the numbering of the initial submission, and the new results are placed in the appendix. Every change is marked in blue. The key updates are listed below:

1. Controlled Comparisons on Predicting Depth Before RGB

  • [Appendix C.3.3; Tab. C6, Fig. C1] Recent Editing Models (Reviewer DBjN): Compared with RGB-only fine-tuning ("RGB Only") under matched inputs, training data, and training settings, D2R lowers the AbsRel (↓) on the category-disjoint (OOD) split by 41% with Qwen-Image-2.1-Turbo (0.129 to 0.076) and by 54% with FLUX.3-Klein (0.154 to 0.071).
  • [Sec. 5, Appendix C.3.4; Tab. C7, Figs. C2/C3] Joint vs. Sequential Generation (Reviewer gK4B): Measured on the final RGB, the OOD AbsRel (↓) of D2R inside the object, where the geometry has to be created, is 37% lower than that of joint denoising (0.081 to 0.051). On the background, joint denoising improves over "RGB Only" (0.125 to 0.075), and D2R remains the most accurate (0.068). We also report the cases in which joint denoising repairs a Stage-1 failure.

2. Robustness to Mask Precision and Scene Integration

  • [Appendix C.3.5, B.3.1; Fig. C4] Silhouettes, Dilated Masks, and Bounding Boxes (Reviewers gK4B and UEjZ): On the same paired cases, a bounding box raises the OOD AbsRel (↓) of both methods (D2R: 0.067 to 0.115), and the AbsRel of D2R remains 52% lower than that of "RGB Only" (0.115 vs. 0.237). The unpaired benchmarks in Tab. 4 of the initial submission already provide only bounding boxes.
  • [Appendix C.4.2; Tabs. C10/C11, Fig. C5] Boundary Transitions, Contact, and Occlusion (Reviewer UEjZ): D2R has the lowest boundary-band depth error of all 16 methods on both the in-distribution (ID) and OOD splits (AbsRel 0.114 and 0.149). On multi-object, contact, and occlusion scenes, its AbsRel (↓) is 46% to 67% lower than that of "RGB Only".
  • [Appendix C.4.5; Fig. C6] User Study (Reviewer UEjZ): The user study is in progress: seven methods are judged on spatial plausibility and appearance harmonization. Our target is 20 or more participants, and we aim to share the results by November 14 (AoE).
  • [Appendix D, C.5; Fig. D7] Challenging Illumination (Reviewer UEjZ): In a matched comparison on a low-light scene, "RGB Only" redraws the surroundings, whereas D2R preserves them. Geometric guidance is complementary to photometric harmonization, and we state illumination-aware compositing as future work.

3. Clarifications

  • [Sec. 2] Related Work (Reviewer gK4B): We discuss important geometry-first and joint RGB-depth papers (e.g., Compose-and-Conquer, 3DIS, JointDiT, and Modality Forcing), which the initial submission discussed mainly in the appendix.
  • [Sec. 3, Sec. 3.2, Sec. 4; Fig. 2] Formulation and Details of Stage 2 (Reviewer gK4B): Eq. (1) writes the two stages as they are computed, without the integral. The method section and Fig. 2 name the base model (FLUX.1-Fill with Redux), which the initial submission stated only in Sec. 4. Citations replace footnote 1.
  • [Sec. 3, Sec. 5, Appendix C.5] Deterministic Stage 1 and Limitations (Reviewer gK4B): We state that Stage 1 returns one depth for each input, so it does not sample multiple poses for non-rigid objects, and that a precise mask conveys much of the pose, which reduces this ambiguity.
  • Presentation (Reviewer gK4B): We will move the ablation tables to the appendix and bring representative qualitative results into the main paper in the camera-ready version, so that the numbering stays unchanged during the discussion.

We trust that this summary and the individual responses make the changes easy to verify. Thank you again for your time, and we remain available for any further questions during the discussion.

Sincerely,

Authors of Submission 31598

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.