acceptodds
Under review as a conference paper at ICLR 2027

Controllable Multi-Object Insertion for Detection Data Augmentation

Abstract

Controllable object insertion provides a targeted approach to augmenting object detection datasets that require category–count fidelity, annotation alignment, and source-content preservation beyond visual realism. Several approaches have been proposed to address this grand challenge, but most suffer from different constraints in precise category–count requests and visual consistency, limited coordination across categories and instances, and poor localization of multiple objects of specific category and count. We design SceneP2F, a two-stage trainable framework that connects scene-aware joint planning with layout-aligned generation leveraging explicit instance-level layouts. SceneP2F consists of two novel designs. The first is ScenePlan which allocates one class-labeled query per requested instance, jointly predicts scene-compatible object centers, and separately estimates box sizes conditioned on the planned centers and scene context. The second is SceneAlign-Fill which combines geometry- and appearance-aware conditioning, box–class-guided regional attention, and an annotation-region-weighted flow-matching objective to render the planned objects while limiting unintended changes to source content. Extensive experiments on three challenging benchmarks UAVDT, VisDrone, and BDD100K show that SceneP2F outperforms representative general-purpose editors and object-insertion methods consistently, demonstrating the value of coordinating instance-level planning and generation for category-targeted detection augmentation.

Primary area:
generative models
Keywords:
Image Editing, Object Insertion, Object Detection, Data Augmentation, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8AasOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SceneP2F, a two-stage framework for controllable multi-object insertion to augment object detection datasets. The framework consists of ScenePlan, which predicts scene-compatible object layouts according to category and count requirements, and SceneAlign-Fill, which synthesizes objects within the planned regions while maintaining visual consistency and preserving existing scene content. Experiments on UAVDT, VisDrone, and BDD100K demonstrate improvements in category-count fidelity, visual consistency, and background preservation over several existing methods. The authors further demonstrate the effectiveness of the generated data for improving the detection performance of underrepresented categories.

Critical strengths
  1. The paper addresses an important problem in object detection data augmentation, particularly the controllable insertion of multiple objects with specified categories and counts.

  2. Well-designed framework. The proposed two-stage architecture effectively connects scene-aware layout planning with instance-level image generation. The integration of geometry-aware conditioning, regional attention, and appearance guidance is reasonable and technically meaningful.

  3. Promising experimental results. The method demonstrates competitive performance across three benchmarks, with clear improvements in request fidelity and scene preservation. The downstream detection experiments further support the practical value of the proposed framework.

Critical weaknesses
  1. The methodological novelty could be further clarified. Several components, including diffusion-based layout planning, region-aware attention, and weighted generation objectives, are closely related to existing techniques. The authors should better distinguish their key contributions from prior methods and provide stronger evidence for the necessity of the proposed design choices.

  2. The absolute accuracy of category-count control remains limited. Although SceneP2F consistently outperforms competing methods, Figure 6 indicates that the exact-count accuracy remains relatively low, particularly for larger insertion requests. Further analysis of failure cases and the discrepancy between planned and successfully generated instances would strengthen the paper.

  3. The evaluation relies heavily on detector-based measurements. The category-count metrics depend on GroundingDINO predictions, which may introduce errors, especially for small or challenging objects. Additional human-verified evaluations or direct measurements of generated object and bounding-box alignment would make the conclusions more convincing.

  4. The comparison with existing methods could be strengthened. Different approaches use different conditioning mechanisms and training configurations, making it difficult to fully isolate the advantages of the proposed framework. More controlled comparisons with layout-conditioned generation methods would help establish the effectiveness of the individual contributions.

  5. The downstream detection evaluation is relatively limited. The augmentation experiments are conducted only on UAVDT with YOLO11-L and two underrepresented categories. Extending the evaluation to additional datasets, detection architectures, and competing generative augmentation methods would provide stronger evidence of generalization.

Questions for authors
  1. How frequently do the planned object layouts correspond to correctly generated instances with accurate categories and bounding boxes, particularly for larger insertion counts?

  2. Could the authors provide additional computational cost analysis and clarify the relative contributions of layout planning, post-processing, and image generation to the final improvements?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8Aas (1)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful and insightful comments!To address your concerns, we provide detailed responses and additional experimental results below.

W1: Distinguishing contribution from prior methods and corresponding evidence

To clarify our main contribution: we propose SceneP2F, a plan-to-fill framework that aims to balances the four requirements of this task: explicit annotations, request fidelity, source preservation, and scene-consistent appearance in multi-object insertion task. The table below compares its capabilities with those of related methods.

MethodExplicit class-labeled boxJoint multi-object insertionObject and background preservationScene-consistent appearance without reference image
GPT-Image-2.5 / FLUX.2 [dev] / Qwen-Image-Edit-2511✗△¹△²✗
Add-it✗✗✓✓
Add-SD✗✗△³✓
Diffree✗✗✓✓
Scene-aware location modeling✓✗✓✓
PDA++✓✗✓✗
SceneP2F (ours)✓✓✓✓

✓: explicit mechanism supported; △: partial or implicit support; ✗: not supported or no public evidence.

¹ Single-run multi-object generation, without instance-specific conditioning. ² No documented preservation operation in the evaluated configuration. ³ Learned from background-consistent pairs; no masking at inference.

How SceneP2F realizes each capability, and the supporting evidence:

Explicit class-labeled box. ScenePlan outputs a set of class-labeled boxes (Eq. 1), one per requested instance. The same boxes serve as the input of SceneAlign-Fill and as the final annotations. This capability makes the outputs directly usable for detection data augmentation. Evidence: using the planned boxes directly as labels improves bus and truck AP50 from 22.9 to 31.0 and from 7.2 to 12.2 (Table 2).

Joint multi-object insertion. In planning, all target centers are denoised jointly (Eq. 3), with self-attention among target tokens and a set-level matching loss (Eq. 4), so the requested categories and counts are fixed at the layout level. In inpainting, all boxes are filled in a single pass: each image token inside a box attends to that box's token, and a dense geometry field encodes each token's position relative to its nearest planned box (App. D). Evidence: on requests with N = 1–5 objects, SceneP2F reduces mTCE from 2.744 to 0.817 on UAVDT and from 5.523 to 1.924 on BDD100K relative to the best baseline (Table 1); with all boxes filled in a single pass, SceneAlign-Fill reaches mTCE 0.77, vs. 1.68–2.12 for pretrained inpainters (Table 3).

Object and background preservation. Preservation follows from the plan-to-fill design itself: ScenePlan fixes a box layout that against overlaps with existing objects (App. C.2), and SceneAlign-Fill generates only within the masked boxes, conditioned on the unmasked source latent. The flow-matching loss (Eq. 12) keeps full supervision outside the boxes. Evidence: Outside PSNR improves by 5.49, 3.94, and 3.33 dB over the best baseline on UAVDT, VisDrone, and BDD100K (Table 1).

Scene-consistent appearance without reference image. In SceneAlign-Fill, a style token is derived only from existing objects in the same image (App. D), on top of the pretrained FLUX.1-Fill prior, so no reference image is needed. Evidence: removing the style token degrades all three in-box metrics (ST0, Table 4); SceneP2F obtains the best DINO similarity and CLIP-I on all datasets (Table 1) and the best LPIPS, DINO similarity, and CLIP-I among inpainters (Table 3).

AuthorsOpenReview ↗

Response to Reviewer 8Aas (2)

Q2: Computational cost and component-wise contribution analysis

Component contributions. To clarify contribution from each component, we replace one component of SceneP2F at a time and evaluate on 200 test requests of UAVDT. Best results are underlined.

VariantLayoutPost-proc.GenerationmTCE ↓mCCE ↓LPIPS ↓DINO sim ↑CLIP-I ↑
Replace allData prior✗FLUX.1-Fill-FT1.801.210.520.440.82
Replace plannerData prior✓SceneAlign-Fill1.491.170.510.480.83
Remove post-processingScenePlan✗SceneAlign-Fill1.251.16‾\underline{1.16}0.480.490.86‾\underline{0.86}
Replace rendererScenePlan✓FLUX.1-Fill-FT0.80‾\underline{0.80}1.190.480.490.85
Full (SceneP2F)ScenePlan✓SceneAlign-Fill0.80‾\underline{0.80}1.16‾\underline{1.16}0.47‾\underline{0.47}0.50‾\underline{0.50}0.86‾\underline{0.86}

Among our components, ScenePlan (layout-planning) contributes most, followed by post-processing and then SceneAlign-Fill (generation). Compared with a dataset-prior layout, ScenePlan reduces the mean total count error from 1.49 to 0.80 and improves all three appearance metrics. Post-processing further reduces this error from 1.25 to 0.80, with little change in other metrics. Compared with FLUX.1-Fill-FT, trained on the same data with the same objective, SceneAlign-Fill keeps the total count error unchanged but reduces mCCE (1.19 → 1.16), showing that the instance-aware tokens improve per-class object binding; it also improves gives a small but consistent gain in this end-to-end setting. Outside-preservation metrics, omitted for brevity, remain nearly identical across all variants (PSNR ≈ 27.5 dB, SSIM ≈ 0.96)

The appearance differences are small in absolute terms, partly because objects inserted in UAVDT are small and fairly uniform in size, so crop-level metrics vary little across variants. Still, the full model is best or tied for best on all five metrics, while replacing all components yields the worst result on every metric, supporting SceneP2F as a framework that balances explicit annotations, request fidelity, source preservation, and scene-consistent appearance.

Computational cost. All SceneP2F components are trained on a single A6000, each within 40 GPU-hours except the BDD100K size predictor (about 140 GPU-hours, due to the larger training set). At inference, joint box planning takes about 1 s per image, while FLUX-based refinement dominates the cost at about 16–17 s per image. Since augmentation is performed offline once, this cost adds no overhead to detector inference.

AuthorsOpenReview ↗

Response to Reviewer 8Aas (3)

W2 & W3 & Q1: The discrepancy between planned and generated instances and failure analysis

Why the absolute values in Figure 6 are low. Two factors lower the absolute values. First, the metric is all-or-nothing: a single missed, extra, or mismatched object makes the whole sample count as a failure. Second, inserted objects are identified by a detector GroundingDINO is imperfect, and we conduct human evaluation (see our response to W3 below) confirms this. The metric therefore reflects detection and matching errors in addition to count control, especially in the dense, small-object scenes of VisDrone and the distant, occluded objects of BDD100K.

Planned boxes vs. generated objects. We analyzed the inference results on UAVDT by request quantity N (200 images per N).

NValid layout (planning)Count accuracy (generation)Average IoU (alignment)RMSE, valid boxes vs. generated objects
189.00%66.00%0.80160.6928
275.00%57.50%0.79141.0075
362.00%48.00%0.78051.3435
456.00%33.00%0.78241.8125
543.50%39.50%0.77822.0869
All65.10%48.80%0.78391.4792

Valid layout: an image whose planned boxes all lie at least 80% inside the SAM3 valid region and overlap other boxes by IoU less than 0.2. Count accuracy: the fraction of images whose detected number of inserted objects equals the request. Average IoU: between planned boxes and their matched detected objects. RMSE: between the numbers of valid planned boxes and of detected inserted objects per image.

Alignment between planning and filling. When a planned box is matched to a generated object, the two agree well in position and size: the average IoU is about 0.78 and stable across N. The discrepancy lies in the number and category of objects rather than in their geometry: the RMSE grows from 0.69 at N = 1 to 2.09 at N = 5, although the error per requested object does not increase (0.69 to 0.42). Generation accuracy also degrades far more slowly than independent per-object errors would imply: 39.5% at N = 5, versus 12.5% if each object failed independently at the N = 1 rate.

Failure analysis. The two stages limit performance in different regimes. At N = 1, 89.0% of layouts are valid but 66.0% of requests are fulfilled, so at least two thirds of the failures occur with a valid layout and stem from the fill stage or the detector. As N grows, the share of fully valid layouts falls from 89.0% to 43.5%, faster than count accuracy (66.0% to 39.5%), so planning becomes the increasingly important factor for larger requests. On the fill side, there are two possible reasons: UAV imagery contains many very small objects, which are hard to render faithfully, and the fill model is likely weaker on long-tail categories. We will add this analysis to the paper and state these limitations explicitly.

W3: Reliance on detector-based measurements

We agree that detector errors can affect the count metrics. However, since all methods are evaluated with the same detector, relative comparisons remain reliable, and we additionally conduct a human evaluation as a reference.

Relative comparison. All methods are evaluated with the same detector, thresholds, and matching rule, so detector errors apply to every method alike. While the absolute values are affected, the relative differences between methods under this shared protocol remain informative.

Human evaluation. To verify the detector-based results, we conduct a human evaluation on one dataset per domain as a reference: UAVDT for aerial scenes and BDD100K for driving scenes, with 100 randomly sampled SceneP2F outputs each. Two volunteers (50 images each) annotated the number and categories of newly inserted objects, and we compare the resulting metrics with the detector-based ones on the same images.

DatasetmTCE (human)mTCE (detector)mCCE (human)mCCE (detector)
UAVDT0.560.980.831.27
BDD100K1.382.051.071.97

The human and detector protocols show a consistent trend : on both metrics, they rank the two datasets in the same order, so the detector-based metrics reflect the same relative behaviour as human judgment. The human-based errors are lower in absolute terms (by 33–43% in mTCE and 35–46% in mCCE), which we attribute to two factors. First, in cluttered scenes, detections of existing objects are unstable between the source and edited images, and the unmatched detections are counted as insertions. Second, correctly inserted objects that overlap their neighbours or are very small are often missed or merged by the detector.

Reviewer_co1eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors propose SceneP2F, a two-stage framework for scene-aware, multi-object insertion for the purposes of data augmentation for generating synthetic data. The authors claim that this framework is controllable in that one can request category and scene counts and the architecture places them in the scene while preserving the source content. There are two stages to this framework. The ScenePlan jointly predicts scene compatible object centers and separately estimates box sizes conditioned on the planned centers and scene context. The SceneAlign-Fill module uses geometry and appearance-aware conditioning, box–class-guided regional attention, and an annotation-region-weighted flow-matching objective to generate objects within the planned regions while preserving the source content. The authors evaluate the method on three object detection benchmarks, UAVDT, VisDrone, and BDD100K and compare their method against other general purpose image generators including ChatGPT and other object insertion methods. The authors show performance imporvement over several metrics including , particularly mean total count error (mTCE), mean class count error (mCCE), and outside preservation.

Critical strengths

The problem is important and with growing number of image generation tools, there is a need for maintaining precise control over object insertion and placement. The issues that the paper addresses, cateogy count fidelity, annottion alignment, and source-content preservation still don't have satisfactory solutions (see weaknesses).

The two-stage framework connecting scene-aware joint planning with layout-aligned generation is reasonable. The use of explicit instance-level layouts provides a way to coordinate scne-compatible object placement and generation while maintaining the requested categories and counts.

The experimental evaluation is extensive, covering UAVDT, VisDrone, and BDD100K. The results in Table 1 show that SceneP2F achieves the lowest mTCE and mCCE among the evaluated methods, along with improvements in visual consistency and outside preservation.

The ablation studies are sound and are designed to evaluate size prediction, attention, geometry and appearance etc.

The authors also show results on underrepresented categories and show improvements in performance.

Critical weaknesses

The novelty of the paper was not immediately obvious. The proposed framework combines scene-aware joint planning with layout-aligned generation for category–count-controlled multi-object insertion. However, several related approaches already use layout planning, bounding-box-conditioned generation, and scene-aware object insertion. While the experimental results demonstrate improvements over the selected baselines, it is unclear which aspects of the proposed framework constitute the main methodological contribution relative to these existing approaches. A more direct comparison with scene-aware location modeling and other planning-based object-insertion methods would help establish the novelty and benefits of SceneP2F.

One of the main point of concern that the mean count accuracy was quite low. Looking at Figure 6, the best values (for SceneP2F) range from 0.17 to 0.31. The authors specifically set out to solve the problem of category count control, however, the results don't align with the central theme of the paper.

The geometry token only encodes the box position and size. So it is really the bounding box information and not really the geometry of the object. These boxes seem that they are horizontal/vertical. There is not information about object orientation (especially vehicles, traveling left to right, turning or aerial scenes). This is information about the planned box, not about the geometry of the object, such as its shape or orientation. Since the boxes are axis-aligned, the token also provides no information about object orientation, which seems important for vehicles in aerial scenes. Further, the ablation result for the gometry token show a minor change. Thus the authors should justify whether SCENEALIGN-FILL can be described as having geometry aware conditioning.

The paper uses a packed inpainting mask feature in Equataion (6). They don't describe how it is generated.

This is a minor point, but does SceneP2F mean Plan to Fill? If so, please specify this explicitly.

Questions for authors

The authors mention a box token (Box-token regional attention) in Equation 7, where the normalized position and the scale of the box is represented. What information does the geometry token provide that is not already contained in the inpainting mask and the box tokens?

In Section 3.3, the authors introduce an inpainting-mask feature tau_mp in Equation (6). However, it is unclear how the inpainting mask is constructed. Is it obtained directly from the predicted bounding boxes, or is an additional segmentation step performed?

What does SceneP2F accomplish that a scene-aware bounding-box predictor followed by a pretrained, box-conditioned inpainting model cannot? I.e. can one design some experimental conditions that can directly test this question.

5 comments
AuthorsOpenReview ↗

Response to Reveiwer co1e (1)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful and insightful comments!To address your concerns, we provide detailed responses and additional experimental results below.

W1 & Q3: Main contribution relative to existing methods

For our target task, multi-object insertion for detection data augmentation, we propose SceneP2F, a plan-to-fill framework that aims to balances the four requirements of this task: explicit annotations, request fidelity, source preservation, and scene-consistent appearance. The table below compares its capabilities with those of related methods.

MethodExplicit class-labeled boxJoint multi-object insertionObject and background preservationScene-consistent appearance without reference image
GPT-Image-2.5 / FLUX.2 [dev] / Qwen-Image-Edit-2511✗△¹△²✗
Add-it✗✗✓✓
Add-SD✗✗△³✓
Diffree✗✗✓✓
Scene-aware location modeling✓✗✓✓
PDA++✓✗✓✗
SceneP2F (ours)✓✓✓✓

✓: explicit mechanism supported; △: partial or implicit support; ✗: not supported or no public evidence.

¹ Single-run multi-object generation, without instance-specific conditioning. ² No documented preservation operation in the evaluated configuration. ³ Learned from background-consistent pairs; no masking at inference.

How SceneP2F realizes each capability, and the supporting evidence:

Explicit class-labeled box. ScenePlan outputs a set of class-labeled boxes (Eq. 1), one per requested instance. The same boxes serve as the input of SceneAlign-Fill and as the final annotations. This capability makes the outputs directly usable for detection data augmentation. Evidence: using the planned boxes directly as labels improves bus and truck AP50 from 22.9 to 31.0 and from 7.2 to 12.2 (Table 2).

Joint multi-object insertion. In planning, all target centers are denoised jointly (Eq. 3), with self-attention among target tokens and a set-level matching loss (Eq. 4), so the requested categories and counts are fixed at the layout level. In inpainting, all boxes are filled in a single pass: each image token inside a box attends to that box's token, and a dense geometry field encodes each token's position relative to its nearest planned box (App. D). Evidence: on requests with N = 1–5 objects, SceneP2F reduces mTCE from 2.744 to 0.817 on UAVDT and from 5.523 to 1.924 on BDD100K relative to the best baseline (Table 1); with all boxes filled in a single pass, SceneAlign-Fill reaches mTCE 0.77, vs. 1.68–2.12 for pretrained inpainters (Table 3).

Object and background preservation. Preservation follows from the plan-to-fill design itself: ScenePlan fixes a box layout that against overlaps with existing objects (App. C.2), and SceneAlign-Fill generates only within the masked boxes, conditioned on the unmasked source latent. The flow-matching loss (Eq. 12) keeps full supervision outside the boxes. Evidence: Outside PSNR improves by 5.49, 3.94, and 3.33 dB over the best baseline on UAVDT, VisDrone, and BDD100K (Table 1).

Scene-consistent appearance without reference image. In SceneAlign-Fill, a style token is derived only from existing objects in the same image (App. D), on top of the pretrained FLUX.1-Fill prior, so no reference image is needed. Evidence: removing the style token degrades all three in-box metrics (ST0, Table 4); SceneP2F obtains the best DINO similarity and CLIP-I on all datasets (Table 1) and the best LPIPS, DINO similarity, and CLIP-I among inpainters (Table 3).

A direct comparison with planning-based methods is given in our response to W1 & Q3 below.

AuthorsOpenReview ↗

Response to Reviewer co1e (2)

W1 & Q3: Direct comparison with planning-based insertion methods and a controlled test

Limited comparability of planning-based object insertion methods. Our task is joint multi-object insertion: given a request of categories and counts, all objects are planned and rendered together in an existing image. Among related planning-based methods, PDA++ requires a reference object image with a pixel-level mask, and Imagining the Unseen predicts the location of a single object, so neither can be run directly in our setting. The closest is the scene-aware location model, which needs only the scene and the requested categories, and can place several objects per image. However, its box planning is statistical and samples objects independently, and its fill stage generates each object in a separate image patch and composites it back, so neither stage is joint.

A controlled test with scene-aware location modeling. This also makes it the most suitable instance of the pipeline described in Q3, i.e., a scene-aware box predictor followed by a box-conditioned inpainting model. Since its code is not released, we re-implemented it in our setting and compare it with SceneP2F, examining planning ability and inference efficiency.

Planning ability. We fix the fill model to FLUX.1-Fill-FT for fair comparison, fine-tuned on the same data as ours, and change only the layout source on 200 UAVDT requests. Best results are underlined.

LayoutmTCE ↓mCCE ↓LPIPS ↓DINO similarity ↑CLIP-I ↑
Scene-aware1.671.12‾\underline{1.12}0.530.450.83
ScenePlan0.80‾\underline{0.80}1.190.48‾\underline{0.48}0.49‾\underline{0.49}0.85‾\underline{0.85}

With the same fill model, ScenePlan layouts reduce the mean total count error from 1.67 to 0.80 and improve all three appearance metrics, while the class-wise count error is slight. The gain in request fidelity therefore comes from planning.

Inference efficiency. We compare the complete per-image inference time, using 50 sampling steps for both methods and excluding the support masks, which both take as input. The scene-aware pipeline is timed in its original form (depth estimation and box sampling, then patch inpainting and mask-based paste-back for each object), whereas SceneP2F plans all instances jointly and renders them in a single pass over the full image. Averaged over one to five objects per image, SceneP2F takes about 20 s versus 45 s for the scene-aware pipeline. Planning is cheap for both (0.5–0.6 s for the scene-aware model vs. 0.7–0.8 s for ScenePlan); the difference comes from the fill stage, whose cost stays nearly constant for SceneP2F but grows by about 14 s per object for the scene-aware pipeline. The latter is thus faster for a single object (about 16 s vs. 20 s) but about 3.7× slower for five objects (about 74 s vs. 20 s).

Direct comparison with planning-based method. For a fair comparison of joint multi-object insertion, we replace the original per-object patch inpainting of the scene-aware method, which is not joint and relies on a mask decoder trained with instance masks, with single-pass rendering using the same generative backbone and training data as ours. Specifically, following the paper, we fit its class-conditional depth, height, and aspect-ratio distributions on each training split and sample boxes for the requested categories and counts with its original procedure, within the same support masks as ScenePlan; all boxes are then rendered in a single pass by FLUX.1-Fill fine-tuned on the same data as our method. Best results are underlined.

DatasetMethodmTCE ↓mCCE ↓LPIPS ↓DINO_sim ↑CLIP-I ↑PSNR ↑SSIM ↑
UAVDTScene-aware location1.7691.1940.5230.4600.83827.470.9631
SceneP2F0.817‾\underline{0.817}1.189‾\underline{1.189}0.482‾\underline{0.482}0.494‾\underline{0.494}0.857‾\underline{0.857}27.78‾\underline{27.78}0.9634‾\underline{0.9634}
VisDroneScene-aware location10.6436.5700.5440.5580.877826.380.9699‾\underline{0.9699}
SceneP2F7.510‾\underline{7.510}5.018‾\underline{5.018}0.536‾\underline{0.536}0.561‾\underline{0.561}0.8784‾\underline{0.8784}26.61‾\underline{26.61}0.9669
BDD100KScene-aware location2.3841.9930.5980.540‾\underline{0.540}0.86231.17‾\underline{31.17}0.9766
SceneP2F1.924‾\underline{1.924}1.648‾\underline{1.648}0.591‾\underline{0.591}0.540‾\underline{0.540}0.863‾\underline{0.863}31.140.9820‾\underline{0.9820}

Results. SceneP2F is clearly better in request fidelity on all three datasets: the mean total count error drops from 1.77 to 0.82 on UAVDT, from 10.64 to 7.51 on VisDrone, and from 2.38 to 1.92 on BDD100K. Visual consistency is slightly better or on par, and outside preservation is comparable.

AuthorsOpenReview ↗

Pesponse to Reviewer co1e (3)

W2: Concern about low absolute count accuracy in Figure 6

We understand the reviewer's concern. Below, we first describe how this metric is computed, which also explains why its absolute numbers look low, and then discuss what the gap to other methods shows.

What the metric measures. Figure 6 reports the Request Count Accuracy (RCA), a strict, sample-level measure. We detect objects in the source and edited images with GroundingDINO, match same-class detections one-to-one, and treat unmatched detections in the edited image as inserted objects. A sample requesting kk objects counts as correct only if exactly kk inserted objects are detected:

RCA(k)=1∣D∣∑d∈D1Nd,k∑i=1Nd,k1(k^d,i=k)\mathrm{RCA}(k)=\frac{1}{|\mathcal D|}\sum_{d\in\mathcal D}\frac{1}{N_{d,k}}\sum_{i=1}^{N_{d,k}}\mathbf 1\left(\hat{k}_{d,i}=k\right)

where Nd,kN_{d,k} is the number of samples requesting kk objects in dataset dd, and k^d,i\hat{k}_{d,i} is the detected number of inserted objects.

Why absolute values are low. Two factors lower the absolute values. First, the metric is all-or-nothing: a single missed, extra, or mismatched object makes the whole sample count as a failure. Second, inserted objects are identified by a detector, GroundingDINO, which is itself imperfect. It therefore reflects detection and matching errors in addition to count control, especially in the dense, small-object scenes of VisDrone and the distant, occluded objects of BDD100K.

What the gap shows. Although absolute low in number, SceneP2F achieves the highest accuracy at every request quantity, from 0.31 at k=1k=1 to 0.19 at k=5k=5, whereas the best competing method ranges from 0.12 down to 0.06. This corresponds to an improvement of 10.66 to 19.59 percentage points, i.e., 2.44× to 3.03×, and indicates consistently better count control across request sizes.

As noted in our limitations, category–count control still leaves room for improvement, which we consider a promising direction for future work.

W3: Is "geometry-aware conditioning" justified, given axis-aligned boxes and no orientation?

Geometry token. We use "geometry" in SceneAlign-Fill for the spatial geometry of each planned instance region, i.e., its position, extent, and scale, together with each image token's location relative to that region. It does not refer to the object's intrinsic shape or orientation. We thank the reviewer for raising this concern and will revise the description in Sec. 3.3 to make this clear.

As described in lines 286–288 and Eq. 8, for every image token we take its associated target box and encode the token's offset from the box center together with the box's log-width and log-height. This dense field is concatenated to the input of every image token and is available from the first block onward. It aims to tell each token which instance it belongs to, where it lies within that instance and the target scale. Regarding the ablation, removing the geometry token (G0, Table 4) degrades all three metrics. These metrics measure appearance, whereas the token mainly targets box fitting and instance separation.

Orientation. We did not add explicit orientation control, considering the setting of our datasets and task. All three benchmarks provide only axis-aligned annotations, and the downstream task is horizontal-box detection, so neither orientation supervision nor evaluation is available. Orientation is also hard to derive from the images: many UAV instances span only a few pixels or are captured at night or under blur, so their heading is often visually ambiguous, and some categories have no well-defined heading in these views (e.g., pedestrians in top-down imagery, traffic lights and signs).

Orientation is nevertheless partially constrained at both stages. In planning, the box aspect ratio correlates with vehicle heading in top-down views, and ScenePlan predicts it from scene context: removing the relative-geometry features raises the aspect-ratio error from 0.273 to 0.317 (Table 5, RF2). In rendering, tokens inside each box attend freely to the surrounding scene, including road structure, lane markings, and neighboring vehicles. The box fixes the extent and aspect ratio, and the remaining ambiguity is left in the inpainting stage.

Reviewer_GegZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SceneP2F for inserting a requested number of objects into an image while producing bounding-box annotations for detection training. ScenePlan jointly predicts object locations and estimates their sizes, followed by rule-based adjustments for placement validity. SceneAlign-Fill adapts FLUX.1-Fill with geometry, category, and appearance conditioning to render the planned objects. It also evaluates insertion quality on UAVDT, VisDrone, and BDD100K, and shows rare-class detection gains on UAVDT using YOLO11-L.

Critical strengths

The method fundamentally guarantees that the generated detection data has accurate ground truth by first explicitly specifying boxes connect category and then generating image inside the boxes which provide annotations for training. And there is an actual downstream benefit. On UAVDT, the generated data improves bus AP50 and truck AP50 grows, outperforming the reported Copy-Paste and X-Paste baselines.

Critical weaknesses
  1. running setting between method and baselinse seem to be a bit different. Appendix F describes different source-image sampling strategies for SceneP2F and the augmentation baselines.

  2. The downstream detection evidence is a bit limited. Although the rare-class gains over the original dataset are encouraging, the improvements over X-Paste are relatively small about 2.7 AP50 points for buses and 2.8 for trucks. I also only see the Detector training evaluated on UAVDT with YOLO11-L, which might be better if we have more than one dataset and more than one model.

Questions for authors
  1. Did Table 2 use identical source images across augmentation methods? Appendix F appears to describe different sampling strategies. If the subsets differ, how much does the sampling strategy alone improve detection?
  2. Are planned boxes used directly as training annotations, or are generated samples checked, filtered, or relabeled? What happens when a requested object is missing or does not fit its box?
  3. How were the baseline adaptation budgets and checkpoints chosen? In particular, were the fine-tuned FLUX-Fill and editor baselines trained sufficiently to make the comparison informative?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GegZ (1)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful and insightful comments!To address your concerns, we provide detailed responses and additional experimental results below.

W1 & Q1: Were source images identical across methods in the detection training, and how much does sampling alone contribute?

Thank you for the concern. In the original Table 2, SceneP2F used a different source-image subset. As described in Appendix F, we stratified the 40% subset by the number of existing target-class instances, so that most insertions fall in scenes where the rare class is absent or scarce. Copy-Paste and X-Paste shared the same uniformly sampled subset. All other settings (augmentation ratio, two instances per image, detector, and training schedule) were identical.

To remove this confound, we re-generated SceneP2F data on exactly the source images used by Copy-Paste and X-Paste and retrained the detector ("SceneP2F, aligned"). Best results are underlined. As shown in the table below, with identical source images, SceneP2F still outperforms both baselines on the inserted classes in general.

InsertedMethodRecallmAP50mAP50-95All mAP50All mAP50-95
2 busesOriginal0.4030.2290.1680.3410.208
Copy-Paste0.4080.2680.1900.3580.211
X-Paste0.4640.2830.2100.3510.212
SceneP2F, aligned0.4970.372‾\underline{0.372}0.285‾\underline{0.285}0.3690.237‾\underline{0.237}
SceneP2F, stratified (paper)0.522‾\underline{0.522}0.3100.2260.371‾\underline{0.371}0.224
2 trucksOriginal0.2460.07180.04500.3410.208
Copy-Paste0.1520.06590.04280.3430.207
X-Paste0.2020.09380.05900.3470.209
SceneP2F, aligned0.320‾\underline{0.320}0.1150.07650.3430.217‾\underline{0.217}
SceneP2F, stratified (paper)0.3160.122‾\underline{0.122}0.0775‾\underline{0.0775}0.351‾\underline{0.351}0.208

Comparing the two SceneP2F rows isolates the sampling strategy. Stratified sampling does not account for the gains: for buses it yields higher recall (+2.5 points) but lower mAP50 (0.310 vs. 0.372), and for trucks the two strategies differ by less than one point on every inserted-class metric. The improvements in Table 2 therefore stem from the quality of the generated data rather than the choice of source images, and may be further improved with a better source-image sampling strategy.

Q2. How are planned boxes used as labels, and what if insertion fails?

In Table 2, the planned boxes are used directly as training annotations. Generated samples are not checked, filtered, or relabeled, and no retry is performed. The explicit layout guarantees that the annotations match the request, but not that every planned instance is successfully rendered. Failure cases are therefore present in the augmented set: a missing object leaves a box on background, which acts as a false-positive training label, and an object that does not fit its box introduces localization noise. The gains in Table 2 were obtained despite this noise.

To measure its effect, we applied two additional steps to the UAVDT two-bus augmentation set used in the paper, both independent of the evaluation detector.

  • Filter. We compute the CLIP similarity between the content of each inserted box and the class text, and keep the top 50% of augmented images within each video sequence, i.e., 20% of the training images. A random 50% subset serves as a quantity-matched control.
  • Relabel. On the filtered set, each planned box is mapped to the generated image and used, together with the text prompt "bus", to prompt SAM3. The label is replaced by the tight bounding box of the matched mask. If two planned boxes yield the same mask, only one annotation is kept. If no mask is matched, the planned box is kept.

All runs use YOLO11-L with the same training settings. Best results are underlined.

Augmented setRecallmAP50mAP50-95All mAP50All mAP50-95
None (original)0.4030.2290.1680.3410.208
Random0.5090.3140.2220.3460.214
Filtered0.538‾\underline{0.538}0.3630.2670.3660.230
Filtered relabel0.5330.378‾\underline{0.378}0.281‾\underline{0.281}0.376‾\underline{0.376}0.242‾\underline{0.242}

As shown above, at the same data quantity, filtering improves bus mAP50 by 4.9 points over the random subset (0.314 → 0.363), confirming that failed insertions introduce label noise. Relabeling adds a smaller gain in accuracy (mAP50 0.363 → 0.378, mAP50-95 0.267 → 0.281) with recall essentially unchanged, consistent with tighter boxes reducing localization noise. Table 2 should therefore be read as a baseline without filtering or relabeling, and verification and relabeling are a promising direction for future work.

AuthorsOpenReview ↗

Response to Reviewer GegZ (2)

W3 :More experiments on downstream detection.

Following the suggestion, we conducted additional downstream detection experiments: a second detector, Faster R-CNN, on UAVDT, and a second dataset, VisDrone, with YOLO11-L, both in the two-bus setting for comparability. Within each setting, all methods share the same training configuration. Best results are underlined.

Dataset / DetectorMethodRecallmAP50mAP50-95All mAP50All mAP50-95
UAVDT / Faster R-CNNOriginal0.3120.1530.1020.2680.156
Copy-Paste0.4750.1530.1030.2830.161
X-Paste0.3910.1880.1310.2990.173
SceneP2F0.600‾\underline{0.600}0.351‾\underline{0.351}0.248‾\underline{0.248}0.332‾\underline{0.332}0.200‾\underline{0.200}
VisDrone / YOLO11-LOriginal0.5830.6570.487‾\underline{0.487}0.4490.268
Copy-Paste0.5920.6550.4830.4490.268
X-Paste0.5910.6490.4770.453‾\underline{0.453}0.270‾\underline{0.270}
SceneP2F0.615‾\underline{0.615}0.663‾\underline{0.663}0.487‾\underline{0.487}0.4500.270‾\underline{0.270}

UAVDT. The gains carry over to Faster R-CNN and are larger than with YOLO11-L generally: SceneP2F improves bus mAP50 from 0.153 to 0.351 and recall from 0.312 to 0.600, exceeding X-Paste by 16.3 mAP50 points.

VisDrone. We chose the bus class for two reasons: it mirrors the UAVDT setting, and it is a rare class in VisDrone, accounting for 2.12% of valid instances, the third rarest of the ten classes. SceneP2F improves bus recall from 0.583 to 0.615, while bus mAP50 changes only from 0.657 to 0.663 and all-class metrics are essentially unchanged; X-Paste is marginally higher on all-class mAP50 (0.453 vs. 0.450). Copy-Paste and X-Paste slightly reduce bus mAP50 (0.657 → 0.655 and 0.649). A likely reason is that bus, although rare, is already well detected on VisDrone (baseline mAP50 0.657 vs. 0.229 on UAVDT), leaving little headroom.

AuthorsOpenReview ↗

Response to Reviewer GegZ (3)

Q3 : Baseline budget and checkpoint selection

Fine-tuned FLUX-Fill. For each dataset, we uses the same removal pairs and the same region-weighted flow-matching loss (insertion-region weight 8) as our SceneAlign-Fill, together with the same LoRA configuration (rank/alpha 16/16, learning rate 1e-4), resolution (768), effective batch size (4), and precision (bf16). The maximum budget (50,000 updates), validation schedule (every 1,000 updates, with early stopping after 5,000 updates without improvement), and checkpoint-selection criterion are also identical: the checkpoint with the best validation score is selected, based on class-balanced CLIP similarity with in-box LPIPS breaking near-ties.

Editor baselines. Budgets follow each method's official recipe where one exists. Add-SD uses the official pipeline (full UNet fine-tuning from SD 1.5, effective batch size 128) for 30,000 updates. Add-it is officially training-free and the official Qwen-Image-Edit LoRA script sets its epoch count for a toy example dataset, so we train both for a fixed 2,000 updates (details in App. E.2), and three use the final checkpoint. Regarding sufficiency, we tracked insertion and background-preservation metrics for Add-it on 100 held-out training pairs every 500 steps; Qwen-Image-Edit is trained with the same budget. We will add the Add-SD settings to Appendix E.2.

Our method SceneP2F. SceneAlign-Fill uses the same budget and selection rule as FLUX-Fill-FT. The budgets and selection criteria of ScenePlan and the size predictor are given in Appendix E.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Summary of our Rebuttal

Dear Area Chair and Reviewers,

First, we would like to express our sincere gratitude for your time, careful reviews, insightful comments, and constructive suggestions! To facilitate your review, we summarize the reviewers initial concern and our efforts below.

Reviewer GegZ (score 3, confidence 3)

The reviewer acknowledges the effectiveness of our generation method and the quality of its results, and raises further questions about the downstream benefits, along with more detail on how the adaptation budgets and checkpoints were chosen.

Our efforts and response:

  • We retrained with aligned source images, which shows that SceneP2F’s gains do not depend on the sampling strategy. We also added filtering and relabeling experiments, which show that the unfiltered results in Table 2 are a conservative baseline. Together, these experiments confirm the quality of our generated data.
  • To broaden the downstream evaluation, we added a second detector and a second dataset, and the results show that SceneP2F’s gains generalize.
  • We detailed the adaptation budgets and checkpoint selection for all methods.

Reviewer co1e (score 2, confidence 2)

The reviewer recognizes the importance of the problem, the soundness of the two-stage framework, the extensive evaluation, and the promising gains on downstream task. The reviewer asks us to clarify our key contribution relative to existing object insertion methods, suggests a comparison with planned conditioned method, and asks about the absolute count accuracy and several method details: the geometry token, the inpainting mask, and the name “SceneP2F”.

Our effort and response:

  • We made explicit the gap in jointly inserting multiple objects to augment detection datasets with a capability comparison table, and we mapped each of the task’s four requirements (explicit annotations, request fidelity, source preservation, scene-consistent appearance) to a specific design in our SceneP2F and its supporting evidence.
  • We re-implemented the closest planning-based method, the scene-aware location model. With the same fill model, ScenePlan gives lower count error and better appearance. And with the fill stage aligned for fairness it consistently lowers the count error on all three datasets.
  • We acknowledge that the absolute count accuracy is low, but because this strict metric also absorbs detector errors and applies equally to all methods, the relative comparison is more informative. SceneP2F reaches 2.44× to 3.03× the accuracy of the best competing method at every request quantity.
  • We clarified that “geometry” refers to each planned region’s position, extent, and scale. Orientation is partially constrained by the predicted aspect ratios and by scene context. The inpainting mask is rasterized directly from the planned boxes, and “SceneP2F” stands for “Scene Plan-to-Fill”.

Reviewer 8Aas (score 3, confidence 3)

The reviewer point out the importance of controllable multi-object insertion for detection data augmentation, the well-designed two-stage framework, and the promising results. The reviewer would like us to clarify the methodological contribution, to provide more controlled comparisons with layout-conditioned methods, and to extend the downstream evaluation. The reviewer also asks for further analysis of the count accuracy.

Our effort and response

  • We provided a clear capability comparison between SceneP2F and representative methods for multi-object insertion. For each of the four capabilities required by the task, we explained how SceneP2F realizes it, pointed to the corresponding equations and appendix sections in the paper, and gave supporting experimental evidence from results in the paper.
  • To clarify the contribution of each component, we conduct a detailed ablation that replaces the planner, removes post-processing, or replaces the renderer one at a time, and a controlled comparison with the closed layout-conditioned method on 3 datasets. It shows that ScenePlan contributes most to request fidelity, followed by post-processing and SceneAlign-Fill, and that the full model is best or tied for best on all metrics.
  • We analyzed alignment and failure cases between planned boxes and generated objects across request quantities, and we ran a human evaluation to compare against the detector-based metrics.
  • We extended the downstream evaluation along all three axes the reviewer suggested: an additional dataset, detector, and a generative baseline. SceneP2F consistently outperforms all augmentation baselines in general.

We are grateful that two reviewers confirmed our responses clarified all of their questions and maintained their positive assessments. We will incorporate the additional experiments and clarifications into the final manuscript, and believe these results further reinforce our central contribution. We sincerely thank all reviewers for their thoughtful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.