- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SceneP2F, a two-stage framework for controllable multi-object insertion to augment object detection datasets. The framework consists of ScenePlan, which predicts scene-compatible object layouts according to category and count requirements, and SceneAlign-Fill, which synthesizes objects within the planned regions while maintaining visual consistency and preserving existing scene content. Experiments on UAVDT, VisDrone, and BDD100K demonstrate improvements in category-count fidelity, visual consistency, and background preservation over several existing methods. The authors further demonstrate the effectiveness of the generated data for improving the detection performance of underrepresented categories.
Critical strengths
-
The paper addresses an important problem in object detection data augmentation, particularly the controllable insertion of multiple objects with specified categories and counts.
-
Well-designed framework. The proposed two-stage architecture effectively connects scene-aware layout planning with instance-level image generation. The integration of geometry-aware conditioning, regional attention, and appearance guidance is reasonable and technically meaningful.
-
Promising experimental results. The method demonstrates competitive performance across three benchmarks, with clear improvements in request fidelity and scene preservation. The downstream detection experiments further support the practical value of the proposed framework.
Critical weaknesses
-
The methodological novelty could be further clarified. Several components, including diffusion-based layout planning, region-aware attention, and weighted generation objectives, are closely related to existing techniques. The authors should better distinguish their key contributions from prior methods and provide stronger evidence for the necessity of the proposed design choices.
-
The absolute accuracy of category-count control remains limited. Although SceneP2F consistently outperforms competing methods, Figure 6 indicates that the exact-count accuracy remains relatively low, particularly for larger insertion requests. Further analysis of failure cases and the discrepancy between planned and successfully generated instances would strengthen the paper.
-
The evaluation relies heavily on detector-based measurements. The category-count metrics depend on GroundingDINO predictions, which may introduce errors, especially for small or challenging objects. Additional human-verified evaluations or direct measurements of generated object and bounding-box alignment would make the conclusions more convincing.
-
The comparison with existing methods could be strengthened. Different approaches use different conditioning mechanisms and training configurations, making it difficult to fully isolate the advantages of the proposed framework. More controlled comparisons with layout-conditioned generation methods would help establish the effectiveness of the individual contributions.
-
The downstream detection evaluation is relatively limited. The augmentation experiments are conducted only on UAVDT with YOLO11-L and two underrepresented categories. Extending the evaluation to additional datasets, detection architectures, and competing generative augmentation methods would provide stronger evidence of generalization.
Questions for authors
-
How frequently do the planned object layouts correspond to correctly generated instances with accurate categories and bounding boxes, particularly for larger insertion counts?
-
Could the authors provide additional computational cost analysis and clarify the relative contributions of layout planning, post-processing, and image generation to the final improvements?
7 comments
Response to Reviewer 8Aas (2)
Q2: Computational cost and component-wise contribution analysis
Component contributions. To clarify contribution from each component, we replace one component of SceneP2F at a time and evaluate on 200 test requests of UAVDT. Best results are underlined.
| Variant | Layout | Post-proc. | Generation | mTCE ↓ | mCCE ↓ | LPIPS ↓ | DINO sim ↑ | CLIP-I ↑ |
|---|---|---|---|---|---|---|---|---|
| Replace all | Data prior | ✗ | FLUX.1-Fill-FT | 1.80 | 1.21 | 0.52 | 0.44 | 0.82 |
| Replace planner | Data prior | ✓ | SceneAlign-Fill | 1.49 | 1.17 | 0.51 | 0.48 | 0.83 |
| Remove post-processing | ScenePlan | ✗ | SceneAlign-Fill | 1.25 | 0.48 | 0.49 | ||
| Replace renderer | ScenePlan | ✓ | FLUX.1-Fill-FT | 1.19 | 0.48 | 0.49 | 0.85 | |
| Full (SceneP2F) | ScenePlan | ✓ | SceneAlign-Fill |
Among our components, ScenePlan (layout-planning) contributes most, followed by post-processing and then SceneAlign-Fill (generation). Compared with a dataset-prior layout, ScenePlan reduces the mean total count error from 1.49 to 0.80 and improves all three appearance metrics. Post-processing further reduces this error from 1.25 to 0.80, with little change in other metrics. Compared with FLUX.1-Fill-FT, trained on the same data with the same objective, SceneAlign-Fill keeps the total count error unchanged but reduces mCCE (1.19 → 1.16), showing that the instance-aware tokens improve per-class object binding; it also improves gives a small but consistent gain in this end-to-end setting. Outside-preservation metrics, omitted for brevity, remain nearly identical across all variants (PSNR ≈ 27.5 dB, SSIM ≈ 0.96)
The appearance differences are small in absolute terms, partly because objects inserted in UAVDT are small and fairly uniform in size, so crop-level metrics vary little across variants. Still, the full model is best or tied for best on all five metrics, while replacing all components yields the worst result on every metric, supporting SceneP2F as a framework that balances explicit annotations, request fidelity, source preservation, and scene-consistent appearance.
Computational cost. All SceneP2F components are trained on a single A6000, each within 40 GPU-hours except the BDD100K size predictor (about 140 GPU-hours, due to the larger training set). At inference, joint box planning takes about 1 s per image, while FLUX-based refinement dominates the cost at about 16–17 s per image. Since augmentation is performed offline once, this cost adds no overhead to detector inference.
Response to Reviewer 8Aas (3)
W2 & W3 & Q1: The discrepancy between planned and generated instances and failure analysis
Why the absolute values in Figure 6 are low. Two factors lower the absolute values. First, the metric is all-or-nothing: a single missed, extra, or mismatched object makes the whole sample count as a failure. Second, inserted objects are identified by a detector GroundingDINO is imperfect, and we conduct human evaluation (see our response to W3 below) confirms this. The metric therefore reflects detection and matching errors in addition to count control, especially in the dense, small-object scenes of VisDrone and the distant, occluded objects of BDD100K.
Planned boxes vs. generated objects. We analyzed the inference results on UAVDT by request quantity N (200 images per N).
| N | Valid layout (planning) | Count accuracy (generation) | Average IoU (alignment) | RMSE, valid boxes vs. generated objects |
|---|---|---|---|---|
| 1 | 89.00% | 66.00% | 0.8016 | 0.6928 |
| 2 | 75.00% | 57.50% | 0.7914 | 1.0075 |
| 3 | 62.00% | 48.00% | 0.7805 | 1.3435 |
| 4 | 56.00% | 33.00% | 0.7824 | 1.8125 |
| 5 | 43.50% | 39.50% | 0.7782 | 2.0869 |
| All | 65.10% | 48.80% | 0.7839 | 1.4792 |
Valid layout: an image whose planned boxes all lie at least 80% inside the SAM3 valid region and overlap other boxes by IoU less than 0.2. Count accuracy: the fraction of images whose detected number of inserted objects equals the request. Average IoU: between planned boxes and their matched detected objects. RMSE: between the numbers of valid planned boxes and of detected inserted objects per image.
Alignment between planning and filling. When a planned box is matched to a generated object, the two agree well in position and size: the average IoU is about 0.78 and stable across N. The discrepancy lies in the number and category of objects rather than in their geometry: the RMSE grows from 0.69 at N = 1 to 2.09 at N = 5, although the error per requested object does not increase (0.69 to 0.42). Generation accuracy also degrades far more slowly than independent per-object errors would imply: 39.5% at N = 5, versus 12.5% if each object failed independently at the N = 1 rate.
Failure analysis. The two stages limit performance in different regimes. At N = 1, 89.0% of layouts are valid but 66.0% of requests are fulfilled, so at least two thirds of the failures occur with a valid layout and stem from the fill stage or the detector. As N grows, the share of fully valid layouts falls from 89.0% to 43.5%, faster than count accuracy (66.0% to 39.5%), so planning becomes the increasingly important factor for larger requests. On the fill side, there are two possible reasons: UAV imagery contains many very small objects, which are hard to render faithfully, and the fill model is likely weaker on long-tail categories. We will add this analysis to the paper and state these limitations explicitly.
W3: Reliance on detector-based measurements
We agree that detector errors can affect the count metrics. However, since all methods are evaluated with the same detector, relative comparisons remain reliable, and we additionally conduct a human evaluation as a reference.
Relative comparison. All methods are evaluated with the same detector, thresholds, and matching rule, so detector errors apply to every method alike. While the absolute values are affected, the relative differences between methods under this shared protocol remain informative.
Human evaluation. To verify the detector-based results, we conduct a human evaluation on one dataset per domain as a reference: UAVDT for aerial scenes and BDD100K for driving scenes, with 100 randomly sampled SceneP2F outputs each. Two volunteers (50 images each) annotated the number and categories of newly inserted objects, and we compare the resulting metrics with the detector-based ones on the same images.
| Dataset | mTCE (human) | mTCE (detector) | mCCE (human) | mCCE (detector) |
|---|---|---|---|---|
| UAVDT | 0.56 | 0.98 | 0.83 | 1.27 |
| BDD100K | 1.38 | 2.05 | 1.07 | 1.97 |
The human and detector protocols show a consistent trend : on both metrics, they rank the two datasets in the same order, so the detector-based metrics reflect the same relative behaviour as human judgment. The human-based errors are lower in absolute terms (by 33–43% in mTCE and 35–46% in mCCE), which we attribute to two factors. First, in cluttered scenes, detections of existing objects are unstable between the source and edited images, and the unmatched detections are counted as insertions. Second, correctly inserted objects that overlap their neighbours or are very small are often missed or merged by the detector.
Response to Reviewer 8Aas (1)
We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful and insightful comments!To address your concerns, we provide detailed responses and additional experimental results below.
To clarify our main contribution: we propose SceneP2F, a plan-to-fill framework that aims to balances the four requirements of this task: explicit annotations, request fidelity, source preservation, and scene-consistent appearance in multi-object insertion task. The table below compares its capabilities with those of related methods.
✓: explicit mechanism supported; △: partial or implicit support; ✗: not supported or no public evidence.
¹ Single-run multi-object generation, without instance-specific conditioning. ² No documented preservation operation in the evaluated configuration. ³ Learned from background-consistent pairs; no masking at inference.
How SceneP2F realizes each capability, and the supporting evidence:
Explicit class-labeled box. ScenePlan outputs a set of class-labeled boxes (Eq. 1), one per requested instance. The same boxes serve as the input of SceneAlign-Fill and as the final annotations. This capability makes the outputs directly usable for detection data augmentation. Evidence: using the planned boxes directly as labels improves bus and truck AP50 from 22.9 to 31.0 and from 7.2 to 12.2 (Table 2).
Joint multi-object insertion. In planning, all target centers are denoised jointly (Eq. 3), with self-attention among target tokens and a set-level matching loss (Eq. 4), so the requested categories and counts are fixed at the layout level. In inpainting, all boxes are filled in a single pass: each image token inside a box attends to that box's token, and a dense geometry field encodes each token's position relative to its nearest planned box (App. D). Evidence: on requests with N = 1–5 objects, SceneP2F reduces mTCE from 2.744 to 0.817 on UAVDT and from 5.523 to 1.924 on BDD100K relative to the best baseline (Table 1); with all boxes filled in a single pass, SceneAlign-Fill reaches mTCE 0.77, vs. 1.68–2.12 for pretrained inpainters (Table 3).
Object and background preservation. Preservation follows from the plan-to-fill design itself: ScenePlan fixes a box layout that against overlaps with existing objects (App. C.2), and SceneAlign-Fill generates only within the masked boxes, conditioned on the unmasked source latent. The flow-matching loss (Eq. 12) keeps full supervision outside the boxes. Evidence: Outside PSNR improves by 5.49, 3.94, and 3.33 dB over the best baseline on UAVDT, VisDrone, and BDD100K (Table 1).
Scene-consistent appearance without reference image. In SceneAlign-Fill, a style token is derived only from existing objects in the same image (App. D), on top of the pretrained FLUX.1-Fill prior, so no reference image is needed. Evidence: removing the style token degrades all three in-box metrics (ST0, Table 4); SceneP2F obtains the best DINO similarity and CLIP-I on all datasets (Table 1) and the best LPIPS, DINO similarity, and CLIP-I among inpainters (Table 3).