acceptodds
Under review as a conference paper at ICLR 2027

HorizonForgeV2: Agentic Scene Editing for Long-Tail Driving Video Generation

Abstract

Long-tail scenarios, such as unusual road users and unexpected obstacles, remain particularly challenging for autonomous-driving systems and are important for both training and evaluation. They are also extremely rare in real-world data: of over 6.39 million miles of Waymo driving logs, only 0.03% qualify as long-tail scenarios. Collecting long-tail scenarios directly is prohibitively expensive, and some, such as an elephant standing in the middle of the road, cannot be staged safely at all. Driving simulation can instead generate these scenarios on demand. Recent methods combine the precise control of 3D scene representations with the photorealism of video generation. These methods break down, however, when the inserted 3D asset falls outside the video generator’s training distribution: the video generator has rarely seen such objects during training and tends to distort or even remove them. Image-editing foundation models, in contrast, are trained on web-scale data and can insert diverse long-tail objects that fit the surrounding scene. Building on this observation, we present HorizonForgeV2, which moves long-tail appearance generation from the video domain to the image domain: image-editing models synthesize realistic keyframes, and a keyframe-conditioned video diffusion model (VDM) turns them into temporally coherent videos. Importantly, HorizonForgeV2 is trained only on existing driving videos, without collecting any new keyframe data. Edited keyframes can still be inconsistent with the 3D scene, and these errors propagate to the generated video. We therefore employ an agentic framework that inspects the intermediate keyframes and videos, locates failed edits, and iteratively refines generated videos with tool calling. Compared with HorizonForge, HorizonForgeV2 raises the Operation Success Rate (OSR), a six-criterion score on a 1–10 scale, from 5.77 to 7.52 on our long-tail benchmark and from 6.47 to 7.75 on common driving-scene edits.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer vision, Autonomous Driving, Video Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zEU7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HorizonForgeV2 generates edited driving videos by separating object appearance from temporal propagation. An image editor supplies sparse keyframes, while a Cosmos-based video model, trained on imperfect keyframes and coarse rendering/depth controls, propagates those references throughout the clip. An agent inspects intermediate outputs and revises keyframes or generation settings within a candidate budget. Evaluation uses 120 long-tail insertions and 109 common edits. The full method reaches 7.52 and 7.75 OSR, respectively, compared with 6.94 and 6.49 for the strongest listed baseline, Difix3D. A blinded human study, a second model judge, component ablations, and a repeated-sampling control provide additional validation.

Critical strengths
  1. The method is well matched to a concrete generalization bottleneck. Rare-object appearance is supplied by an image editor rather than inferred solely from the video generator's driving-data prior; coarse rendering and depth retain structural control. The model is trained on existing driving clips with perturbed references, avoiding the need to collect rare-object driving videos for each category (Section 3.2; Appendix A.3). Removing the auxiliary branches lowers initial OSR from 6.58 to 6.00, while removing keyframe augmentation lowers it to 6.45 (Table 5). This is a coherent and practically useful system contribution.

  2. The evidence goes beyond the agent's own evaluator and includes a meaningful sampling control. The blinded human study favors the full system in 57.14% of samples, and the independent Qwen judge also ranks it first (Table 4; Appendix B.1, Table 9). Appendix B.6 compares adaptive revision with repeated sampling sharing the first candidate: at four candidates, scores are 7.52 versus 7.27, and the agent's advantage concentrates on initially difficult cases. These controls substantially strengthen the claim that the complete pipeline improves edit quality.

Critical weaknesses
  1. The incremental benefit of the agent is less independently established than the quality of the complete system. The same verifier supplies feedback, selects the best candidate, and defines the OSR used in the agent-versus-sampling comparison (Sections 3.3 and 4.1; Appendix B.6). Independent judging and the human study compare the full system against other methods, but do not directly validate the relatively small 0.25-point advantage over four-candidate repeated sampling. Candidate counts are matched, while total compute is not: the agent adds inspection calls and can increase generation effort. No uncertainty intervals are given for this incremental gain. Thus the results support a useful pipeline more strongly than a precisely measured advantage of agentic diagnosis.

  2. Evidence that better videos become useful long-tail driving data remains narrow. The downstream test is a single-camera, car-only detector, with an absolute AP increase of 0.0017 and no reported seed variability (Section 4.4). It does not test the pedestrian/animal categories or driving-policy behavior. Labels are inherited from the edited 3D scene even though generated appearance and geometry can drift; the five-frame OSR evaluation does not quantify framewise 3D-label fidelity (Section 4.1; Appendix A.4). These limitations do not negate the video-editing contribution, but they restrict the practical training and evaluation claims motivating it.

Questions for authors
  1. Validation request: Does the four-candidate agent still beat four-candidate repeated sampling when the retained outputs are compared by the independent judge or blinded humans? Please include uncertainty clustered by source scene and report total video-generation time or GPU cost alongside API cost.

  2. Clarification request: How are the 147 human-study samples drawn from the 120 long-tail cases, how many participants rated each case, and how are repeated ratings aggregated? This determines the independent sample size and the uncertainty of the reported win rate.

  3. Validation request: Is the detector AP improvement consistent across training seeds, and how well do generated objects agree with the inherited 3D boxes? This would determine how much weight to place on the downstream-data contribution.

3 comments

We thank the reviewer for the detailed review, for recognizing "a coherent and practically useful system contribution", and for the concrete validation requests. We ran each one. Experiments shared with other reviewers are in the General Response (G1 to G4).

1. Independent-judge validation, scene-clustered CIs, and compute cost (W1, Q1)

We refer the reviewer to G2 (App. B.6), which answers each part of W1 and Q1: under the independent Qwen judge, the agent beats repeated sampling at both 4 and 8 candidates, with 95% intervals from a cluster bootstrap over the 59 source scenes, and G2 reports GPU time alongside API cost. Repeated sampling needs 8 candidates, at 66% more GPU time, to match the 4-candidate agent's in-loop OSR, and still scores lower under Qwen. The average gain is moderate because most first candidates already succeed; it concentrates on failed first attempts (OSR < 6), and G1 demonstrates that this gain comes from accurate stage diagnosis rather than untargeted retries.

2. Multi-seed detection, long-tail categories, and 3D-box fidelity (W2, Q3)

  • Multi-seed stability and broader categories (G3, Sec. 4.4, App. B.9). Please see G3 for experiments. Expanding the 3-seed evaluation to a multi-category detector tested on out-of-domain real benchmarks shows the same expected improvement on rare-object detection.
  • 3D-label fidelity (G4, App. B.10). G4 quantifies how well the generated objects agree with their inherited 3D boxes: it projects the boxes onto four frames per clip and has a blind judge check whether each inserted object fits its box and has the requested class.
  • Driving policy. We leave closed-loop policy evaluation to future work. Reliable perception is its prerequisite: a driving policy can respond appropriately to a rare hazard only once the object is accurately recognized rather than misclassified as ordinary traffic.

3. Human-study sampling and aggregation protocol (Q2)

Each of the 21 participants rated 7 cases sampled uniformly at random without replacement from the 120 long-tail cases, yielding 21×7=14721 \times 7 = 147 evaluation samples (~1.67 ratings per case on average, covering 88 unique cases). One sample corresponds to a single participant's blind side-by-side rating of all compared methods on one case. Table 4 averages uniformly across all 147 samples, splitting ties evenly in win-rate calculations.

As the author–reviewer discussion period ends on November 18 (AoE), we would greatly appreciate it if the reviewer could check our response, so that we can still address any further questions before the discussion closes.
In brief, we ran each requested validation: the independent-judge comparison with scene-clustered intervals and GPU/API cost (G2), the multi-seed, multi-category detector (G3), the agreement with inherited 3D boxes (G4), and the human-study protocol (our response, point 3).
If our response addresses the reviewer's concerns, we would be grateful if the reviewer could consider reflecting this in the rating.

Thanks the authors for providing detailed feedback. I will maintain my original rating.

Reviewer_d3ZMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HorizonForgeV2, an agentic scene-editing framework for generating realistic long-tail driving videos from edited 3D scenes. To address the difficulty of preserving rare objects in rendering-conditioned video generation, the proposed method decouples object appearance synthesis from temporal video generation by first producing realistic edited keyframes and then propagating them through a keyframe-conditioned video diffusion model. It further incorporates keyframe perturbation training, auxiliary geometric conditions, and an agentic refinement mechanism to improve generation robustness and consistency. Experiments on 120 long-tail and 109 common driving-scene editing cases demonstrate improvements over existing baselines, achieving an Operation Success Rate (OSR) of 7.52 on the long-tail benchmark, compared with 6.94 for the strongest baseline. Additional ablation studies, human evaluations, and downstream detection experiments further examine the effectiveness of the proposed framework.

Critical strengths

The paper identifies an important inconsistency and controllability issue in direct editing-plus-generation pipelines for driving videos, and proposes a coherent solution to address it.

Critical weaknesses

The criteria for attributing failures to keyframes or video generation remain unclear. More direct validation of the agent’s diagnosis and action selection would strengthen confidence in its targeted corrections.

The second-judge experiment re-evaluates existing outputs rather than testing a different verifier within the revision loop. It remains unclear how sensitive the corrective decisions and final video quality are to the verifier model. Testing alternative in-loop verifiers would help establish the robustness of the revision process.

Questions for authors

The framework appears broadly applicable to preserving edit fidelity and temporal consistency. Could the authors clarify what evidence distinguishes its benefits for rare-object generation from its improvements to the reliability of controllable video editing more generally?

3 comments

We thank the reviewer for the careful review, and for recognizing that the paper "identifies an important inconsistency and controllability issue" and "proposes a coherent solution". We ran experiments for both validation requests. Experiments shared with other reviewers are in the General Response (G1 to G4).

1. Failure attribution criteria and direct validation (W1)

  • Criteria. As detailed in G1, we give the agent a guideline built on two principles: trust the coarse 3D rendering for position and motion (not for its crude appearance), and check the upstream keyframes before the VDM. The agent applies them by comparing the coarse rendering, nearby keyframes, and generated video at matching timestamps.
  • Validation. We refer the reviewer to G1, which tests the agent's diagnosis and action selection directly. The experiments are also added to Sec. 4.3 and App. B.7 of the revised paper, with figures showing the agent's reasoning (Figs. 6 and 7).

2. Sensitivity to replacing the in-loop verifier with a different model family (W2)

To test sensitivity to the in-loop critic directly rather than only re-scoring fixed outputs, we reran the full revision loop across all 120 cases while replacing the paper's verifier (gpt-5.6-luna) with a model from a different family, Gemini (gemini-4-argon), for both in-loop defect diagnosis and candidate selection. The planner, executor, image editor, VDM, and shared first candidate remain identical. We evaluate the final retained clips from both loops using the independent Qwen judge (App. B.1), which neither loop observes (App. B.8).

  • Identical overall final quality. Both revision loops achieve an overall Qwen score of 8.28 across the 120 cases.
  • Consistent gains concentrated on initially failing cases. Grouping the 120 cases by the initial OSR of the shared first candidate, the table below reports mean Δ\Delta Qwen score (retained best-of-4 candidate minus first candidate):
In-loop verifierInitial OSR < 6Initial OSR 6 to 8Initial OSR ≥\ge 8
gpt-5.6-luna (paper)+0.38+0.04+0.05
gemini-4-argon+0.36-0.04+0.08

Both verifiers concentrate improvements on failed initial renders with nearly identical magnitude (+0.38 vs. +0.36 Qwen) while leaving already strong renders intact, confirming that the revision process is robust across verifier model families.

3. Evidence specific to rare-object generation vs. general controllable editing (Q1)

Conceptually, the key distinction lies in data requirements: improving common-object or trajectory editing can be achieved by collecting more driving clips and fine-tuning a VDM, whereas scaling driving-video collection across viewpoints, motions, and lighting for every rare category is infeasible (Sec. 3.2). Our VDM is trained solely on existing driving videos—without deliberately collecting any long-tail footage—and imports rare-object appearance from an open-world image prior.

Three pieces of empirical evidence show that our gains are largest where the object's appearance must be synthesized rather than reused, which is always the case for rare objects:

  • Insertion vs. trajectory edits in single-pass VDM (Table 3). Without the agent loop, trajectory edits (reusing existing scene objects) rise by +0.37 OSR (6.49 to 6.86), whereas object insertion (requiring synthesized object appearance) jumps by +1.21 OSR (6.43 to 7.64).
  • Semantic preservation on rare categories (G4). HorizonForge often turns a rare object into a common one, such as an elephant into a truck (Fig. 9), and keeps the requested class in only 66% of boxes, against 94% for ours.
  • Category-specific downstream gains (G3). Our clips raise detector AP on scarce categories (traffic cones, road-scene animals, pedestrians), while abundant ordinary cars stay unchanged.

The agentic repair loop then provides general failure recovery on top of this capability.

Thank you for the clarifications and additional experiments. The new results on failure diagnosis and alternative in-loop verifiers help address my concerns, and the explanation of the long-tail motivation is also helpful.

We thank the reviewer for the follow-up, and we are glad that the new results address the reviewer's concerns. If our response has resolved them, we would be grateful if the reviewer could consider reflecting this in the rating.

Reviewer_gKKWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The pipeline edits keyframes of a reconstructed driving scene, then uses them to guide video generation. The image editor supplies rare-object appearance that the driving video model struggles to invent. An agent inspects intermediate outputs and retries with different keyframes or generation settings.

Critical strengths

Getting appearance from a broad image prior and motion from the edited scene is a sensible split. Learning to propagate those references from ordinary driving footage avoids needing a new video dataset for every rare object.

Critical weaknesses
  1. I don't buy the agentic framing as the main research contribution. Underneath the role names, this is an image editor, a conditioned video generator, and a model choosing what to retry. It can be useful software. But calling the steps planner, executor, verifier, and reasoner doesn't tell us anything new about why a repair works. That's not agentic.
  2. The actual question is how to tell what's broken. A bad reference and a video model failing to follow a good reference can produce similar symptoms. They need different fixes. The loop tries revisions and keeps better outputs, but the paper doesn't extract much insight about which observations distinguish these causes. Most of that judgment remains inside the general-purpose model. Showing that feedback helps doesn't, by itself, explain what was learned about failure diagnosis.
  3. The long-tail framing also needs more care. An unusual object following a supplied trajectory is an appearance stress test, not a new traffic interaction. Appearance coverage matters, but it's not the same as coverage of rare driving situations. On the narrower editing task, sparse edited frames plus video guidance are already familiar from NOVA and related work. I see a better assembled editing pipeline, not a new principle or a convincing case that a major bottleneck has been removed.
Questions for authors
  1. What new understanding should we take from the agent's repair decisions, beyond a general-purpose model inspecting outputs and trying different tools?
  2. What tells the system that a failure comes from a keyframe rather than video propagation? When geometry and appearance references disagree, which one should win, and why?
  3. Which important part of the long-tail problem does this setup actually remove, rather than make easier to visualize?
6 comments

We thank the reviewer for the detailed comments, and for recognizing that taking appearance from an image prior and motion from the edited scene is "a sensible split". Experiments shared with other reviewers are in the General Response (G1 to G4).

1. Why the framework is agentic, and why that matters (W1)

We respectfully disagree that the framework is not agentic. What makes it agentic is that each action depends on what the agent observes: it inspects the intermediate keyframes and video, decides which stage caused the failure, and calls the tool for that stage. When a repair does not fix the defect, it uses that outcome to choose differently. A retry loop would instead resample every stage regardless of the failure, as repeated sampling does (G2). We agree that the role names are not the contribution, and we do not claim them as one. As stated in the paper, our second contribution is an agent that inspects intermediate keyframes and videos, attributes each failure to the keyframes or to video generation, and revises the responsible stage. New experiments show that its repairs are effective and its choice of stage is correct (G1, G2):

  • Targeted repair beats untargeted alternatives. Repairing the stage the agent blames beats repairing the other stage by +0.32 OSR (G1, Test 1), and beats repeated sampling under an independent judge at the same candidate budget, as well as at matched GPU time (G2). Both margins grow on hard cases where the first candidate scores below 6 (+0.94 and +1.01 OSR).
  • The agent's attributions match human judgment. Blind human inspection of the coarse rendering, keyframes, and video agrees with the agent's stage choice in 18 of 20 sampled cases (G1, Test 2).

2. How the agent distinguishes keyframe vs. VDM failures (W2, Q1, Q2)

What we take from the agent's decisions is that the two causes the reviewer describes are separable by where a defect first appears: already in a keyframe, at keyframe timestamps despite correct keyframes, or only between keyframes, with the coarse rendering as the reference for geometry but not for appearance. The agent therefore compares the coarse rendering, nearby keyframes, and generated video at matching timestamps rather than judging the final video alone (G1).

3. When geometry and appearance references disagree, geometry wins (Q2)

Following Principle 1 in G1, we treat the edited 3D scene as the specification of the user's request. Following HorizonForge, the scene is reconstructed as Gaussian splats with inserted object meshes that define target placement and motion (Sec. 3.1). When the geometry and keyframes disagree on where an object is or how it moves, geometry wins—a keyframe whose object placement departs from the coarse rendering is treated as a keyframe defect and regenerated (G1), while keyframes supply appearance only. We take the inserted object meshes as given; fixing inaccurate 3D assets belongs to the upstream scene-editing stage (e.g., Articraft [3]). Geometric noise in the reconstructed Gaussian splats, by contrast, is handled by our VDM: following HorizonForge's cycle-consistent construction of training pairs, such artifacts appear in the coarse renderings the VDM is trained on.

4. Rare objects as a core driving bottleneck, beyond visualization (W3, Q3)

We agree that rare traffic interactions are an important part of the long tail. Following HorizonForge, our upstream 3D editing stage already supports modifying ego and surrounding-agent trajectories, and HorizonForgeV2 preserves and improves these edits: on the 109 HorizonSuite edits, which include ego and other-vehicle trajectory changes, OSR rises from 6.47 to 7.75 with gains across every edit type (Table 3, App. B.4 Table 11).

HorizonForgeV2 addresses the complementary bottleneck where prior driving generators break down: rare object categories [1,2]. In autonomous driving, rare objects are not merely visual variety—an animal, construction zone, or specialized vehicle on the roadway requires a different driving response than an ordinary car on the same trajectory, and a planner can react appropriately only once perception recognizes what the object is. Indeed, WOD-E2E [2] mines Waymo logs for scenarios occurring in under 0.03% of driving, and its long-tail clusters include special vehicles, construction zones, pedestrians, and foreign objects/animals—matching our benchmark's four categories.

Crucially, our setup removes a concrete data-collection bottleneck rather than merely visualizing edits: without driving videos of each rare class, rendering-conditioned models frequently morph rare objects into common vehicles (preserving the requested class in only 66% of boxes for HorizonForge vs. 94% for ours, G4). Removing this bottleneck translates directly into real-world perception gains on out-of-domain benchmarks across 3 seeds—raising AP on traffic cones (+0.080), road-scene animals (+0.106), and pedestrians (+0.033), while avoiding HorizonForge's degradation on large vehicles (G3).

5. Distinction from NOVA and prior keyframe-guided video editing (W3)

We do not claim sparse keyframe conditioning itself as new; our design differs from NOVA in both VDM training for driving and closed-loop recovery:

  • Open-world semantic import under multi-view keyframe conflict. Our keyframes import rare-object semantics absent from driving video priors. Because fast ego-motion causes large viewpoint shifts between sparse driving keyframes, edited keyframes, even when edited jointly, often disagree at the object level—going beyond the low-level blur and affine warps modeled in NOVA. We therefore train with real object-level edits (a keyframe replaced by an image-editor addition or removal, App. A.3) and anchor motion and structure to the coarse 3D rendering and depth. A single pass of our VDM already reaches 6.58 OSR, outperforming all keyframe-guided baselines including NOVA (≤\le 5.83, Table 2).
  • Recovering when keyframe generation itself fails. Any single-pass propagation model—NOVA included—propagates whatever keyframes it receives, so it cannot recover when they depict the wrong object or miss it, as often happens under large driving viewpoint changes. A quarter of our initial candidates score below 6 OSR (App. B.6, Table 14); our agent detects whether the keyframes or the VDM failed and repairs the responsible stage, raising OSR from 6.58 to 7.52.

6. Ethics review flag

The review sets "Flag For Ethics Review: Yes" without mentioning a specific concern in the text. We would be grateful if the reviewer could specify it, so that we can respond.

References

[1] Li, Kaican, et al. "CODA: A real-world road corner case dataset for object detection in autonomous driving." ECCV 2022.
[2] Xu, Runsheng, et al. "WOD-E2E: Waymo open dataset for end-to-end driving in challenging long-tail scenarios." CVPR 2026.
[3] Zhou, Matt, et al. "Articraft: An Agentic System for Scalable Articulated 3D Asset Generation." arXiv:2605.15187, 2026.

Reviewer_gKKWOpenReview ↗

final rating: 2 - weak rejection

The additional experiments strengthen the empirical case, particularly the independent-judge comparison and downstream detection results. The new fixed-action control also addresses my earlier concern about mixing attribution with repair action.

My remaining concern is contribution and scalability. Can the pipeline combine its existing editing operations into harder, controllable scenarios without a growing set of case-specific fixes and retries? That is where I'd expect the "agentic" to add real value. The current results demonstrate more reliable editing, but don't yet establish that compositional capability.

For further development, I'd encourage the authors to clarify what each repair must preserve and how reliability and repair cost change when several edits interact. The perception results already demonstrate useful data augmentation. Testing whether the generated scenarios also improve a driving policy on held-out corner cases would make the connection to broader driving utility stronger.

I recognize the practical value of rare-object appearance transfer and adaptive repair. However, I still find the advance over existing frame-guided editing incremental, without enough new modeling insight or a sufficiently large capability gain. I therefore maintain 2: Weak rejection, based on significance rather than an unresolved soundness issue.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/3): Stage Diagnosis (G1)

We thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers find the core design "a sensible split" that "avoids needing a new video dataset for every rare object" (gKKW), "a coherent solution" to "an important inconsistency and controllability issue" (d3ZM), and "a coherent and practically useful system contribution" that is "well matched to a concrete generalization bottleneck" (zEU7), and that all three find the paper clear and easy to follow. Four new experiments answer questions shared by several reviewers: G1 and G2 test the agent, and G3 and G4 test the data generated by the keyframe-conditioned VDM. We have uploaded a revised PDF. The new results are in Sec. 4.3, Sec. 4.4, and App. B.6 to B.10.

G1. How the agent decides which stage to repair, and whether it decides correctly (gKKW W1, W2, Q1, Q2; d3ZM W1)

A failed video can come from either stage: the edited keyframes may show the wrong object or placement, or the VDM may fail to follow or propagate good keyframes. To diagnose which stage failed, we give the agent a guideline, not a fixed rule, built on two principles. First, the coarse rendering is trusted for the object's position and motion, since it is rendered from the edited 3D scene that defines the request, but not for its appearance, which is crude (e.g., mesh-like surfaces and missing shadows). Second, the keyframes come before the VDM, and a VDM cannot recover from wrong keyframes, so the agent checks the keyframes first. After each render, the agent locates the defect and compares the coarse rendering, the nearby keyframes, and the generated video at matching timestamps. Typical cases are:

  • Visible in a keyframe (wrong class or shape, or a position that departs from the coarse rendering): the keyframes are at fault.
  • Keyframes correct, but the video departs from them at keyframe timestamps: the VDM is not following the keyframes.
  • Video matches the keyframes at their timestamps, but fails between them: the VDM fails to propagate.

These cases are examples, and the agent judges each defect case by case. When a defect remains ambiguous—for instance, when keyframes look plausible at a glance yet the rendered object lacks detail—the agent repairs one stage and, if the defect persists, switches to the other (Fig. 4: a blurry tow truck is first treated as a VDM issue and then as a keyframe issue, improving in each round).

To check whether the chosen stage makes a difference (Test 1) and whether the attribution itself is right (Test 2), we run two tests (Sec. 4.3, App. B.7):

  • Test 1: does the chosen stage matter? A repair round can raise the score simply because the agent tries again, so we check whether repairing the other stage would have done as well. For each of the 120 cases, we restart from the first candidate and replace the agent's action with a standard repair of the stage it did not blame (regenerating keyframes when it blamed the VDM, or strengthening VDM keyframe conditioning when it blamed the keyframes; details in App. B.7). The agent's own repair scores +0.32 OSR higher across all 120 cases, and +0.94 OSR higher on the 30 cases whose first candidate scores below 6, showing that stage selection matters most when the first attempt fails (Fig. 6 shows one case).
  • Test 2: is the attribution itself right? Test 1 measures the effect through scores; here we check directly whether the agent's diagnosis agrees with human judgment. We randomly sampled 20 decisions (10 per stage, covering all four categories). Blind to the agent's choice, a human inspected the coarse rendering, keyframes, and video and judged which stage caused the failure. Human judgment agrees with the agent in 18 of 20 cases (App. B.7; Fig. 7 shows one of these decisions with the agent's reasoning).
AuthorsOpenReview ↗

General Response (2/3): Comparison with Repeated Sampling (G2)

G2. The agent beats repeated sampling under the paper’s independent judge, and reaches the same OSR at lower cost (zEU7 W1, Q1; gKKW W1)

Because the in-loop verifier also defines OSR and candidate counts alone do not capture inspection or re-editing overhead, we compare the agent against untargeted repeated sampling under an independent judge and measure total GPU time and API cost (App. B.6). Both methods share the exact same first candidate and generate a total of 4 or 8 candidates per case across all 120 cases. Repeated sampling reruns the initial pipeline with fresh image-editing samples and video seeds without feedback; the agent revises either the keyframes or VDM settings based on the verifier's diagnosis. Each method retains the candidate scored highest by the in-loop verifier, and we evaluate the retained clips with two judges: OSR (the in-loop judge) and Qwen (App. B.1, an independent VLM judge unseen by either method). Each cell below reports the mean paired difference (Agent minus Repeated Sampling) over the 120 cases, with 95% confidence intervals from a paired cluster bootstrap over the 59 source scenes.

Candidate budgetΔ\Delta OSR (Agent −- Sampling)Δ\Delta Qwen (Agent −- Sampling)
4 candidates+0.25 [+0.09, +0.42]+0.12 [+0.03, +0.30]
8 candidates+0.34 [+0.18, +0.53]+0.20 [+0.03, +0.38]

All intervals exclude zero: the agent outperforms repeated sampling at both budgets under both judges. Importantly, overall averages dilute the gain because three-quarters of first candidates already score ≥6\ge 6 and require minimal repair; on the 30 failed initial cases (first-candidate OSR < 6), the agent leads repeated sampling by +1.01 OSR even at 8 candidates.
Compute and API cost. The table below reports per-case averages over the 120 cases on identical H100 GPUs. The first two columns compare the two methods at the same budget; the last gives repeated sampling the 8 candidates it needs to match the agent's OSR. GPU time sums all video-generation time per case; API cost sums all external model calls (image editor, verifier, planner, and executor).

Metric (per case)Agent (4 candidates)Repeated sampling (4)Repeated sampling (8)
OSR7.527.277.52
Qwen score8.288.178.14
GPU time (s)380314629
API cost (USD)0.1410.0850.177

At 4 candidates, the agent uses 21% more GPU time than repeated sampling. However, for repeated sampling to match the agent's 7.52 OSR, it requires 8 candidates—costing 66% more GPU time and 26% more API spend, while still trailing on the independent Qwen judge (8.14 vs. 8.28). At matched GPU time (6 agent candidates vs. 8 repeated samples), the agent still leads by +0.23 OSR [+0.06, +0.40].

AuthorsOpenReview ↗

General Response (3/3): Generated Data for Perception (G3–G4)

G3. Our clips are useful training data for long-tail categories, across 3 seeds (zEU7 W2, Q3; gKKW W3, Q3; d3ZM Q1)

Our original downstream test evaluated only cars under a single training seed. As the reviewers noted, a single seed does not capture training variance, and a car-only detector cannot reflect long-tail benefits because ordinary cars are already abundant in real driving logs.

To directly test long-tail categories and account for seed variation, we fine-tune a COCO-initialized Faster R-CNN across 3 random seeds on 8,000 real Waymo frames, alone or augmented with frames from the 60 generated clips of HorizonForge or of ours (now Sec. 4.4). Generated training frames are labeled automatically. All test images are real benchmarks outside Waymo: nuImages validation (front camera), and animal images from COCO and from Objects365 road scenes.

AP, mean ± std over 3 seedsReal only+ HorizonForge+ Ours
Pedestrian0.399 ± 0.0160.429 ± 0.0300.432 ± 0.015
Traffic cone0.220 ± 0.0150.292 ± 0.0340.300 ± 0.032
Car0.693 ± 0.0180.693 ± 0.0180.695 ± 0.007
Truck / bus / construction vehicle / trailer0.507 ± 0.0110.477 ± 0.0310.512 ± 0.024
Animal (COCO)0.622 ± 0.0580.637 ± 0.0480.673 ± 0.027
Animal (Objects365 road scenes)0.606 ± 0.0370.704 ± 0.0170.712 ± 0.033

Adding our clips gives the highest mean AP for all six class–dataset pairs. Over real data alone, the gains on traffic cones (+0.080) and Objects365 road-scene animals (+0.106) are well beyond seed variation, and pedestrians (+0.033) improve in every seed, whereas abundant cars stay unchanged (0.693 vs. 0.695). Against HorizonForge, the clear difference is trucks, buses, and construction vehicles, where HorizonForge clips lower AP below real-only training (0.477 vs. 0.507) while ours do not (0.512, ahead of HorizonForge in all 3 seeds).

G4. Frame-level class fidelity and agreement with inherited 3D bounding boxes (zEU7 W2, Q3; d3ZM Q1; gKKW W3, Q3)

When generated clips are used for perception training without re-labeling, labels are inherited directly from the edited 3D scene. OSR does not check these labels, so we measure agreement with them directly: we project the edited 3D bounding boxes onto four frames per clip across all 120 cases (480 frames per method). A blind VLM judge (gpt-6.1-sol) evaluates each projected box on two criteria: (i) whether the rendered object matches the requested semantic class, and (ii) how tightly it aligns with the projected 3D box from 3 (tight fit) to 0 (object missing) (App. B.10).

Metric (inserted-object boxes on 480 frames per method)HorizonForgeOurs
Correct semantic class66%94%
Box fit 3 (tight)56%68%
Box fit 2 (minor misalignment)28%21%
Box fit 1 (clearly offset / wrong size)11%5%
Box fit 0 (object missing)5%6%
Mean box fit (0–3)2.352.50

HorizonForge often renders a more common object in place of the requested rare category (Fig. 9) and keeps the requested class in only 66% of boxes, which would introduce severe class-label noise if labels were inherited directly. Our method preserves both the requested rare class (94%, +28 pp) and tight geometric alignment (68% tight fit vs. 56%; 89% with fit ≥2\ge 2), confirming that inherited 3D boxes remain reliable labels.

Dear Reviewers,

Thank you again for your reviews of this submission. The authors have now posted their responses, and the author–reviewer discussion period ends on November 18, 2026 (AoE).

Please read the authors' rebuttal and let them know whether it addresses your concerns. A short reply is enough, for example:

  • which of your concerns are now resolved;
  • which remain open, and what evidence or clarification would resolve them;
  • whether you plan to update your rating.

If the rebuttal changes your assessment, please update your review and rating before the discussion period ends. If you have already engaged, thank you, and no further action is needed unless the authors reply again.

Your participation is important for a fair and well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.