- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HorizonForgeV2 generates edited driving videos by separating object appearance from temporal propagation. An image editor supplies sparse keyframes, while a Cosmos-based video model, trained on imperfect keyframes and coarse rendering/depth controls, propagates those references throughout the clip. An agent inspects intermediate outputs and revises keyframes or generation settings within a candidate budget. Evaluation uses 120 long-tail insertions and 109 common edits. The full method reaches 7.52 and 7.75 OSR, respectively, compared with 6.94 and 6.49 for the strongest listed baseline, Difix3D. A blinded human study, a second model judge, component ablations, and a repeated-sampling control provide additional validation.
Critical strengths
-
The method is well matched to a concrete generalization bottleneck. Rare-object appearance is supplied by an image editor rather than inferred solely from the video generator's driving-data prior; coarse rendering and depth retain structural control. The model is trained on existing driving clips with perturbed references, avoiding the need to collect rare-object driving videos for each category (Section 3.2; Appendix A.3). Removing the auxiliary branches lowers initial OSR from 6.58 to 6.00, while removing keyframe augmentation lowers it to 6.45 (Table 5). This is a coherent and practically useful system contribution.
-
The evidence goes beyond the agent's own evaluator and includes a meaningful sampling control. The blinded human study favors the full system in 57.14% of samples, and the independent Qwen judge also ranks it first (Table 4; Appendix B.1, Table 9). Appendix B.6 compares adaptive revision with repeated sampling sharing the first candidate: at four candidates, scores are 7.52 versus 7.27, and the agent's advantage concentrates on initially difficult cases. These controls substantially strengthen the claim that the complete pipeline improves edit quality.
Critical weaknesses
-
The incremental benefit of the agent is less independently established than the quality of the complete system. The same verifier supplies feedback, selects the best candidate, and defines the OSR used in the agent-versus-sampling comparison (Sections 3.3 and 4.1; Appendix B.6). Independent judging and the human study compare the full system against other methods, but do not directly validate the relatively small 0.25-point advantage over four-candidate repeated sampling. Candidate counts are matched, while total compute is not: the agent adds inspection calls and can increase generation effort. No uncertainty intervals are given for this incremental gain. Thus the results support a useful pipeline more strongly than a precisely measured advantage of agentic diagnosis.
-
Evidence that better videos become useful long-tail driving data remains narrow. The downstream test is a single-camera, car-only detector, with an absolute AP increase of 0.0017 and no reported seed variability (Section 4.4). It does not test the pedestrian/animal categories or driving-policy behavior. Labels are inherited from the edited 3D scene even though generated appearance and geometry can drift; the five-frame OSR evaluation does not quantify framewise 3D-label fidelity (Section 4.1; Appendix A.4). These limitations do not negate the video-editing contribution, but they restrict the practical training and evaluation claims motivating it.
Questions for authors
-
Validation request: Does the four-candidate agent still beat four-candidate repeated sampling when the retained outputs are compared by the independent judge or blinded humans? Please include uncertainty clustered by source scene and report total video-generation time or GPU cost alongside API cost.
-
Clarification request: How are the 147 human-study samples drawn from the 120 long-tail cases, how many participants rated each case, and how are repeated ratings aggregated? This determines the independent sample size and the uncertainty of the reported win rate.
-
Validation request: Is the detector AP improvement consistent across training seeds, and how well do generated objects agree with the inherited 3D boxes? This would determine how much weight to place on the downstream-data contribution.
3 comments
As the author–reviewer discussion period ends on November 18 (AoE), we would greatly appreciate it if the reviewer could check our response, so that we can still address any further questions before the discussion closes.
In brief, we ran each requested validation: the independent-judge comparison with scene-clustered intervals and GPU/API cost (G2), the multi-seed, multi-category detector (G3), the agreement with inherited 3D boxes (G4), and the human-study protocol (our response, point 3).
If our response addresses the reviewer's concerns, we would be grateful if the reviewer could consider reflecting this in the rating.
Thanks the authors for providing detailed feedback. I will maintain my original rating.
We thank the reviewer for the detailed review, for recognizing "a coherent and practically useful system contribution", and for the concrete validation requests. We ran each one. Experiments shared with other reviewers are in the General Response (G1 to G4).
1. Independent-judge validation, scene-clustered CIs, and compute cost (W1, Q1)
We refer the reviewer to G2 (App. B.6), which answers each part of W1 and Q1: under the independent Qwen judge, the agent beats repeated sampling at both 4 and 8 candidates, with 95% intervals from a cluster bootstrap over the 59 source scenes, and G2 reports GPU time alongside API cost. Repeated sampling needs 8 candidates, at 66% more GPU time, to match the 4-candidate agent's in-loop OSR, and still scores lower under Qwen. The average gain is moderate because most first candidates already succeed; it concentrates on failed first attempts (OSR < 6), and G1 demonstrates that this gain comes from accurate stage diagnosis rather than untargeted retries.
2. Multi-seed detection, long-tail categories, and 3D-box fidelity (W2, Q3)
3. Human-study sampling and aggregation protocol (Q2)
Each of the 21 participants rated 7 cases sampled uniformly at random without replacement from the 120 long-tail cases, yielding 21×7=147 evaluation samples (~1.67 ratings per case on average, covering 88 unique cases). One sample corresponds to a single participant's blind side-by-side rating of all compared methods on one case. Table 4 averages uniformly across all 147 samples, splitting ties evenly in win-rate calculations.