- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CamRM combines video frames, a 3×3 spatial optical-flow summary, and a global image-motion estimate in a vision-language model to judge whether a clip matches a requested camera motion. The paper presents a tuned Gemini model, CamRM-G; a tuned Qwen model, CamRM-Q; and a tool-using variant without parameter updates, CamRM-A. These evaluators are used in a prompt-rewriting and regeneration loop. The test set contains 1,064 clips: 571 from Veo 3.1 or Omni and 493 from CameraBench. Table 1 reports 79.21% weighted average balanced accuracy for CamRM-G, 6.60 percentage points above the strongest non-author baseline at 72.61%.
Critical strengths
A simple design supported across model families. Providing inexpensive motion measurements as text to a VLM is a practical design choice. In Table 2, the untuned model improves from 68.67% balanced accuracy in the vanilla setting to 74.33% with the full cue configuration, so the gains cannot be attributed entirely to fine-tuning. In the matched Qwen SFT ablation, combined cues also outperform flow or global motion alone.
Critical weaknesses
Major concern: central results need a reconciled evaluation protocol. Figure 3(a) reports 53.8% post-loop alignment on 75 Veo I2V items, whereas Figure 3(b) and Table 3 report 61.3% for CamRM-G on the same page. This might reflect averaging across raters versus majority voting per item, but that distinction is not clearly explained. A more direct issue is that Appendix A.6 says the two pass@k strategies share their first candidate, yet Figure 4 shows different k=1 success rates: 54.5% versus 54.2% pooled, 40.6% versus 41.2% on Veo, and 68.9% versus 67.7% on Omni. With identical candidates and evaluation labels, these starting points should agree. The paper needs a unified account of the items, label aggregation, grading rounds, and item-level pairing, including the binary outcomes used for the McNemar tests. My concern is that the results cannot yet be reconciled, rather than that the gains necessarily disappear.
Questions for authors
Q1: Please reconcile the 53.8% in Figure 3(a) with the 61.3% in Table 3/Figure 3(b), and the differing k=1 rates in Figure 4 despite the shared first candidate. Do these results use the same items, grading rounds, and label-aggregation rules? Please provide item-level paired outcomes and update the significance tests where necessary. If the main gains hold after reconciliation, this would directly improve my assessment.
Q2: How exactly are geometric estimates converted into motion categories and binary alignment judgments? Please provide per-model parsing/reconstruction coverage and comparisons that handle failures consistently and use identical mappable subsets. If CamRM retains a clear advantage under this common protocol, it would address my main concern about the baseline comparison.
5 comments
Response to Reviewer QqiR (2/2)
Q2. How geometric estimates become categories and binary judgments; coverage; consistent failures; identical subsets
All geometry baselines share one rule-based decoder with no learned parameters, so they differ only in the camera poses they estimate, and each method sees the same frames as the VLM raters. The decoder turns the estimated trajectory into net rotation, depth-normalised translation and focal change. It then scores each of the 20 requested motions along its canonical axis with fixed geometric rules; compound motions such as orbits combine axes. A clip is accepted when the requested motion scores at least as high as every competitor, with one threshold fixed in advance (τ = 0) and never fitted on test data. Every RM answers the same request, so pan/truck and zoom/dolly stay distinct for all models, and VGGT's per-frame intrinsics give it the cue that separates a zoom from a dolly (App. A.18).
On coverage, CamRM-G and CamRM-A return a verdict for every clip. The VLM baselines parse at least 98% of clips, except Qwen3.6-27B (about 88%), and the Gemini and Claude raters return a verdict for at least 99.5% of the generated test clips. VGGT, CUT3R and MegaSAM reconstruct nearly every clip, and COLMAP about 75%. FilmOps has no tilt, pedestal or dolly-zoom class and is scored on the clips it can map (marked * in Table 1).
Because our raters have no missing verdicts, any common failure rule leaves their scores unchanged. Counting every missing verdict as an error, which is the rule Table 1 uses for geometry, can only lower the baselines. Dropping missing verdicts instead mainly helps COLMAP, the only geometry baseline with many failures, and even on the clips it reconstructs COLMAP stays far below every CamRM variant. On identical subsets, CamRM-G reaches 78.2% (Veo) and 71.2% (Omni) bAcc on FilmOps's mappable subset, versus 66.4% and 58.9% for FilmOps. CamRM therefore keeps a clear advantage under a common protocol. App. A.18 gives the full description and a per-model coverage table.
Presentation
We added a protocol table (App. A.16) that gives the items, generator version, grader setting, label aggregation and pairing of every human evaluation, and the Fig. 4 caption now states the generator version and how the shared first candidate is graded. Every headline number is scoped to its experiment.
Dear Reviewer QqiR,
Thank you again for your detailed review and for stating so clearly what would inform your assessment. As the discussion period draws to a close, we would like to briefly note how our responses and revised paper (changes in blue) address your points. We reconcile the 53.8% and 61.3% figures and the k = 1 rates in Fig. 4. We also re-grade both loop experiments with one common grader group. Under that group, the loop improves every set (p ≤ 0.002), and CamRM-A and CamRM-G remain the two best rating models. A new protocol table gives the items, aggregation, pairing and paired counts of every test. For the baselines, we document the geometry decoder and per-model coverage, and show that CamRM keeps a clear lead under the same failure rule and on identical subsets.
We would be glad to hear any further thoughts and are happy to address any remaining questions. Thank you once more for your valuable comments and your time.
Response to Reviewer QqiR (1/2)
Thank you for the detailed review and for stating clearly what would change your assessment. Below we reconcile every number the reviewer raised, add a common-grader evaluation, and update the tests. The main gains hold in every case.
Major concern and Q1 (a). 53.8% in Fig. 3(a) versus 61.3% in Table 3 / Fig. 3(b)
The two numbers come from two separate experiments. Fig. 3(a) measures the full system across four generated sets, and Table 3 compares seven rating models on Veo I2V. Each experiment was graded in its own round by its own group of graders, who focused on that experiment's comparison. The same 75 Veo I2V videos and the same before-loop baseline therefore received different absolute rates in the two experiments; every comparison inside an experiment uses one grader group throughout. The gap does not come from averaging versus majority voting: in both experiments, the after-loop rate is the mean of the graders' labels. Sec. 4 now describes this grading design.
To show that the conclusions do not depend on the grader group, we also evaluated both experiments with one common group of graders, who labelled the returned clips of every configuration in both experiments (App. A.16). Under this common protocol, the two experiments give the same value for CamRM-G on Veo I2V (56.7%). The loop improves every set:
In the rating-model comparison, CamRM-A and CamRM-G are again the two best rating models in the loop:
The gains are also significant under the paper's own grading: in Fig. 3(a), every set improves after the loop (paired sign-flip tests, p = 0.008, 0.048, 0.002 and < 0.001 for Veo I2V, Veo T2V, Omni I2V and Omni T2V).
Q1 (b). Fig. 4: different k = 1 rates despite the shared first candidate
At k = 1 both strategies show the same first generation, the identical file. The labels are not shared. Each strategy's eight candidates are graded together as one shuffled set, so that graders cannot infer anything from the order, and the shared video is therefore judged once inside each set. A few of these two judgments disagree, which moves the k = 1 rates by at most about 1% (54.2% vs. 54.5% pooled, 40.6% vs. 41.2% on Veo, 68.9% vs. 67.7% on Omni). That shift is well within the 95% intervals of Fig. 4 and an order of magnitude smaller than the k = 8 gap (88.3% vs. 78.4% pooled; McNemar 43 : 10, p < 10⁻⁵). Both strategies are graded in exactly the same way. With one label for the shared video, the loop still reaches 97.6% vs. 90.9% at k = 8 on Omni, and 81.2% vs. at most 66.5% on the Veo 3.1 study of App. A.6; p ≤ 0.001 for both. App. A.6 now states this explicitly.
Q1 (c). A unified account of items, label aggregation, grading rounds and item-level pairing
New App. A.16 adds one protocol table that covers every human evaluation in the paper: the items, what is graded, how labels are combined and which paired test is used. In short, each experiment is graded in its own round by its own group of graders, and every comparison inside an experiment is paired by item. For the loop experiments the per-item outcome is the label of the returned clip; for Fig. 4 it is whether an aligned clip appears among the first k candidates.
Q1 (d). Item-level paired outcomes and updated tests
Every test is paired by item. App. A.16 lists, for every test, the paired counts it uses, i.e. the items that one configuration gets right and the other does not; below are the ones quoted in the text. Table 3 rows use the paper's grading; common-grader rows give the counts of the two graders separately (grader 1; grader 2).