acceptodds
Under review as a conference paper at ICLR 2027

A Benchmark for Camera Controllability in Interactive World Models

Abstract

Interactive world models increasingly require precise camera control, yet existing benchmarks rarely treat commanded motion amplitude and delivery rate as jointly controlled experimental variables, and few calibrate their pose estimator against ground truth to establish which conditions are reliable enough to score. We propose a benchmark protocol for camera controllability in world models that takes explicit continuous pose commands as input and evaluates models against a structured set of commanded trajectories: axis-pure motions across the rotational camera degrees of freedom and a null condition for measuring hallucinated ego-motion, paired with a calibrated measurement protocol that establishes when the underlying pose estimator can be trusted. We apply this protocol to two models, evaluating WorldCam across all three rotational axes and both command directions together with a null-drift test, and evaluating SANA-WM under positive yaw motion, finding that amplitude error varies with commanded angle but is non-monotonic in rotation rate, that controllability failures are scene-dependent, and that the two models exhibit qualitatively different leakage patterns despite operating under different conditioning regimes.

Primary area:
datasets and benchmarks
Keywords:
Camera controllability, World models, Benchmark, Pose estimation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7JbWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for evaluating camera controllability in interactive video world models. It receives SE(3) control signs and generates videos, and the camera poses are estimated by ViPE and compared with the ground truth camera signals.

Critical strengths

propose a method to evaluate the camera-controllability or command-following capabilities of interactive video world model precisely with explicit rotation tests.

Critical weaknesses
  1. the experiment is weak, only limited models and scenes are tested with a small part of metrics.
  2. lack of novelty in evaluating the camera controllability of interactive video generation models, only incremental improvement in testing the command-following capabilities on explicit rotations. However, the author didn't demonstrate qualitatively or quantitatively that why evaluating explicit rotations is more accurate than previous methods, which usually test models on whole trajectories.
Questions for authors

see weakness. More comprehensive experiments on the benchmark and also demonstrations of why such precise and explicit camera-control tests are better than previous benchmarks.

Reviewer_gfyROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for camera controllability in pose-conditioned interactive world models. It takes explicit continuous camera-pose SE(3) as input and evaluates models on a structured commanded-trajectory taxonomy, a null condition to measure hallucinated ego-motion, and naturalistic roaming. A unified metric suite (APE_rot, per-axis recovery/leakage MAE) is computed from relative-pose comparison, and the ViPE pose estimator is calibrated against engine-rendered ground truth so that models are scored only within its reliable operating range.

Critical strengths

The evaluation axis is well-motivated. Prior pose-conditioned benchmarks test only the direction and geometry of motion, they neither treat commanded amplitude and delivery rate as experimental variables, nor audit pose-estimator reliability before scoring. Treating motion amplitude as a first-class experimental axis and calibrating the pose estimator prior to evaluation are therefore genuine contributions.

Critical weaknesses
  1. The study is a pilot with very limited coverage. Only 2 models × 2 scenes are evaluated; only rotation is scored (translation is excluded); SANA-WM is tested on positive yaw only; the null condition is run for WorldCam only; and naturalistic roaming is defined but never evaluated. The empirical basis is too thin to substantiate a benchmark contribution at a main conference.
  2. Only 3 of the 6 commanded DOF are scored. Although the protocol accepts explicit SE(3) commands, quantitative evaluation is restricted to the rotational DOF, and pitch/roll/negative-yaw estimates are not verified against ground truth; the claimed input space is therefore only partially covered.
  3. The presentation would benefit from any figure, e.g., qualitative comparisons of generated vs. commanded trajectories, representative failure-mode examples.
Questions for authors
  1. Can the coverage be expanded? Such as more models, more scenes, together with the null condition and naturalistic roaming.
  2. How can translational controllability be recovered?
  3. Which additional figures will be provided (e.g., generated-vs-commanded trajectory comparisons, failure-mode examples, and a calibration/scoring pipeline schematic)?
Reviewer_tbSHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark protocol for evaluating camera controllability in pose-conditioned interactive world models. The protocol commands explicit camera poses and treats the commanded amplitude Φ\Phi and the delivery rate rr (degrees per frame) as experimental variables. It defines three trajectory types: axis-pure rotations (yaw, pitch, or roll), a null condition with no commanded motion to measure hallucinated ego-motion, and naturalistic roaming, which is left to future work. Camera poses of the generated videos are estimated with ViPE and compared with the commanded poses through the rotational absolute pose error (APErot\mathrm{APE}_{\mathrm{rot}}) and per-axis recovery and leakage errors. Before scoring, the authors calibrate ViPE against engine-rendered ground truth to identify the conditions under which it is reliable, and exclude translation from scoring due to inconsistent scale estimates. As a pilot study, they evaluate WorldCam across all rotational axes and directions with a null-drift test, and SANA-WM under positive yaw. They find that rotation error depends jointly on the commanded amplitude and rate, is non-monotonic, and varies across scenes, and that the two models show different leakage patterns.

Critical strengths

S1. Treating commanded amplitude and rate as controlled variables

Existing evaluations of camera control typically score a fixed set of trajectories. This paper instead sweeps the commanded amplitude and rate, and decomposes the error into recovery on the commanded axis and leakage onto the other axes, together with a null condition for hallucinated ego-motion. Table 1 suggests that this design can reveal behavior that a single condition would miss. For example, for WorldCam at Φ=60∘\Phi = 60^\circ, APErot\mathrm{APE}_{\mathrm{rot}} is 8.04° at r=2r = 2 but 19.69° at r=10r = 10, so evaluating at a single rate would give a very different picture of the same model.

Why it is consequential: Precise control of how much and how fast the camera moves is important for interactive world models, and the protocol provides a structured way to measure it. If the concerns in W1 and W2 are addressed, it could become a useful evaluation tool.

S2. Calibrating the pose estimator before scoring

Benchmarks that rely on off-the-shelf pose estimators rarely check whether the estimator is reliable under the evaluated conditions. This paper calibrates ViPE against engine-rendered ground truth, measures its noise floor under static input, and documents its failure modes, such as translation aligned with the camera's forward axis (App. A). Based on this analysis, it excludes translation from scoring rather than reporting unreliable numbers, and labels results outside the calibrated regime as exploratory.

Why it is consequential: This makes the measurement assumptions explicit, which is a good practice for evaluation papers and could be adopted by other benchmarks.

Critical weaknesses

W1. The pilot study is too limited to establish the repeatability and diagnostic value of the protocol

The value of a benchmark protocol depends on whether its measurements are repeatable and whether they support reliable diagnoses of model behavior. The current pilot study is limited in this respect.

(a) Scene and seed coverage. Each model is evaluated on two scenes with two seeds, and Tables 1 and 2 report only averages, without the variation across scenes or seeds. Since the paper presents scene dependence as a main finding, it is difficult to tell whether the observed patterns, such as the non-monotonic error and the cross-scene reversal (Sec. 4.2), are stable or would repeat on other scenes.

(b) Comparison between models. The two models are evaluated on different scenes, with different conditioning inputs (a video clip for WorldCam and a single image for SANA-WM), and SANA-WM is evaluated under positive yaw alone without a null condition. The authors acknowledge that the comparison is descriptive. However, the abstract reports that the two models show qualitatively different leakage patterns, and these differences may partly reflect the different scenes and inputs rather than the models themselves.

(c) Scope of the evaluation. The quantitative evaluation is limited to rotation, and naturalistic roaming is defined but not evaluated. Excluding unreliable translation scores is reasonable, but it means that the benchmark currently covers rotational controllability alone.

Why it is consequential: Without evidence that the measurements are repeatable and that model differences can be attributed to the models, it is unclear how reliably the protocol can be used to diagnose camera controllability, which is the main claim of the paper. (See Q1–Q2.)

W2. The measured effects may be affected by the evaluation setup

(a) Sequence construction. For WorldCam, the camera rotates for ⌊Φ/r⌋\lfloor \Phi / r \rfloor frames and then holds the final pose for the remainder of the sequence, with the total length rounded up to a multiple of 4. For SANA-WM, the rotation ramps over the same number of frames before holding, and the total length is rounded up to a multiple of 8 (Sec. 3.4). APErot\mathrm{APE}_{\mathrm{rot}} is averaged over all generated frames. Changing rr therefore changes the number of rotating frames and, depending on how the total length is set, the proportion of holding frames. However, neither the total sequence length nor the number of holding frames is reported. With the current reporting, it is not possible to separate the effect of the rate from that of the sequence construction, although the joint dependence on amplitude and rate is the main finding.

(b) Resolution of the estimator. The calibration covers Φ∈{30∘,60∘}\Phi \in \{30^\circ, 60^\circ\} under positive yaw (Table 4), and the rates used in the calibration sequences are not stated, whereas the benchmark also uses Φ=10∘\Phi = 10^\circ and rates up to 10 deg/frame. In addition, about half of the leakage values in Table 2 are below 0.5°. The calibration reports the total geodesic error, which does not establish the per-axis estimation error relevant to these small leakage values. The static null control shows a low noise floor, but it does not characterize the accuracy on generated videos with motion or scene changes, as the authors note.

Why it is consequential: The main findings concern how the error and leakage change with amplitude and rate. If part of these changes comes from the sequence construction, or if small leakage differences cannot be resolved by the estimator, the conclusions about model behavior may not hold. (See Q3–Q4.)

Questions for authors

I find the motivation and the focus on reliable measurement valuable. However, the current experiments are too limited to establish the reliability of the proposed benchmark. Each model is tested on two scenes with two seeds, the two models are tested under different conditions, and some of the reported effects may come from how the sequences are constructed or from estimator error. These issues affect the main evidence supporting the paper’s conclusions. Substantially broader validation appears necessary, and clarifying the experimental settings alone would not resolve all of these concerns. The questions below ask whether additional evidence or analysis can strengthen the current conclusions.

[R] = additional result requested, [C] = clarification.

1. Repeatability and model comparison (W1)

  • Q1 [R] (W1a). Could the authors evaluate more scenes and seeds, and report the variation across them (e.g., standard deviations or per-scene results)? Reporting per-scene signed errors on the commanded axis would also help. Tables 1 and 2 report only absolute errors averaged over scenes, so the overshoot and undershoot described in Sec. 4.2 cannot be verified from the tables. This would help determine whether the non-monotonic error and the cross-scene reversal are consistent patterns.
  • Q2 [R] (W1b). Could the authors compare the two models under positive yaw on a shared set of scenes, and report the null condition for SANA-WM? The difference in conditioning inputs would remain, but this would reduce the confound from different scenes in the comparison of leakage patterns.

2. Measurement setup (W2)

  • Q3 [R/C] (W2a). What are the total sequence length and the number of holding frames for each condition and model? Could the authors report APErot\mathrm{APE}_{\mathrm{rot}} separately for the rotating and holding segments, or the error at the end of the rotation, so that the effect of the rate can be separated from that of the sequence construction?
  • Q4 [R/C] (W2b). Which rates were used in the calibration sequences, and how accurate is ViPE at Φ=10∘\Phi = 10^\circ and at high rates? Could the authors also report per-axis calibration errors on sequences with motion, so that the small leakage values in Table 2 can be interpreted relative to the estimator's resolution?

Suggestions (not affecting the assessment)

  • A recent arXiv paper, Probing into Camera Control of Video Models (CamProbe, arXiv 2605.14815), also probes how video models respond to camera motion of different strengths, although it targets general I2V/T2V models. A brief discussion of how the proposed protocol differs from it would be helpful.
  • The value of COND_FRAMES (L157) is not given in the text. Reporting it would help reproducibility.
  • The YXZ Euler order is stated, but specifying whether the decomposition is intrinsic or extrinsic, along with the camera axis and sign conventions, would help reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.