W1. The pilot study is too limited to establish the repeatability and diagnostic value of the protocol
The value of a benchmark protocol depends on whether its measurements are repeatable and whether they support reliable diagnoses of model behavior. The current pilot study is limited in this respect.
(a) Scene and seed coverage. Each model is evaluated on two scenes with two seeds, and Tables 1 and 2 report only averages, without the variation across scenes or seeds. Since the paper presents scene dependence as a main finding, it is difficult to tell whether the observed patterns, such as the non-monotonic error and the cross-scene reversal (Sec. 4.2), are stable or would repeat on other scenes.
(b) Comparison between models. The two models are evaluated on different scenes, with different conditioning inputs (a video clip for WorldCam and a single image for SANA-WM), and SANA-WM is evaluated under positive yaw alone without a null condition. The authors acknowledge that the comparison is descriptive. However, the abstract reports that the two models show qualitatively different leakage patterns, and these differences may partly reflect the different scenes and inputs rather than the models themselves.
(c) Scope of the evaluation. The quantitative evaluation is limited to rotation, and naturalistic roaming is defined but not evaluated. Excluding unreliable translation scores is reasonable, but it means that the benchmark currently covers rotational controllability alone.
Why it is consequential: Without evidence that the measurements are repeatable and that model differences can be attributed to the models, it is unclear how reliably the protocol can be used to diagnose camera controllability, which is the main claim of the paper. (See Q1–Q2.)
W2. The measured effects may be affected by the evaluation setup
(a) Sequence construction. For WorldCam, the camera rotates for ⌊Φ/r⌋ frames and then holds the final pose for the remainder of the sequence, with the total length rounded up to a multiple of 4. For SANA-WM, the rotation ramps over the same number of frames before holding, and the total length is rounded up to a multiple of 8 (Sec. 3.4). APErot is averaged over all generated frames. Changing r therefore changes the number of rotating frames and, depending on how the total length is set, the proportion of holding frames. However, neither the total sequence length nor the number of holding frames is reported. With the current reporting, it is not possible to separate the effect of the rate from that of the sequence construction, although the joint dependence on amplitude and rate is the main finding.
(b) Resolution of the estimator. The calibration covers Φ∈{30∘,60∘} under positive yaw (Table 4), and the rates used in the calibration sequences are not stated, whereas the benchmark also uses Φ=10∘ and rates up to 10 deg/frame. In addition, about half of the leakage values in Table 2 are below 0.5°. The calibration reports the total geodesic error, which does not establish the per-axis estimation error relevant to these small leakage values. The static null control shows a low noise floor, but it does not characterize the accuracy on generated videos with motion or scene changes, as the authors note.
Why it is consequential: The main findings concern how the error and leakage change with amplitude and rate. If part of these changes comes from the sequence construction, or if small leakage differences cannot be resolved by the estimator, the conclusions about model behavior may not hold. (See Q3–Q4.)