- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VIGEO, a unified video in context learning framework that formulates diverse video tasks as spatiotemporal analogy and canvas completion. The method is evaluated on camera control, modality guided generation, and video manipulation, including unseen tasks and modalities. The paper also identifies task internalization, where the query format can override the demonstration.
Critical strengths
- VIGEO provides a simple and general framework for handling diverse video tasks without task specific modules or textual instructions.
- The evaluation spans camera control, modality guided generation, and video manipulation, which helps demonstrate the flexibility of the proposed formulation.
- The evaluation includes unseen tasks and modalities, rather than focusing only on tasks observed during training.
- The study of task internalization is particularly useful because it reveals that prompt format can override the demonstration, and the authors further show that this shortcut can be mitigated with a small amount of unrelated data.
Critical weaknesses
- VIGEO succeeds on 7 of 12 unseen manipulation tasks and still fails on operations such as flipping and deblurring. This suggests that its generalization depends considerably on the training task distribution.
- Evidence for genuine analogy learning could be stronger. Additional controls using missing, incorrect, or shuffled demonstrations would better distinguish demonstration based inference from task recognition based on the query format.
- Some baseline comparisons are not fully controlled. In camera view transfer, VIGEO is trained on data from BridgeData-v2 and DROID, while XFactor has a different training distribution. This makes it difficult to attribute the performance gap entirely to the proposed analogy formulation.
Questions for authors
- How does VIGEO perform when the demonstration is removed, shuffled, or replaced with an incorrect demonstration? Such controls would help clarify whether the model is truly following the demonstration rather than recognizing the task from the query format.
- Could the authors clarify which evaluation tasks and modalities are completely absent from training in any form? This would help distinguish genuinely unseen modalities from unseen conditioning roles or task formulations.
- For camera view transfer, how much of VIGEO's advantage over XFactor remains under a more closely matched training distribution? Since VIGEO uses BridgeData-v2 and DROID during training, it is difficult to determine how much of the performance gap comes from the analogy formulation itself.
2 comments
Thank you for addressing my questions; most of them have been resolved. However, I hope the author will clarify their arguments in the final revised version. I am content to keep my original rating.
We thank the reviewer for the careful review and for the concrete suggestions on controls, which we have run.
W2 / Q1: Removed, incorrect, or shuffled demonstrations.
Please refer to General Response Q1 for the full response and experiments. These control experiments show that the demonstration determines which operation ViGeo performs.
Q2: Which tasks and modalities are absent from training?
We agree that "unseen" should be stated precisely. Every evaluation task is held out at the task level, and every evaluation clip comes from source videos disjoint from training. For the guidance modalities:
est_flow)est_point, tracks overlaid on RGB)Event data is therefore a genuinely unseen modality, while flow and point tracks are unseen as conditioning.
W1: Dependence on the training distribution.
We agree that generalization depends on the training distribution, and the correspondence above shows that this dependence is structured. Our results suggest that held-out tasks that recombine primitives present in training can transfer, as does an entirely new sensor (event data, rendered as images). Tasks that require an operation with no training analogue are more challenging: a global mirror reflection, or inverting a destructive process (deblurring). This matches our analysis in Sec. 4.3: in-context inference recombines what training provides, so the structural diversity of the training taxonomy matters more than its size.
W3 / Q3: Camera view transfer vs. XFactor under matched training data.
We agree that the training distributions differ; Sec. 4.2 notes that this comparison "may be unfair due to a difference in training domains", and we do not attribute the gap solely to the analogy formulation. On camera motion transfer on DL3DV, a domain in both models' training data, XFactor performs best (Table 3). XFactor is a dedicated pose model trained on four multi-view datasets (CO3Dv2, DL3DV, RE10K, MVImgNet), while ViGeo is one generic model with no camera-specific module. ViGeo is nonetheless competitive, and outperforms the dedicated video-level method CamCloneMaster in rotation and overall camera error.
Our claim is not that the analogy formulation beats dedicated methods under matched data, but that it lets a single generic framework absorb diverse camera tasks. Trained on synchronized multi-camera rigs, ViGeo infers large multi-axis viewpoint changes from one demonstration pair. This holds even when the demonstration and query come from different held-out environments, so the model cannot copy the environment from the demonstration (Table 6). Quantifying how much of the view-transfer gap remains under fully matched data would require fine-tuning XFactor on synchronized Bridge/DROID pairs and ours on additional multi-view datasets, which we leave to future work. We will revise Sec. 4.2 to state the claim in this scoped form.