acceptodds
Under review as a conference paper at ICLR 2027

Unifying Video Tasks via Spatiotemporal Analogy

Abstract

Adapting video models to new tasks typically requires dedicated data curation and fine-tuning. While visual analogy provides a training-free alternative by specifying tasks in-context, it remains restricted to the image domain. To explore whether analogy-based methods can unify diverse video tasks and generalize to out-of-distribution scenarios, we introduce ViGeo, a framework that extends visual in-context learning to the video domain via spatiotemporal canvas completion. Evaluated on a diverse task taxonomy with a strict train-test split, ViGeo generalizes to unseen video manipulations and zero-shot modalities (e.g., event cameras). Finally, we identify *task internalization*, where a query format associated with a pretrained task overrides the demonstration, and show that this shortcut can be removed with a small amount of task-unrelated data, highlighting the need to decorrelate prompt format from task identity.

Primary area:
generative models
Keywords:
video generation, visual analogy, spatiotemporal completion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5yrcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VIGEO, a unified video in context learning framework that formulates diverse video tasks as spatiotemporal analogy and canvas completion. The method is evaluated on camera control, modality guided generation, and video manipulation, including unseen tasks and modalities. The paper also identifies task internalization, where the query format can override the demonstration.

Critical strengths
  1. VIGEO provides a simple and general framework for handling diverse video tasks without task specific modules or textual instructions.
  2. The evaluation spans camera control, modality guided generation, and video manipulation, which helps demonstrate the flexibility of the proposed formulation.
  3. The evaluation includes unseen tasks and modalities, rather than focusing only on tasks observed during training.
  4. The study of task internalization is particularly useful because it reveals that prompt format can override the demonstration, and the authors further show that this shortcut can be mitigated with a small amount of unrelated data.
Critical weaknesses
  1. VIGEO succeeds on 7 of 12 unseen manipulation tasks and still fails on operations such as flipping and deblurring. This suggests that its generalization depends considerably on the training task distribution.
  2. Evidence for genuine analogy learning could be stronger. Additional controls using missing, incorrect, or shuffled demonstrations would better distinguish demonstration based inference from task recognition based on the query format.
  3. Some baseline comparisons are not fully controlled. In camera view transfer, VIGEO is trained on data from BridgeData-v2 and DROID, while XFactor has a different training distribution. This makes it difficult to attribute the performance gap entirely to the proposed analogy formulation.
Questions for authors
  1. How does VIGEO perform when the demonstration is removed, shuffled, or replaced with an incorrect demonstration? Such controls would help clarify whether the model is truly following the demonstration rather than recognizing the task from the query format.
  2. Could the authors clarify which evaluation tasks and modalities are completely absent from training in any form? This would help distinguish genuinely unseen modalities from unseen conditioning roles or task formulations.
  3. For camera view transfer, how much of VIGEO's advantage over XFactor remains under a more closely matched training distribution? Since VIGEO uses BridgeData-v2 and DROID during training, it is difficult to determine how much of the performance gap comes from the analogy formulation itself.
2 comments

We thank the reviewer for the careful review and for the concrete suggestions on controls, which we have run.


W2 / Q1: Removed, incorrect, or shuffled demonstrations.

Please refer to General Response Q1 for the full response and experiments. These control experiments show that the demonstration determines which operation ViGeo performs.

Q2: Which tasks and modalities are absent from training?

We agree that "unseen" should be stated precisely. Every evaluation task is held out at the task level, and every evaluation clip comes from source videos disjoint from training. For the guidance modalities:

modalityused as input (conditioning) in trainingused as output (estimation target) in training
event data✗✗ (absent from every training task)
optical flow✗✓ (est_flow)
point tracks✗✓ (est_point, tracks overlaid on RGB)

Event data is therefore a genuinely unseen modality, while flow and point tracks are unseen as conditioning.


W1: Dependence on the training distribution.

We agree that generalization depends on the training distribution, and the correspondence above shows that this dependence is structured. Our results suggest that held-out tasks that recombine primitives present in training can transfer, as does an entirely new sensor (event data, rendered as images). Tasks that require an operation with no training analogue are more challenging: a global mirror reflection, or inverting a destructive process (deblurring). This matches our analysis in Sec. 4.3: in-context inference recombines what training provides, so the structural diversity of the training taxonomy matters more than its size.


W3 / Q3: Camera view transfer vs. XFactor under matched training data.

We agree that the training distributions differ; Sec. 4.2 notes that this comparison "may be unfair due to a difference in training domains", and we do not attribute the gap solely to the analogy formulation. On camera motion transfer on DL3DV, a domain in both models' training data, XFactor performs best (Table 3). XFactor is a dedicated pose model trained on four multi-view datasets (CO3Dv2, DL3DV, RE10K, MVImgNet), while ViGeo is one generic model with no camera-specific module. ViGeo is nonetheless competitive, and outperforms the dedicated video-level method CamCloneMaster in rotation and overall camera error.

Our claim is not that the analogy formulation beats dedicated methods under matched data, but that it lets a single generic framework absorb diverse camera tasks. Trained on synchronized multi-camera rigs, ViGeo infers large multi-axis viewpoint changes from one demonstration pair. This holds even when the demonstration and query come from different held-out environments, so the model cannot copy the environment from the demonstration (Table 6). Quantifying how much of the view-transfer gap remains under fully matched data would require fine-tuning XFactor on synchronized Bridge/DROID pairs and ours on additional multi-view datasets, which we leave to future work. We will revise Sec. 4.2 to state the claim in this scoped form.

Thank you for addressing my questions; most of them have been resolved. However, I hope the author will clarify their arguments in the final revised version. I am content to keep my original rating.

Reviewer_rUy8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ViGeo, a video analogy framework where a demonstration input-output pair shows the task to be applied to a new query video. It extends ideas from image-based methods such as Visual Prompting to videos by arranging the demonstration and query into a video grid and generating the missing query output. The framework covers a range of tasks, including camera control, modality-guided generation, and video manipulation. Some evaluation tasks are excluded from training to test generalization, while camera control is evaluated on unseen videos and environments. The authors compare ViGeo with methods specifically designed for guided generation and camera control, and with image-based demonstration methods for unseen video manipulation tasks. The results show generalization to several held-out tasks, although the model struggles with deblurring and flipping.

Critical strengths
  1. The idea of extending visual analogy from images to a broad range of video tasks is interesting. The paper uses the same demonstration-based formulation for camera control, modality-guided generation, and video manipulation, without having to use separate modules for each task.
  2. The evaluation covers multiple task categories, and the comparisons with dedicated methods are convincing that the approach works. The results on event-guided generation and camera view transfer are also encouraging, showing that the same framework can handle different types of inputs and outputs.
  3. The experiments also show generalization to unseen tasks in spatial completion, jigsaw, temporal synthesis, and low-light enhancement. The authors clearly report the failures on deblurring and flipping, where copying the input (baseline) is comparable or better. Showing both successful and unsuccessful cases gives a clear picture of the current capabilities and limitations.
  4. The task-internalization analysis is also interesting. It shows how a familiar input format can cause the model to make mistakes because of patterns learned during training.
Critical weaknesses
  1. The comparisons on video manipulation tasks are limited to image-based demonstration methods and the copy baseline. Comparisons with dedicated methods, particularly for video restoration, would help understand where the proposed method stands on these tasks (similar to guided generation and camera control). Although the results already show poor performance on deblurring, they do not show how large the gap is compared to methods designed for this task. Reporting standard restoration metrics such as PSNR and SSIM alongside these comparisons would also be good.
  2. A broader analysis of how much the model relies on the demonstration across the evaluated tasks would have been useful. The task-internalization analysis provides some evidence, but for several tasks, the input itself can give a clue about the expected operation. Further qualitative (comparing correct/unrelated/no demonstrations while keeping the query fixed) or quantitative analysis would help clarify whether the demonstrations actually guides the model.
  3. A minor concern is that the benefit of using full video context is not clearly established. Although the framework processes and generates videos, tasks such as jigsaw, low-light enhancement, and horizontal flipping can be mostly solved at the frame level. Temporal interpolation shows positive evidence of temporal capabilities, but the failures on temporal deblurring and temporal flipping leave questions about their extent.
Questions for authors

Please address the concerns raised in the weaknesses above.

2 comments

We thank the reviewer for the thoughtful review and for recognizing the breadth of the evaluation and our reporting of failure cases.

W1: Dedicated baselines and restoration metrics.

Please refer to General Response Q2 for the full response. It now compares ViGeo with dedicated methods on all applicable manipulation tasks (ProPainter, RIFE, Restormer), reporting PSNR / SSIM / LPIPS and a per-sample success rate, as requested.

W2: Reliance on the demonstration.

Please refer to General Response Q1 for the full response and experiments. The results show that the demonstration, not the query alone, determines the operation ViGeo performs.

W3: Benefit of full video context.

We agree that some held-out manipulation tasks (jigsaw, low-light enhancement, horizontal flip) are mostly solvable per frame. Several other tasks cannot be solved frame by frame. Camera motion transfer requires reproducing a trajectory that unfolds over time. Flow- and point-guided generation synthesize an entire video from a single frame and a motion signal. Event-guided interpolation reconstructs intermediate frames from event data. We acknowledge that the failures on temporal deblurring and temporal flipping show that temporal operations without a training analogue do not yet transfer, and we will state this limitation explicitly in the revision.

Thank you for addressing my concerns. My major concerns (W1 and W2) are largely addressed. W3 remains unresolved, but I consider it a minor weakness; I will maintain my original score.

Reviewer_xtgGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends visual in-context learning to the video domain through spatiotemporal canvas completion. The model is trained on several template tasks and demonstrates generalization to unseen patterns. Extensive experiments validate the effectiveness of the approach. The authors also identify a "task internalization" phenomenon and show that incorporating data with additional patterns can mitigate it, encouraging the model to follow demonstrations and reducing shortcut learning.

Critical strengths
  1. The paper presents the first application of in-context learning to video editing and achieves strong experimental results.
  2. The paper identifies task internalization and mitigates it by introducing examples of new patterns.
  3. The framework supports a diverse range of patterns, covering most types of video transformations.
Critical weaknesses
  1. The evaluation would benefit from additional metrics, such as task success rate. Using only 1−MSE for the 12 MANIP tasks is insufficiently convincing. Complementary assessments by a video-capable multimodal LLM would help evaluate whether the outputs successfully fulfill the intended tasks.
  2. The analysis of task internalization relies on a single task, which limits the strength of the evidence. Additional cases across different tasks would help establish the generality of this phenomenon and the effectiveness of the proposed mitigation.
Questions for authors

See the weaknesses.

2 comments

We thank the reviewer for the positive assessment and constructive suggestions.

W1: Additional metrics.

Following the suggestion, General Response Q2 now reports PSNR / SSIM / LPIPS and a per-sample success rate for all 12 held-out tasks, together with dedicated baselines.

We additionally evaluated the outputs with a video-capable MLLM (Qwen3.5-27B). For each sample, the model is shown four videos: (1) the demonstration input, (2) the demonstration output, (3) the query input, and (4) a candidate output. It is asked: "Is Video 4 the result of applying the Video 1 → Video 2 transformation to Video 3? Answer Yes or No."

We validate it on each task with two candidates whose answers are known: the ground-truth target (Qwen should answer Yes) and an unchanged copy of the query (Qwen should answer No). We consider the judge reliable on a task if it answers both correctly for ≥ 80% of samples. Our result shows that the judge is reliable on 9 of 12 tasks, but not on temporal deblurring, temporal flipping, and sparse 4-frame interpolation. For example, in the temporal deblurring task, the model considers 49% of the "query copy" video as "correct" and 9% of the "GT target" video as incorrect. As a result, we find that an MLLM judge is itself not guaranteed to be reliable.

W2: Task internalization on a single task.

We respectfully clarify that the analysis covers two held-out tasks, depth→flow and flow→depth, and that both the phenomenon and the mitigation are reported for both.

The authors have addressed W1 well, but I am still not fully satisfied with their response to W2. I believe the final revision should include more cases of this kind to better characterize the phenomenon; otherwise, it may be viewed as an isolated occurrence specific to a particular task. Nevertheless, I remain willing to maintain my positive assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback.

We are glad that the reviewers found that ViGeo offers a simple and general formulation (5yrc) that unifies diverse video tasks in a single canvas without task-specific modules or textual instructions (rUy8, 5yrc), as the first application of in-context learning to video editing with strong results (xtgG). They noted that the evaluation is broad and goes beyond training tasks, covering held-out tasks and modalities (5yrc), that comparisons with dedicated methods on guided generation and camera control are convincing (rUy8), and that the results on event-guided generation and camera view transfer are encouraging (rUy8). Finally, all three reviewers found the task-internalization analysis interesting and useful (xtgG, rUy8, 5yrc).


Q1: Does ViGeo rely on the demonstration? [rUy8 W2; 5yrc W2, Q1]

We test this directly with demonstration controls: we keep the query identical and change only the demonstration row of the canvas. We consider three experimental setups.

Exp 1: Removing or nullifying the demonstration. For all 55 held-out canvases of low-light enhancement and 3D inpainting, we keep the query rows identical and change only the demonstration rows. We consider two ablations: no demonstration (the demonstration row is all black) and an identity demonstration (the demonstration target equals the demonstration source).

task (PSNR)correct demono demoidentity demo
low-light enhancement18.39.18.9
3D inpainting26.521.022.8

Removing the demonstration drops ViGeo close to the copy baseline on low-light enhancement, even though the dark query alone may suggest enhancement. On inpainting, the demonstration provides a 5.5 dB gain. Both tasks show that the demonstration determines the task.

Exp 2: Demonstration of a different task. On the same two tasks, we replace the demonstration with a jigsaw-apply demonstration: a clean clip (source) and its 3×3 tile shuffle by one of 10 fixed permutations π (target). We report the PSNR to the original task's ground truth, and how often the model instead performs the demonstrated shuffle, by matching each output to the query under the identity and all 10 permutations (chance of matching π: 1/11).

taskcorrect demo (PSNR)different-task demo (PSNR)outputs realizing exactly π
low-light enhancement18.37.385%
3D inpainting26.514.051%

On low-light enhancement, although the dark query suggests enhancement, 85% of outputs are the dark query shuffled by exactly the demonstrated permutation: ViGeo executes the demonstrated operation rather than the one suggested by the query. On 3D inpainting, 51% of outputs apply exactly the demonstrated permutation, an operation the query alone never suggests.

Exp 3: Same query, different demonstrations (jigsaw apply). To measure how precisely ViGeo follows the demonstration, we use jigsaw apply, where the query is an ordinary video and only the demonstration specifies which permutation to apply. We pair each of 55 queries with 10 demonstrations, each showing one of the 10 fixed permutations (550 canvases in total). 549/550 outputs realize the demonstrated permutation.

All the three experiments show the generation faithfully relies on the demonstration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.