acceptodds
Under review as a conference paper at ICLR 2027

What is this Object and Where is it Going?

Abstract

Video outpainting aims to extend content beyond the original frame boundaries while preserving inter-frame and intra-frame consistency. Existing methods have achieved promising results, but still fall short in complex scenes and motions. In particular, video outpainting inherently involves partially observed objects, posing significant challenges in maintaining object-level consistency in both appearance and motion. To address these issues, we propose RAVO, a retrieval-augmented test-time adaptation framework that complements incomplete object observations with relevant videos to support object-consistent video outpainting. However, di- rectly exploiting retrieved videos is non-trivial due to discrepancies in visual char- acteristics and dynamics between the retrieved and source objects. To bridge this discrepancy, we introduce Object-Specific Adaptation (OSA), which learns trans- ferable part-level correspondences between source and retrieved objects while retaining source-specific identity cues. Furthermore, conventional 3D attention tightly couples spatial and temporal interactions, limiting the effective incorpora- tion of heterogeneous object-level cues during adaptation. We therefore propose Spatial-Temporal Decoupled Adaptation (STDA), which separately adapts spatial appearance and temporal dynamics to mitigate their mutual interference. Beyond these methodological contributions, we introduce ObjectBench to address the lack of benchmarks for object-level evaluation, providing an object-centric assessment of appearance and motion consistency. Extensive experiments show that RAVO achieves state-of-the-art performance on standard benchmarks and consistently outperforms existing methods on ObjectBench, demonstrating broad efficacy.

Primary area:
generative models
Keywords:
Video Generation, Video Outpainting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DCqrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper focus on video outpainting when objects are only partially visible in the scene. The author proposes a retrieval-augmented test-time adaptation framework built on Wan2.1-VACE video generation model. It retrieves related videos, segments the reference object, and then converts sampled superpixel regions into reference tokens for appearance adaptation. And then adapts spatial and temporal attention heads in two stages using two LoRAs. The paper also introduces ObjectBench, which evaluates object appearance through DINOv2 features and motion through point trajectories.

Critical strengths
  1. The main idea is intuitive and well motivated. When the input shows only part of an object, retrieving related videos provides additional visual information that can help complete the missing parts.
  2. The qualitative results show more complete object structures and better motion consistency than the compared methods, particularly when objects are heavily truncated.
Critical weaknesses
  1. It is unclear how the ObjectBench is built. The paper explains how scores are calculated, but not how objects and crops are selected or how generated objects are segmented. It also does not say whether scoring includes the unchanged input regions, which could hide errors in the generated parts.
  2. The experiments do not fully show why the proposed designs work. The paper does not test whether its attention-head grouping is valid or whether sampling small object regions is better than using the whole object. Adding retrieval without decoupling also makes LPIPS and FVD worse than the baseline, which needs further explanation.
  3. Several implementation details and the processing cost are missing. Specifically, the RAG part might take much more computation and time.
Questions for authors
  1. Spatial-temporal head classification lacks direct validation against alternative or random groupings. Moreover, removing decoupling worsens LPIPS and FVD relative to the baseline. This raises questions about when retrieval helps.

  2. Figure 3 is difficult to connect to the method description and equations. The masking and supervision associated with “Random frame” are not shown clearly, and WarpFusion is not explicitly defined in the text.

  3. The implementation details and practical costs int the RAG part are missing. Head assignment, retrieval ranking, and background propagation are only briefly described.

  4. The treatment of multiple objects is unclear. The paper does not explain whether each object receives a separate reference or how reference features are associated with the correct source object.

  5. ObjectBench’s construction is underspecified. The paper describes its metrics but does not explain target-object selection, crop placement, or how generated-object masks are obtained. It is also unclear whether scoring covers the full object or only the generated regions.

  6. OSA randomly samples one SLIC region per adaptation step, without explaining why this should establish useful part-level correspondences. The treatment of irregular regions, including cropping, resizing, padding, and positional information, is missing.

1 comment

Thank you to the authors for their responses. As most of my concerns have been addressed, I have raised my score.

Reviewer_hW6eOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents RAVO, a retrieval-augmented test-time adaptation framework for video outpainting, with a particular focus on maintaining object appearance and motion consistency when the source video provides only partial observations of an object. RAVO retrieves semantically related videos and applies Object-Specific Adaptation (OSA) to incorporate part-level appearance cues from the retrieved references while preserving the identity of the source object. The method also introduces Spatial-Temporal Decoupled Adaptation, which separately adapts spatial appearance and temporal dynamics. In addition, the paper proposes ObjectBench for object-centric evaluation. Experiments on DAVIS and YouTube-VOS report consistent improvements over prior methods on both standard and object-level metrics.

Critical strengths
  1. Practical relevance. The paper addresses a practical limitation of video outpainting that arises when videos are reformatted for different aspect ratios or display formats. Objects near the frame boundary may be only partially observed, making it difficult to reconstruct their missing appearance while maintaining temporally consistent motion. RAVO directly addresses this case by incorporating external object references during test-time adaptation. This is a meaningful problem setting and targets a concrete failure mode of existing video outpainting methods.
Critical weaknesses
  1. Sensitivity to retrieval quality. A central component of RAVO is the retrieval of relevant object references, but the paper does not examine how sensitive the method is to the quality of those references. This is important because OSA is intended to handle differences between the source object and the retrieved object, yet the experiments do not show how far this discrepancy can increase before adaptation becomes ineffective or even harmful. For example, it would be useful to evaluate references with progressively lower similarity to the source object, including differences in appearance, pose, or motion, as well as clearly incorrect retrievals. Without such an analysis, it is difficult to assess how robust RAVO would be when the retrieval database has limited coverage or when the retrieval stage fails.

  2. Unclear applicability to general video outpainting scenarios. RAVO seems most applicable when a salient object is partially visible at the frame boundary. However, in many video outpainting scenarios, the region to be extended may consist mainly of background or scene content, with no relevant object available for retrieval or adaptation. It is not clear how RAVO handles such cases within a general outpainting pipeline. In particular, the paper should clarify whether the object-specific components can simply remain inactive when no suitable object is present, or whether a separate mechanism is needed to determine when these components should be applied.

Questions for authors
  1. Sensitivity to retrieval quality.
    How does RAVO behave as the similarity between the source object and the retrieved reference decreases? In particular, does OSA remain effective when the reference differs substantially in appearance, pose, or motion?

  2. Generalization beyond object-centric cases. How does RAVO perform when the outpainting region is dominated by background or scene content and no salient object crosses the frame boundary? Does the proposed framework provide consistent gains in these cases, or are its improvements primarily limited to object-centric outpainting?

Reviewer_F37EOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a retrieval-augmented test-time adaptation framework for video outpainting called RAVO. Its main contributions are: supplementing partially visible objects using retrieved videos; injecting sampled object-part features into attention; separately adapting spatial and temporal attention heads; and finally, an object-centric evaluation protocol combining appearance similarity and trajectory fidelity. Built on Wan2.1-VACE, RAVO reports improvements on DAVIS and YouTube-VOS.

Generally, this is a well-motivated paper with promising empirical results. Its strongest idea is to address missing object information explicitly through external references (Object RAG), while separating appearance adaptation from motion learning. However, the current results does not fully support the novelty of the specific design choices. The major concerns are insufficiently isolated component contributions, limited validation of the proposed benchmark, and incomplete reporting of the retrieval process and corresponding computational cost. The results support the effectiveness of the complete system, but stronger controlled experiments are expected to explain why it works and when it remains reliable.

Critical strengths
  1. The paper addresses an important limitation of video outpainting, where plausible backgrounds can coexist with distorted or inconsistent foreground objects. External object references via retrieval are a reasonable response to this problem.
  2. Using source queries with reference keys and values provides a plausible mechanism for incorporating additional appearance cues. Separately adapting appearance and motion also addresses a meaningful potential conflict.
  3. ObjectBench explicitly considers object appearance and trajectories, complementing metrics that can be dominated by background quality.
Critical weaknesses
  1. The proposed STDA builds on existing attention decoupling, while OSA combines established components. Comparisons with whole-object conditioning and prior decoupled fine-tuning on the same backbone are needed to clarify the methodological novelty.
  2. In Table 3, the ablation study separates retrieval and decoupling, but does not independently test part sampling, background propagation, the motion loss, or the head-classification strategy.
  3. The title of this paper is too implicit. Merelly looking at the title is difficult to know the topic, task or the contributions of this work.
  4. Some of the figure cations are too simple.
Questions for authors

See Critical Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.