acceptodds
Withdrawn from ICLR 2027

Towards Coherent Scene Reconstruction in Video Object Removal

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video object removal requires coherent scene reconstruction after eliminating unwanted objects and their associated effects. Cross-frame observations can reveal occluded content, yet mixed input representations, misleading cues, and uneven reconstruction support limit their effective use. We propose TIE, a framework integrating Type-Isolated Attention and Evidence-Guided Supervision for coherent scene reconstruction. Type-Isolated Attention organizes the original video, mask, and noisy target into distinct token streams and directs their interaction, maintaining target access to conditions while preventing noisy-target feedback into condition representations. Beyond accessibility, Evidence-Guided Supervision uses paired edited targets and frozen cross-frame matching to assess whether observations provide valid support for the scene after removal. The resulting evidence calibrates the allocation of complementary point-wise and region-wise auxiliary supervision, emphasizing precise matching where support is strong and local appearance and variation constraints where it is weaker. This allocation supplements dense reconstruction to account for differences in support across locations. Extensive experiments demonstrate that TIE achieves state-of-the-art overall performance, enabling more complete removal of objects and their associated effects, more faithful reconstruction of occluded content, and improved temporal coherence while preserving the surrounding scene.

Primary area:
generative models
Keywords:
Video Object Removal, Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_48VzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies coherent scene reconstruction for video object removal. TIE represents the source video, object mask, and noisy target as separate token streams. Its directed attention keeps the noisy target from updating the condition streams. During training, Evidence-Guided Supervision uses paired edited targets and frozen cross-frame matching to estimate where the source provides valid reconstruction evidence, then allocates point-wise and region-wise auxiliary supervision. At inference, the method uses only the source video, mask, and text. The paper evaluates TIE on PROVE-M and VORD-L, component ablations, and a 40-participant user study.

Critical strengths
  1. The problem and the information-flow design are well motivated. Unmasked source regions can still contain an object's shadows, reflections, or trails, so treating all visible source content as reliable background can harm reconstruction. Separating condition and target streams is an intuitive way to address this issue, and the EGS branch is used only during training.

  2. The complete system gives strong results on important measures. On PROVE-M, TIE reaches 28.9711 PSNR and 0.2490 RC-T, improving over the best listed comparison on both measures. The user study includes 40 participants and 30 clips; the pooled preferences favor TIE for removal completeness (79.00%) and visual consistency (77.69%). These results support a practical benefit on the tested samples.

Critical weaknesses
  1. The current ablations do not establish a fair quality-cost comparison for the three-stream design. In Table 3, changing from channel concatenation to token concatenation gives the largest PSNR increase, about 4.78 dB, while adding directed type isolation gives a much smaller change with mixed movement in other metrics. The paper does not fully specify the channel baseline's mask encoding, input projection, initialization or freezing, and shared training settings. The three streams also increase the token sequence from N to 3N, but matched FLOPs, peak memory, training time, and end-to-end inference time are not reported. The main gain is therefore hard to separate from the representation change and its extra cost.

  2. The added value of evidence-guided allocation is not fully isolated. The paper compares no EGS, geometric support, raw matching, and uniform point-wise supervision, but it does not keep both point-wise and region-wise losses and total auxiliary strength fixed while changing only the allocation rule. A matched constant or shuffled allocation is needed. Table 5 also shows that uniform point-wise supervision already matches the full model's RC-T of 0.249, while several reported margins are small and have no paired uncertainty. Separate shadow/reflection results and tests with moderate mask-boundary errors would make the EGS claim much stronger.

Questions for authors
  1. How was the channel baseline's input projection initialized and trained, and what training settings were used for each concatenation design? Does the advantage persist with comparable training compute? Please report the corresponding quality and cost measurements, including complete inference timing and any condition caching.

  2. On shadow and reflection cases, how much does full EGS improve over both the existing no-EGS baseline and a fixed mixture of the same auxiliary losses? Is the improvement stable across runs and when object masks have moderate boundary errors?

Reviewer_xrjLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to leverage available contextual information for video object removal (VOR) and how to guide models to develop this capability during training. Based on these insights, the authors propose TIE, which achieves competitive results.

Critical strengths
  1. I appreciate the authors’ insights into the use of cross-frame information, which is central to video object removal (VOR) and a key distinction from image object removal. Guiding models to determine when to exploit cross-frame evidence and when to infer missing content from within-frame context is a promising direction for improving VOR.

  2. The proposed model achieves state-of-the-art overall performance on PROVE-M.

Critical weaknesses
  1. The token concatenation strategy substantially increases the sequence length and, consequently, the computational cost of attention. Although Type-Isolated Attention restricts information flow through masking, this does not necessarily translate into efficient execution. I would expect TIE to incur substantially higher inference latency than comparably sized models such as EffectErase. However, the paper provides no comparisons of inference time or computational cost. Such measurements would help assess the practical trade-off between reconstruction quality and efficiency.

  2. The benchmark coverage is limited. The evaluation omits PROVE-H, which complements PROVE-M. Reporting results only on PROVE-M in Table 1 is insufficient as it contains just 80 video pairs and appears to exhibit a domain bias. Evaluation on more diverse and challenging benchmarks would make the conclusions more convincing.

  3. The implementation of the channel-concatenation baseline in Table 3 requires clarification. Its PSNR of 23.6559 is unexpectedly low, particularly given that EffectErase and Generative Omnimatte also use channel concatenation and achieve substantially better results. This discrepancy raises questions about whether the baseline is appropriately implemented and configured. Further implementation and training details are needed to support the comparison.

  4. The balance between exploiting cross-frame information and inferring missing content from within-frame context has previously been discussed in OmnimatteZero, yet the paper neither cites nor discusses this work. Generative Omnimatte is another important related method that is missing from the experimental comparisons.

  5. The number of decimal places reported for the same metric is inconsistent in Tables 2 and 3. Numerical precision should be standardized for clarity and readability.

Questions for authors

See the weaknesses above.

Reviewer_y8AeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TIE, a video object removal method in videos. The main contributions are 1. Type-Isolated Attention (limit the interaction of video and mask with noisy target) 2. Evidence-Guided supervision (supervision to reconstruct the background) Experiments on PROVE-M and VORD-L show improvement in object removal metric.

Critical strengths
  1. The problem of object removal while reconstructing the BG is important and well introduced.
  2. The qualitative examples seems to remove the object and associated effects well.
Critical weaknesses
  1. The comparison dataset, VORD-L does not seem to be publicly released. Only VORD (smaller version of VORD-L) seems to be available. The authors should clarify this.
  2. Comparison with previous works: Similar to type-isolated attention, Attention manipulation has been explored in previous object removal methods eg. OmnimatteZero (2025), Object-WIPER (2026), and Attentive Eraser (2025). The authors should compare, discuss and explain how their attention method is novel and different form the previous ones.
  3. Evaluation metric: ReMoVE was primarily introduced for image. Maybe a better metric would be Tok-Sim (Object-WIPER 2026).
  4. Impact of inpainting pretraining: The proposed method builds on Wan2.1-Fun InP, a pretrained video inpainting model. How much of its performance comes from this initialization?
  5. Limited reconstruction improvements: Despite the paper claims to improve coherent scene reconstruction, the reconstruction scores (PSNR,SSIM,LPIPS) compared to baselines are slightly better or even worse. These results do not consistently support the claimed superiority in reconstruction quality.

Minor weakness:

  • Paper is slightly difficult to follow. Maybe some terminology can be simplified. For eg. "TIE" does not clearly communicate the method's purpose. "Evidence-Guided Supervision" is somewhat vague.
Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.