acceptodds
Under review as a conference paper at ICLR 2027

Beyond Dense Context: Efficient Video Inpainting via Position-aware Context Extraction

Abstract

Video inpainting is inherently asymmetric, as only the masked region requires synthesis, while the non-mask region primarily provides context. Existing efficient methods either summarize non-mask content into spatially coarse global features or estimate dense spatial features, creating a trade-off between computational efficiency and fine-grained context preservation. To understand which context is needed, we analyze masked-to-non-mask attention in a dense baseline. We find that masked queries concentrate their attention on a small subset of non-mask tokens, particularly those near the mask and in semantically informative regions. Motivated by this observation, we introduce PACE, a video inpainting framework that extracts compact, position-aware context. PACE assigns sparse spatial anchors to a small set of learnable queries. Through attention that incorporates both content and relative spatiotemporal position, these queries aggregate non-mask information from across the video into compact context features. An adaptive attention fusion mechanism then dynamically balances the extracted context with mask-side information. By compressing dense non-mask tokens into compact context representations, PACE accelerates masked-region denoising while preserving the contextual information needed for high-quality inpainting. Experiments on DAVIS, YouTube-VOS, and AdErase, our new advertising video benchmark for text and logo removal, demonstrate strong inpainting quality. PACE outperforms the efficient baselines EditCtrl and YOSE in quality and achieves an average – speedup over them across different mask ratios.

Primary area:
generative models
Keywords:
Video inpainting, Efficient video diffusion, Context compression

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3EDJOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACE, an efficient video inpainting framework that exploits the asymmetric roles of masked and non-mask regions. Instead of propagating the full non-mask token sequence through the DiT, PACE compresses the known non-mask trajectory into a small set of position-aware context slots and combines them with masked-region features through adaptive attention fusion. The method is motivated by an analysis showing that masked queries concentrate attention on a relatively small subset of non-mask tokens. Experiments on DAVIS, YouTube-VOS, and the proposed AdErase benchmark show improved quality over efficient baselines while achieving faster inference.

Critical strengths
  1. The method directly exploits the asymmetry between masked-region synthesis and non-mask context processing.
  2. PACE improves over EditCtrl and YOSE while remaining competitive with full-video methods, with a reported 1.7×–2.1× average speedup.
  3. Evaluated on AdErase, DAVIS, and YouTube-VOS, covering both advertising removal and public video inpainting benchmarks, and matches or exceeds the efficient baselines across the reported metrics.
Critical weaknesses
  1. Limited evaluation of the synthesized regions. PSNR, SSIM, and LPIPS are computed only outside the removal mask, while the remaining quality metrics are evaluated on the full video. This is relevant because many masks are small, with 55.9% of analyzed frames below an 8% mask ratio. A controlled benchmark with clean masked-region targets would provide more direct evidence of completion quality.
  2. The ablations do not fully isolate the proposed design choices. The context ablation removes the entire branch, including positional information, and the calibration ablation evaluates the count correction and learned residual jointly. The results therefore support the complete modules, but do not show how much the spatial anchors or individual calibration terms contribute. More targeted ablations would make the attribution clearer.
Questions for authors
  1. In Fig. 5, the labels "w/o PACE" and "w/o adaptive fusion" are used to denote the two ablation studies, which is inconsistent with the table on the left. Although the authors provide an explanation in the main text, the notation is confusing at first glance, why not use consistent terminology?
  2. After increasing the slot budget from K=91 to K=390, some metrics such as LPIPS actually worsen slightly, can you explain why?
2 comments

Thank you for your thoughtful and constructive feedback. We are encouraged that you appreciate our method design, favorable efficiency–quality trade-off, and comprehensive experimental evaluation. Below are our responses to your concerns.

W1: Clean-target evaluation

(1) AdErase covers a broad range of mask-area ratios. We sampled videos evenly across four mask-area bins: 0–10%, 10–20%, 20–30%, and 30–40% of the frame area. The cited 55.9% statistic describes the online source footage, not our test set. We will clarify this sampling strategy in the dataset description.

(2) The supplemented clean-target evaluation supports PACE's reconstruction quality. We construct 500 input–target pairs from clean online videos by synthetically masking regions and retaining the original videos as clean targets. In addition to the original metrics, we report PSNR, SSIM, and LPIPS within the masked regions. PACE outperforms efficient baselines EditCtrl and YOSE on all metrics. PACE also achieves higher masked PSNR and lower masked LPIPS than the full-video methods MiniMax-RM, ROSE, and VACE. Although ProPainter achieves higher masked PSNR and SSIM, its substantially worse masked LPIPS (0.247 vs. 0.196) is consistent with its blurrier textures compared with PACE’s sharper details in Fig. 4.

MethodNon-mask PSNR ↑Non-mask SSIM ↑Non-mask LPIPS ↓Motion Smoothness ↑Aesthetic Quality ↑Imaging Quality ↑Masked PSNR ↑Masked SSIM ↑Masked LPIPS ↓
ProPainter28.340.9040.0880.9780.4350.70025.930.7420.247
DiffuEraser27.780.8990.0650.9770.4470.70523.880.6820.246
MiniMax-RM30.970.9320.0450.9790.4420.70824.770.7070.218
ROSE29.680.9210.0550.9810.4430.70223.490.6680.247
VACE31.050.9320.0470.9790.4450.70824.210.6950.214
EditCtrl31.170.9310.0510.9790.4420.70620.600.6040.321
YOSE31.000.9290.0490.9790.4390.70323.050.6460.281
PACE (Ours)31.990.9360.0420.9790.4430.70924.910.7020.196

W2: Additional ablation study

(1) Spatial anchoring improves the reported scores while retaining the context branch. The original Table 3 reports w/o context branch. In this rebuttal, we add w/o spatial anchors, which removes the positional encoding. Removing spatial anchors lowers Mask-Image Quality from 0.468 to 0.461, whole-video Imaging Quality from 0.570 to 0.558, and non-mask PSNR from 33.63 to 33.30 dB. These results support the contribution of query anchoring to local image quality and background preservation in this setting.

SettingNon-mask PSNR ↑Non-mask SSIM ↑Non-mask LPIPS ↓Motion Smoothness ↑Aesthetic Quality ↑Imaging Quality ↑Mask-Image Quality ↑
w/o spatial anchors33.300.9060.0700.9740.4000.5580.461
Full PACE33.630.9070.0690.9740.4050.5700.468

(2) The combined count and residual correction achieves the best or tied results on all seven metrics. The total correction is β = β_cnt + δ. The count term β_cnt = log(N_nm/K) compensates for token-count imbalance, while the learned residual δ provides a learned bias and a content-dependent adjustment. The original Table 3 reports w/o calibration (β=0). In this rebuttal, we add w/o count term (β_cnt = 0) and w/o learned residual (δ = 0). Removing either term lowers the performance. The combined configuration performs best overall in this comparison, supporting the use of both terms together.

SettingNon-mask PSNR ↑Non-mask SSIM ↑Non-mask LPIPS ↓Motion Smoothness ↑Aesthetic Quality ↑Imaging Quality ↑Mask-Image Quality ↑
w/o count term (β_cnt = 0)33.190.9060.0710.9740.4020.5640.464
w/o learned residual (δ = 0)33.080.9050.0720.9740.4000.5620.464
Full PACE33.630.9070.0690.9740.4050.5700.468

Q1: Table labels

We will use “w/o context” and “w/o calibration” consistently in Fig. 5 and Table 3 for clarity.

Q2: Analysis of slot budget

The perceptual declines are small: non-mask LPIPS rises by 0.001, and Aesthetic Quality and Imaging Quality fall by 0.002 and 0.003, respectively. A plausible explanation is that the extra slots encode finer details of the same background, where masked queries already focus on a small set of relevant cues (Fig. 2). Those details may reduce pixel error while slightly altering local textures near the mask boundary during decoding, which can degrad LPIPS even as PSNR improves.

Thank you for your response and the additional experiments. I have no further concerns. I will increase my rating.

One small suggestion: If possible, it would be better to include a few visual examples from the new clean-target evaluation, with the ground truth for comparison.

Good luck!

Reviewer_uaGrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PACE, which speeds up diffusion-transformer video inpainting for small masks such as subtitles by processing only masked tokens and compressing the background at each block. Its first core component is a context extractor whose 91 learnable, position-anchored queries summarize the noisy background latent and its target velocity into slots. Its second core component is a fusion rule that merges attention over mask tokens and slots using a count-based bias plus a learned correction. Built on Wan2.1-Fun 1.3B with LoRA, PACE is tested on DAVIS, YouTube-VOS and a new 450-clip advertising benchmark against EditCtrl and YOSE fine-tuned on the same data. It matches or exceeds both on every reported metric, with average speedups of 1.7x to 2.1x for mask ratios from 10% to 80%.

Critical strengths
  1. The insight and design are intuitive. On 50 DAVIS clips with the unmodified Wan2.1-Fun model, all mask tokens plus the top 10% of background tokens receive about 85% of the masked queries' attention (Figure 2). This motivates 91 learned, position-anchored background slots per block.
  2. PACE is faster than both efficient baselines and never trails them on any metric. It outpaces EditCtrl and YOSE at every mask ratio from 10% to 80% (Figure 1b). It matches or exceeds both on all metrics for AdErase, DAVIS and YouTube-VOS (Tables 1-2).
Critical weaknesses
  1. No metric isolates the filled region, so the compact-context claim lacks direct support. Fidelity scores (PSNR, SSIM, LPIPS) exclude the mask and VBench rates whole videos, and on these metrics even the variant without any context branch beats fine-tuned EditCtrl in five of six YouTube-VOS cases (Tables 2-3).
  2. The configuration of the main baselines is underspecified. EditCtrl and YOSE are only described as fine-tuned on the same training data, without the starting checkpoint, fine-tuning steps or the sampling settings used for the quality tables. Since the quality comparison rests on these two models, it remains unclear whether they were tuned as carefully as PACE.
Questions for authors

None.

2 comments

Thank you for your thoughtful and constructive feedback. We are encouraged that you appreciate our intuitive method design, computational efficiency, and strong experimental results. Below are our responses to your concerns.

W1: Effectiveness of compact context

(1) Additional masked-region quality evaluation further supports the benefits of compact context. In this rebuttal, we apply the Imaging Quality score (Huang et al., 2024) to masked-region crops and denote it as Mask-Image Quality. With the same backbone adaptation and inference settings, adding compact context raises this score from 0.459 to 0.468. This local gain accompanies improvements in all six original metrics, demonstrating the effectiveness of compact context.

Method / VariantNon-mask PSNR ↑Non-mask SSIM ↑Non-mask LPIPS ↓Motion Smoothness ↑Aesthetic Quality ↑Imaging Quality ↑Mask-Image Quality ↑
w/o context branch32.260.9010.0800.9730.3940.5460.459
Full PACE33.630.9070.0690.9740.4050.5700.468

(2) Our ablation isolates the contribution of compact context. EditCtrl builds on Wan2.1-VACE and uses an additional Transformer control branch alongside the denoising backbone. In contrast, PACE builds on Wan2.1-Fun and concatenates conditioning inputs directly into a single denoising backbone, avoiding the computational overhead of a separate control branch. Because the two methods differ in both pretrained backbones and conditioning architectures, comparing EditCtrl with PACE without compact context cannot establish the source of their quality difference. We therefore evaluate the contribution of compact context through an ablation within PACE, keeping the pretrained backbone and conditioning architecture fixed.

W2: Baseline configurations

We follow each model's public defaults, with a common fine-tuning dataset and evaluation budget to support a fair comparison. EditCtrl starts from Wan-AI/Wan2.1-VACE-1.3B, YOSE from zibojia/minimax-remover, and PACE from Wan2.1-Fun-V1.1-1.3B-InP. Each model is fine-tuned for 6,000 steps with an effective batch size of 64. For evaluation, all three methods use 49-frame clips at 480 × 832, 25 denoising steps, and the same test videos and removal masks.

Thanks for your response! I have no further concerns and I will keep my positive rating.

Reviewer_hFSXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACE exploits the asymmetry of video inpainting by propagating only masked tokens and compressing known-background states and flow velocities into a small set of context slots using spatially anchored queries. A token-count correction and learned residual calibrate mask/context attention fusion. Across three datasets, the reported metrics match or improve on EditCtrl and YOSE, with average speedups of 1.7×–2.1× in a controlled timing configuration.

Critical strengths
  • reasonable design: the design separates context extraction from masked-region synthesis, replacing dense non-mask tokens with K slots while retaining access to background content across frames.

  • clear mathematical interpretation: the branch fusion is expressed as sigmoid mixing of log-normalizers and shown to equal adding a shared bias to all context logits.

  • good performance: in Tables 1–2, PACE matches or exceeds EditCtrl and YOSE on every reported metric, covering non-mask preservation and whole-video perceptual quality. For example, YouTube-VOS PSNR is 33.63 versus 31.31 (EditCtrl) and 32.77 (YOSE).

Critical weaknesses
  • More ablations required: Removing the entire context branch in Table 3 does not distinguish the effects of context availability, spatial anchors, and 3D RoPE; Table 4 varies only the slot budget. In addition, the uncalibrated variant removes both the count term and learned residual, leaving their contributions unresolved.

  • The relation between metrics and real performance: PSNR, SSIM, and LPIPS are measured exclusively outside the mask, while the other metrics assess the whole video. For small subtitles and logos, these scores may be dominated by unchanged background. Equation (13) also preserves the original non-mask latents by construction. Better preservation therefore does not independently establish better synthesized content. The qualitative examples are useful, but insufficient to settle the central completion-quality claim.

Questions for authors

see Weaknesses

============================================

Thanks for authors' responses. I am willing to keep my initial score.

1 comment

Thank you for your thoughtful and constructive feedback. We are encouraged that you appreciate our method design, theoretical formulation, and experimental results. Below are our responses to your concerns.

W1: Component ablations of spatial anchors, the count term and learned residual

(1) Spatial anchoring improves the reported scores while retaining the context branch. The original Table 3 reports w/o context branch. In this rebuttal, we add w/o spatial anchors, which removes the positional encoding. Removing spatial anchors lowers Mask-Image Quality (MUSIQ on masked-region crops) from 0.468 to 0.461, whole-video Imaging Quality from 0.570 to 0.558, and non-mask PSNR from 33.63 to 33.30 dB. These results support the contribution of query anchoring to local image quality and background preservation in this setting.

SettingNon-mask PSNR ↑Non-mask SSIM ↑Non-mask LPIPS ↓Motion Smoothness ↑Aesthetic Quality ↑Imaging Quality ↑Mask-Image Quality ↑
w/o spatial anchors33.300.9060.0700.9740.4000.5580.461
Full PACE33.630.9070.0690.9740.4050.5700.468

(2) The combined count and residual correction achieves the best or tied results on all seven metrics. The total correction is β = β_cnt + δ. The count term β_cnt = log(N_nm/K) compensates for token-count imbalance, while the learned residual δ provides a learned bias and a content-dependent adjustment. The original Table 3 reports w/o calibration (β=0). In this rebuttal, we add w/o count term (β_cnt = 0) and w/o learned residual (δ = 0). Removing either term lowers the performance. The combined configuration performs best overall in this comparison, supporting the use of both terms together.

SettingNon-mask PSNR ↑Non-mask SSIM ↑Non-mask LPIPS ↓Motion Smoothness ↑Aesthetic Quality ↑Imaging Quality ↑Mask-Image Quality ↑
w/o count term (β_cnt = 0)33.190.9060.0710.9740.4020.5640.464
w/o learned residual (δ = 0)33.080.9050.0720.9740.4000.5620.464
Full PACE33.630.9070.0690.9740.4050.5700.468

W2: The relation between metrics and real performance

(1) The additional masked-region evaluation shows competitive local image-quality scores for PACE. We apply the Imaging Quality score (Huang et al., 2024) to masked-region crops and denote it as Mask-Image Quality. Compared with efficient baselines, PACE exceeds EditCtrl and YOSE on all three datasets. Compared with full-video methods, PACE ranks second on AdErase and DAVIS. On YouTube-VOS, PACE is close to ROSE and VACE. These crop-based scores provide additional evidence of competitive local image quality, alongside the whole-video results.

MethodAdEraseDAVISYouTube-VOS
ProPainter0.5590.4920.444
DiffuEraser0.5690.5020.461
MiniMax-RM0.5680.5110.462
ROSE0.5610.5090.470
VACE0.5770.5500.479
EditCtrl0.5700.4900.464
YOSE0.5540.5050.452
PACE (Ours)0.5730.5240.468

(2) The original AdErase benchmark covers diverse mask sizes, so its whole-video scores also provide evidence of completion quality. The original test set samples videos evenly across four mask-area bins: 0–10%, 10–20%, 20–30%, and 30–40% of the frame area. Larger edits therefore form a substantial part of the evaluation, reducing the risk that background dominates the whole-video quality comparison.

(3) PACE's background preservation is a design advantage, while the added masked-region evaluation provides evidence of local image quality. Full-frame generation may alter content outside the intended edit. Eq. (13) limits such changes by combining generated mask-region latents with the original non-mask latents. Because the VAE jointly decodes this assembled grid, the generated latents can still influence background pixels near the mask boundary. Preserving non-mask latents therefore does not guarantee exact pixel-level preservation. Following YOSE (Wu et al., 2026a), we use non-mask PSNR, SSIM, and LPIPS to measure the remaining differences between the output background and the input video. These metrics quantify background preservation but do not directly evaluate the synthesized content. We therefore additionally report Mask-Image Quality on masked-region crops to assess local image quality in the edited areas.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.