acceptodds
Withdrawn from ICLR 2027

RolloutRemover: Scribble-Guided Object Removal via Latent Rollout Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Object removal aims to remove target objects and recover the background they occlude. Existing methods have achieved strong visual fidelity, but they still rely on mask-conditioned inpainting and therefore require an accurate object mask as input. In practice, however, users often provide only a casual scribble, while current pipelines either rely on limited scribble generalization or introduce external models to bridge the gap. We present **RolloutRemover**, a video diffusion framework for scribble-guided object removal. Instead of directly predicting the final removed image, RolloutRemover interprets the scribble as removal intent and performs removal through a structured latent rollout that explicitly disentangles object corruption from background restoration, providing stronger supervision over the removal process. We also introduce **SGORBench** with multiple mask sets for comprehensive evaluation of image object removal. Experiments show that RolloutRemover achieves strong removal quality and is substantially more robust to sparse user-provided scribbles than prior mask-dependent pipelines.

Primary area:
generative models
Keywords:
video diffusion model, image editing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y8KKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RolloutRemover, a scribble-guided object removal method built on Wan2.1-T2V-1.3B with LoRA fine-tuning. Image removal is cast as generating a 14-frame latent sequence: a scribble-intent frame, a predicted removal-region mask (f2), the source frame, a 10-frame latent "rollout", and the object-removed target. The rollout is built analytically in latent space. Corruption frames interpolate from the source latent to a noise anchor, in which the removal region is replaced by Gaussian noise; restoration frames then interpolate from the anchor to the target (Sec. 3.2, Eqs. 3–8). The model is trained with flow matching on about 50K pairs from OBER and CORNE. The authors also introduce SGORBench: 300 samples, each with a procedurally generated scribble, an object mask and an object-effect mask.

Critical strengths
  1. Practical problem: users give rough scribbles, and the model jointly infers the removal region and result.
  2. Ablations of rollout order, f2f_2 and endpoint-only prediction show that ordered intermediate supervision helps.
Critical weaknesses
  1. Scribble comparison is unfair: test scribbles share the training generator, while baselines simply use scribbles as masks.
  2. With dense masks it trails OSOR and OmniPaint (Table 2), so the rollout's added benefit is unsupported.
  3. γ\gamma, KcK_c and KrK_r are ablated on the 300-sample test set, in single runs without variance.
  4. The Wan2.1-T2V-1.3B backbone choice is unjustified, and training steps or epochs are never reported.
Questions for authors
  1. How exactly were the SAM 2.1 masks in Table 10 produced (prompt type, number of points, mask selection, dilation)? What is their IoU against the ground-truth masks? Why do all baselines do worse with SAM masks than with the raw scribble?
  2. Were γ\gamma, KcK_c and KrK_r selected on the SGORBench test set? Can you report the ablations on a held-out split with multiple seeds and confidence intervals?
  3. Why was Wan2.1-T2V-1.3B chosen over other video backbones (e.g., CogVideoX, HunyuanVideo) or an image backbone such as FLUX? How many training steps or epochs were used?
Reviewer_nWNyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RolloutRemover for scribble-guided object removal from a single image using a LoRA-adapted Wan2.1-T2V-1.3B video diffusion backbone and a structured latent rollout. Specifically, RolloutRemover is trained on approximately 50K OBER/CORNE image pairs and introduces SGORBench, containing 300 samples from OBER, RORD-Val, and CORNE sources with aligned scribble, object, and object-effect masks. Experimental results include quantitative comparisons using ground-truth removal pairs and a user study.

Critical strengths
  • RolloutRemover aims to reduce the cost of preparing precise dense masks at inference time. In Table 1, it outperforms existing removal models fine-tuned with object masks when they are directly given scribble prompts.

  • The new benchmark, SGORBench, adds scribble prompts to existing samples from OBER, RORD-Val, and CORNE, providing aligned scribble, object, and object-effect masks for each sample.

Critical weaknesses
  • [W1] Weak Novelty. Predicting an intended removal mask from scribble prompts does not by itself justify a new learned removal framework over a simple combination of interactive segmentation (e.g., SAM 3 [A]) and an existing object remover (e.g., OmniPaint/OSOR in Tables 1/2). Although Appendix F already compares frozen SAM 2.1 pipelines, the scribble-to-prompt conversion, mask selection, and IoU of the intended-object/effect masks are unspecified. It is also unclear that most baselines degrade substantially with SAM 2.1 in Table 10, despite competitive results in Tables 1/2. For the daytime examples with visually distinct targets, such degradation is unexpected and requires evidence explaining whether it arises from segmentation failures or mask handling. Please compare a state-of-the-art segmentation pipeline (e.g., SAM 3 [A]), separate intended-object segmentation errors (e.g., binary IoU) from effect inference/removal and background restoration, and clarify the removal-specific contribution beyond existing interactive segmentation, scribble segmentation, and coarse-to-fine mask refinement alternatives [A, B, C, D].

  • [W2] Task Importance. Scribble prompts may reduce the effort of drawing dense masks for target objects, but their practical advantage over point/box prompts remains unclear. Both failures in Figure 2 reflect misinterpretation of sparse or incomplete guidance, which may be addressed by interactive segmentation (e.g., [A]). Appendix G shows robustness to human-drawn scribbles with similar synthetic scribbles but does not compare input time, stroke/click counts, or specific labeling time in practical scenarios. Please compare removal success and unintended edits under matched interaction budgets, including point/box prompts followed by interactive segmentation, to establish the practical advantage of scribble guidance for object removal.

  • [W3] Benchmark Details. The proposed SGORBench contains only 300 samples selected from existing OBER/RORD-Val/CORNE datasets, leaving its additional value beyond scribble annotations unclear. Please report the sample-selection criteria, correspondence to official splits, and results on standard OBER-Test/RORD-Val evaluations. In particular, evaluate difficult object-effect attribution, such as distinguishing target shadows from dark backgrounds that resemble object effects or from unrelated shadows at night. Please quantify intended-object and effect-region localization and connect these errors to removal failures. Common metrics such as PSNR, SSIM, and LPIPS alone do not establish whether the video diffusion model correctly interprets scribble intent or identifies object-associated effects.

  • [W4] Unclear Training and Inference Costs. Appendix E fine-tunes OmniEraser/SmartEraser, but the adaptation details (e.g., costs and trainable parameters) relative to RolloutRemover are unclear. Table 12 in Appendix J shows 78.83s per image (RolloutRemover: video-diffusion-based) versus 0.29~15.13s for image-diffusion-based baselines. Please report design choices (or matched Wan2.1-1.3B backbone with existing removal methods), trainable parameters, training cost, and LoRA-rank trade-offs, and compare end-to-end accuracy and cost under matched adaptation budgets.

References:

[A] SAM 3: Segment Anything with Concepts, ICLR 2026.

[B] ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image, ECCV 2024.

[C] SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement, ICLR 2025.

[D] SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process, NeurIPS 2023.

Questions for authors

Please see the Weaknesses for details.

  • Q1. [W1] What causes baselines to degrade substantially when paired with SAM 2.1 in Table 10 compared to Tables 1/2? When compared against a state-of-the-art segmentation pipeline (e.g., SAM 3 [A]), what is the removal-specific contribution when separating intended-object segmentation errors (e.g., binary IoU) from effect inference and background restoration beyond existing interactive segmentation and mask refinement [A, B, C, D]?

  • Q2. [W2] Under matched user interaction budgets, comparing input time, stroke/click counts, and labeling effort against point/box prompts paired with interactive segmentation (e.g., [A]), what evidence demonstrates the practical advantage of scribble guidance in mitigating previous failures from sparse inputs (Fig. 2) and reducing unintended edits?

  • Q3. [W3] What are the sample-selection criteria and correspondences to official splits for SGORBench (300 samples), and how does the method perform on standard OBER-Test and RORD-Val benchmarks? Can the authors provide object and effect-region localization results (especially for distinguishing target shadows from dark backgrounds or unrelated night shadows) to directly connect unwanted localization errors to removal failures beyond PSNR, SSIM, and LPIPS?

  • Q4. [W4] Given the inference latency gap in Table 12 (78.83s for video-diffusion-based RolloutRemover vs. 0.29~15.13s for image-diffusion baselines), what are the trainable parameters, training costs, and LoRA-rank trade-offs? Under matched adaptation budgets (or a matched Wan2.1-1.3B backbone), how does RolloutRemover compare against adapted baselines (e.g., OmniEraser/SmartEraser) in end-to-end accuracy and cost?

Reviewer_nic5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows new solution, a scribble-guided object removal framework built using aa pretrained video diffusion model. The method aims to find the intended removal region from sparse user-provided scribbles without requiring an accurate object mask. The main contribution is a structured latent rollout that divides object removal into two phases: object corruption and background restoration. The intermediate states are constructed through latent-space interpolation using a noise anchor, and the resulting sequence provides additional supervision during training. The framework also jointly predicts a removal-region frame and the final object-removed image. The authors introduce SGORBench, a benchmark containing 300 samples with scribble, object, and object-effect masks. They evaluate the method against five existing object removal approaches using quantitative metrics and human evaluations. The proposed method achieves strong results under scribble guidance, and additional ablation studies investigate the effects of rollout construction, frame ordering, and the two-phase design.

Critical strengths
  1. Practical and well-motivated problem formulation: The paper addresses an important limitation of conventional mask-conditioned object removal; the difficulty of obtaining accurate object masks from users' random interactions. Showing scribbles as signs of removal intent rather than exact editing regions is a meaningful problem formulation. The proposed two-phase latent rollout is relatively simple to implement and provides a structured way to supervise intermediate editing states.
  2. Good empirical results under scribble guidance. The proposed method achieves the best results across all six reported metrics on the scribble-mask setting of SGORBench (Table 1). The human evaluation also indicates a substantial preference for RolloutRemover over the evaluated baselines. Importantly, Appendix G evaluates the method using human-annotated scribbles, where its performance remains largely safe with the generated scribble setting. This provides useful evidence that the enhancements are not entirely dependent on the synthetic scribble-generation procedure.
Critical weaknesses
  1. Insufficient evidence isolating the contribution of the proposed rollout mechanism. Although the reported results are promising, the experiments do not fully establish whether the improvements arise specifically from the proposed corruption-restoration process or more generally from additional intermediate supervision in a pretrained video diffusion model. Several ablations, including the endpoint-only and phase-removal experiments (Appendix C), change the sequence length with the component being evaluated. So, these experiments do not fully isolate the effect of the intermediate supervision from differences in sequence length and model computation.
  2. Limited evidence of a favorable quality-efficiency trade-off and incomplete comparability with existing methods. The performance advantage is strongest under scribble guidance on SGORBench, while the method does not consistently outperform existing approaches on the object-mask setting or OmniPaint-Bench (Tables 2 and 3). Moreover, the main comparisons involve methods with different architectures, training procedures, and conditioning strategies. Although the additional fine-tuning experiments in Appendix E are useful, they do not fully establish comparable training budgets across all methods. The computational cost is also substantial. Appendix J reports an inference latency of 78.83 seconds for RolloutRemover, compared with 0.29 seconds for OSOR. TaylorSeer reduces the latency to 22.78 seconds, but the resulting method remains considerably slower while the results are not that much impressive to ignore latency.
Questions for authors

Q1. In the ablations of Appendix C, removing intermediate frames or individual rollout phases also changes the overall sequence length. Could the authors clarify how much of the improvement is attributable specifically to the corruption-restoration targets rather than additional temporal tokens or auxiliary supervision? In particular, is there evidence from a sequence-length-matched baseline with alternative intermediate targets?

Q2. The framework supervises the removal-region frame, but all unknown frames are jointly denoised at inference time. Could the authors report the segmentation quality of the predicted removal region, and clarify whether the final generation actually depends on the correctness of this intermediate prediction?

Q3. In Section 4.1, the authors state that perceptually similar scenes were excluded through image similarity and manual verification. Could the authors specify the exact similarity metric or feature representation used, the similarity thresholds, and the manual verification protocol? How many candidate samples were excluded through this process?

My current point is Weak Reject. However, I find the problem relevant and the reported results promising. I am open to revising my point to Weak Accept if the authors provide convincing clarifications and supporting evidence that address the main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.