- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses instruction-guided video background replacement and foreground relighting. The core idea is Recast Residual Correction. For each denoising step, a forward branch produces the target-background and illumination proposal, a backward recasting branch places the edited foreground state back into the source background domain. The difference between this recast prediction and the analytically aligned source trajectory is treated as an estimate of the foreground degradation caused by the edit and added back to the target proposal. Both branches are operating modes of a shared Wan2.2 5B DiT, conditioned by Qwen3VL-Embedding-8B through learned semantic-, foreground-, and background-centric query tokens that replace the hand-crafted mask, edge, and depth conditions of FlowPortal. Evaluation is on Sparkle-Bench against Kiwi-Edit, Sparkle, Lumen, and FlowPortal, where the method is best on 8 of 9 metrics.
Critical strengths
- The recast idea is well motivated, and the delta over FlowPortal is unusually clearly drawn. FlowPortal's correction is a source-reconstruction residual that never observes the edited state, which is why it needs a foreground mask, Canny/HED/depth controls, and high-frequency transfer — exactly the constraints that suppress natural lighting responses. WorldPortal's recast branch observes the current edit and estimates its actual degradation. The gains also land where the mechanism predicts.
- The practical profile is attractive. Input is a single instruction, with no foreground mask, no source or target captions, and no hand-built structural conditions. Qualitatively, the outputs show lighting responses such as shadow direction, highlight placement, and color cast that the constrained training-free pipeline does not produce.
Critical weaknesses
- The backward recasting branch is trained on a background reference whose construction is never specified. The other conditioning slot has the same problem: at training the foreground reference is a clean paired video, while at inference it is the one-step clean estimate, which is close to noise early in sampling. The influence of this gap is not discussed.
- The correction is applied without ever being checked. Equation 13 adds the full-frame residual unmasked, at unit weight, and reused for five consecutive steps, where FlowPortal applies it under a foreground mask. Nothing measures whether the residual actually stays in the foreground region, and no ablation covers a residual weight or the reuse interval. The concern is that the residual is computed in the source domain but applied to a target-domain proposal. The difference measures degradation only if the recasting branch first removes the intended relighting; otherwise the correction brings source appearance back and undoes the relighting it was asked to produce. Nothing tests which of the two happens, and the ablation is too thin to decide it either, with only two variants and no VLM metrics.
Questions for authors
- What exactly generates the background reference, under which instruction, and with what filtering? Please also validate the recasting branch on its own: with no edit applied, does the residual go to zero, and how faithfully does the branch place a given foreground into a specified background domain?
- Report how much of the residual energy falls inside versus outside the foreground mask, and ablate masked versus unmasked correction, a residual weight, and the reuse interval. On instructions that do not entail a lighting change, does the residual still help, or does it start pulling the foreground back toward the source?
3 comments
Response to Reviewer G19M Part 2
Q4. Localization, masking, and residual leakage
The recasting branch is trained to combine a foreground reference with the specified background and illumination. Comparing its prediction with the aligned source trajectory is intended to suppress the desired domain change before estimating foreground discrepancies.
Our analysis finds 83.73% of residual energy inside the foreground mask and 16.27% outside. This supports spatial localization but does not, by itself, distinguish structural error from lighting differences or establish negligible background effects. We will specify the energy definition, aggregation, and mask area, and compare with under identical sampling settings.
| Correction | Detail ↑ | Identity ↑ | CLIP-T ↑ | VLM-Inst. ↑ | VLM-Quality ↑ |
|---|---|---|---|---|---|
| No correction | 30.862 | 0.551 | 0.240 | 3.896 | 3.831 |
| Foreground-masked | 33.211 | 0.752 | 0.253 | 4.015 | 3.960 |
| Unmasked | 33.344 | 0.757 | 0.252 | 4.018 | 3.975 |
The masked/unmasked pixel difference is 0.0038 over the full frame and 0.0031 over the background, measured as RGB MAE on [0, 1]. Since most of the residual energy is concentrated within the foreground mask and applying the mask produces negligible differences in the reported metrics, RRC can operate without an explicit foreground mask at inference, avoiding additional mask estimation while maintaining comparable editing performance.
Although most residual energy is concentrated in the foreground, as you noted, the residual may still carry illumination information from the source video. We claim that, most of the recast residual primarily compensates for structural and fine-detail degradation. We examine its energy distribution across spatial-frequency bands, ordered from low to high:
| Spatial-frequency band (low → high) | Share of total residual energy (%) |
|---|---|
| 0–10% | 0.43% |
| 10–20% | 0.91% |
| 20–30% | 1.37% |
| 30–40% | 1.85% |
| 40–50% | 2.64% |
| 50–60% | 3.18% |
| 60–70% | 4.06% |
| 70–80% | 5.29% |
| 80–90% | 17.46% |
| 90–100% | 62.81% |
The concentration of residual energy at high frequencies suggests that the residual primarily provides structural and fine-detail compensation, with limited transfer of source illumination to the target video.
Q5. Correction weight
Unit weight follows the condition-consistency principle underlying residual correction in FlowPortal [2]: adding the full residual restores the source-domain prediction to the original video trajectory. We evaluate with different s .
| Weight | Identity ↑ | Detail ↑ | VLM-Inst. ↑ | VLM-Quality ↑ |
|---|---|---|---|---|
| 0 | 0.551 | 30.862 | 3.896 | 3.831 |
| 0.5 | 0.706 | 32.549 | 3.971 | 3.947 |
| 1 | 0.757 | 33.344 | 4.018 | 3.975 |
| 1.5 | 0.765 | 33.425 | 3.941 | 3.925 |
An overly small may leave details insufficiently restored, whereas an overly large may impose excessive constraints on details and push the sampling trajectory out of distribution, reducing visual quality.
Q6. Refresh interval
The default interval , adopted from FlowPortal, is intended to balance correction freshness and computation. We compare refresh intervals under the same 50-step sampler. With forward CFG and no backward CFG, the DiT pass count is .
| Interval | DiT passes | Detail ↑ | Identity ↑ | Structure ↑ | VLM-Inst. ↑ | VLM-Quality ↑ | Average runtime per video(s) ↓ |
|---|---|---|---|---|---|---|---|
| 1 | 150 | 33.474 | 0.760 | 0.711 | 4.020 | 3.986 | 240 |
| 2 | 125 | 33.399 | 0.753 | 0.714 | 4.021 | 3.980 | 204 |
| 5 | 110 | 33.344 | 0.757 | 0.717 | 4.018 | 3.975 | 189 |
| 10 | 105 | 32.950 | 0.734 | 0.698 | 3.990 | 3.935 | 174 |
More frequent refreshes provide modest quality gains at a higher computational cost, while less frequent refreshes improve efficiency but may weaken correction. The default interval offers a balance between editing quality and efficiency.
Q7. Broader ablations and VLM metrics
We additionally evaluate more ablation variants using VLM metrics:
| Variant | VLM-Inst. ↑ | VLM-Const. ↑ | VLM-Quality ↑ |
|---|---|---|---|
| w/o RRC | 3.887 | 3.719 | 3.831 |
| w/o VLM conditioning and tokens | 3.549 | 3.478 | 3.504 |
| Foreground-masked RRC | 4.007 | 4.061 | 3.960 |
| Source-reconstruction residual | 3.993 | 3.989 | 3.936 |
| Full model | 4.018 | 4.082 | 3.975 |
References
[1] Jiang et al. VACE: All-in-One Video Creation and Editing. ICCV, 2025.
[2] Gao et al. FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement. 2025.
Thanks for the authors feedback. It is very helpful to see the details of background-reference constructions. The comprehensive ablation about different variants of the proposal and the hyper-parameters. My concerns have been addressed and I will keep my initial rating for acceptance.
Response to Reviewer G19M Part 1
We thank the reviewer for the thorough review, constructive comments, and valuable questions. We also appreciate the reviewer’s recognition of the well-motivated recasting mechanism, its clearly explained distinction from FlowPortal, the practical simplicity of instruction-only input, and the natural lighting responses demonstrated by our results. Below, we address each concern and answer all of the reviewer’s questions.
Q1. Background-reference construction
Due to space limitations, we described the reference's role in Appendix C.1 but did not fully explain how it was constructed. For each triplet, we construct the background reference using one of two methods, selected with equal probability:
Both methods produce a reference with the same background as the source video but a different foreground, providing the recasting branch with a reference for returning to the source background. We will include the detailed construction procedure in the revision.
During training, foreground variations are simulated through this preparation process, whereas at inference the source video itself serves as the source-domain reference. Although the references are obtained differently, the recasting objective is consistent: transferring the given foreground back into the specified source background. Our experimental results support the effectiveness of this approach.
Q2. Identity recast without editing
We conducted an experiment using the source video as both foreground and background reference, with the null prompt, and compare the recast proposal against the analytical source trajectory at matched timesteps. The residual norm is 0.006 RMS per latent element, compared with 0.041 RMS per latent element under actual edits, averaged per video over matched timesteps and then over the evaluation set. An accurate recasting model should produce a small identity residual, but exact zero is not guaranteed by a learned predictor. This measures the recasting-error floor.
Q3. Standalone recasting accuracy
We also evaluate recasting independently on held-out triplets with a known target video, using the original Identity, Structure, Detail, and VLM-Quality metrics.
In WorldPortal, the recasting branch primarily serves to model the degradation introduced during forward editing. Because forward editing and backward recasting share the same model parameters, the backward process exhibit degradation similar to that of the forward process. Computing a residual between the recast prediction and the original video in the source domain therefore provides a correction signal that helps recover structural details lost during forward editing.
Continuing response in the thread (References are included in the last part of the thread)