- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript presents RefRestore, a framework for adverse-weather image restoration that uses an unaligned clean image of the same scene as supplementary information. Its two main mechanisms are Content-Addressed Reference Injection (CARI), which removes explicit positional encoding from reference tokens, and Evidence-Gated Routing (EGR), which suppresses unreliable degraded-image keys based on estimated local degradation severity. Combined with severity-gated perceptual regularization and mismatch-calibrated training, these mechanisms aim to support selective reference use without explicit geometric alignment.
The manuscript also introduces RefWeatherTrain-456 and RefWeatherTest-20, alongside diagnostics for reference use and potential misplacement. On RefWeatherTest-20, RefRestore achieves the best scores among the compared methods on five of six no-reference quality metrics and the second-best score on the remaining metric. Ablations and qualitative comparisons indicate reduced reference-copying behavior relative to naive reference conditioning. These results support improved perceptual quality and more controlled reference use, while the absence of aligned clean ground truth limits conclusions about the factual correctness of reconstructed details.
Critical strengths
The manuscript proposes using unaligned reference images of the same scene to perform severe adverse weather restoration tasks, which has practical significance. The method provides a clear design approach for reference-assisted restoration from the perspectives of spatial bias and insufficient observation information: CARI is responsible for eliminating the explicit influence of reference coordinates, and EGR is responsible for suppressing the degenerate context that is estimated as unreliable. Furthermore, the manuscript is well-structured, logically sound, and of high quality.
Critical weaknesses
Some qualitative results altered the still recognizable background structure in the input, such as the sky in the fourth row of Figure 3. Therefore, the existing experimental results make it difficult to determine when the model uses a reference, when it relies on generative priors, and whether it can preserve recognizable background content in the input. This directly impacts the manuscript's core objective of selectively utilizing reference scene information for image restoration.
Questions for authors
1.The authors mentioned system prompt, user prompt, and optional region-consistency quarantine in Figure 2, but I observed that the authors did not mention these in the experimental setup. Please supplement the relevant information.
2.RefRestore recovers the main content well, but some backgrounds are different from the reference image and also change the structure that is still recognizable in the input. It is difficult to determine whether these changes come from the reference information or the generation prior. The discard and replacement strategies are insufficient to explain how the model selects information sources and whether it can retain visible content in the input. The authors are requested to address this issue in detail.
3.Increasing τ strengthens the suppression of degenerate keys, and RefUse and RefMisplace also become smaller compared to the no-reference baseline. However, this does not seem to determine whether the model more accurately transfers reference information or reduces the overall influence of reference information.
4.RefMisplace is marked as uparrow in Table 2 but as downarrow in Table 3. Please unify the direction. In addition, an increase in RefUse may come from incorrect copying, while correctly restoring occluded content may also increase RefMisplace. Please explain how to interpret RefUse and RefMisplace together, and how to distinguish between erroneous migrations and correct restoration. 5.Regarding the effectiveness of CARI and EGR, current ablation experiments only discuss the combined effect of the two and cannot demonstrate their individual contributions. It is suggested that the authors provide further clarification on this matter.
4 comments
Response to Questions from Reviewer
We thank the reviewer for this observation. We understand the concern regarding background alterations and the ambiguity of whether these changes stem from the reference or the generation prior. We interpret the reviewer’s mention of 'discard and replacement' as referring to our Evidence-Gated Routing (EGR) mechanism.
EGR uses channel variance as a proxy for evidence strength to dynamically route attention:
- Evidence-Rich (High Variance): Regions with sharp details route attention to the degraded input, preserving observable structures.
- Evidence-Poor (Low Variance): Regions obscured by fog or smooth textures route attention to the reference to synthesize missing details.
To demonstrate that the model actively selects and rejects mismatched reference content rather than blindly replacing the input, we provide ablation with an idiosyncratic reference in Appendix A.4 Figure 10. The aim of the ablation is to prove reference utilization.
Response to rebuttal
Thanks for the authors' rebuttal. Part of my concerns are addressed, but the evidence to support the main claim of this work is not yet sufficiently established, as mentioned in weakness "the existing experimental results make it difficult to determine when the model uses a reference, when it relies on generative priors, and whether it can preserve recognizable background content in the input. "
Thus, I keep the initial rating.
Response to Questions from Reviewer
We thank the reviewer for their continued engagement and for acknowledging that our rebuttal has addressed part of their concerns. While we respect their decision to maintain their initial rating, we hope the following explicit mapping of our experimental evidence to their core concern will clarify the model's behavior for the Area Chair and the broader community.
To make this determination completely transparent, we have summarize our ablation in the revised manuscript to explicitly isolate and visualize these three distinct behaviors:
- Identifying when the model uses the reference:
The model uses the reference when the degraded input is evidence-poor (low channel variance, e.g., heavy fog or snow) and the reference provides semantically compatible details to fill that void. The Idiosyncratic Reference Test (Figure 10) proves this condition. Unique reference objects (e.g., the balloon) appear in the output only when the corresponding input region is heavily degraded (evidence-poor), proving the model actively retrieves reference content specifically to resolve information loss. This is further quantified by the Attention Mass Visualizations (Figure 9), which provide mechanistic proof of degraded queries attending to reference keys.
- Identifying when the model relies on generative priors:
The model relies on the generative prior when the input is evidence-poor, but the reference is either unhelpful, globally mismatched, or actively blocked to prevent semantic leakage. The Region-Consistency Quarantine (RCQ) ablation (Figure 5) explicitly triggers this condition. When RCQ blocks the reference for the winter trees to prevent summer foliage leakage, the model is forced to rely on the prior, synthesizing plausible winter textures (Figure 5 panel f) . Similarly, in severe degradation scenarios (Figure 11), when the original pixels is destroyed and the reference cannot provide a perfect match, the model relies on the prior to synthesize plausible high-frequency textures to resolve the ill-posed problem.
- Identifying when the model preserves recognizable background content:
The model preserves the input's content when the degraded input retains high-frequency structural evidence (high channel variance). The EGR Severity Maps (Figure 5, panel c) show that sharp structural outlines (e.g., the dome, buildings) have lower pixel intensities (high variance). When this condition is met, EGR penalty to degraded keys is reduced, allowing the attention mechanism to naturally anchor to the degraded keys. We do not claim pixel-perfect preservation, as micro-level hallucinations are an inherent part of resolving ill-posed degradation. However, when the input has structural evidence, the model successfully preserves the macro-structural layout and semantic state of the background, using the generative prior only to synthesize plausible micro-textures where the exact original pixels are mathematically unrecoverable.
Response to Questions from Reviewer
For GPT-Image-2.0, Nano Banana 2 and baseline Qwen-Image-Edit the system prompt is kept unchanged, the supplied user prompt is "remove falling precipitation such as rain streaks, snow, and restore the image to clear weather while maintaining the spatial layout of objects in the scene.".
For fine-tuned Qwen-Image-Edit, including RefRestore, the modified system prompt is "Describe the key features of picture 1 (color, shape, size, texture, objects, background), then explain how the user's text instruction should alter or modify the image. Generate a new image that meets the user's requirements while maintaining consistency with the original input where appropriate." and the supplied user prompt is "restore the image to clear weather.".
We thank reviewer for the feedback and commit to include these prompt engineering details in the revised Appendix (A.2). Region-Consistency Quarantine has been briefly discussed in the Appendix (Section A.3), we have further expanded this section based on feedback.
Measuring true 'accuracy of reference transfer' fundamentally requires a clean, spatially aligned Ground Truth (GT). In our real-world, unaligned setting, this is impossible because: (1) we lack aligned GT by definition, (2) reference-degraded viewpoint misalignment prevents the use of traditional pixel-wise metrics (PSNR, SSIM), and (3) feature-based metrics (like DINO) cannot distinguish between accurately retrieved features and plausible hallucinations from the generic prior. Because of these barriers, RefUse and RefMisplace are not designed to measure strict accuracy. Instead, they are proxies for controlled utilization: RefUse measures the necessity of retrieval in evidence-poor regions, while RefMisplace measures the safety of not overwriting observable input.
The τ ablation in Table 2 serves to show the RefUse/RefMisplace trade-off of controlled utilization:
We select the moderate τ because it balances structural anchoring with reference routing. While the τ ablation alone cannot prove accuracy, on top of proxy metrics RefUse and RefMisplace, we rely on visual inspection of qualitative results to confirm that this setting achieves controlled transfer.
We thank the reviewer for catching this typo and will correct RefMisplace to (lower is better) across all tables.
RefUse and RefMisplace are designed to be interpreted jointly to distinguish model behaviors:
We thank the reviewer for this suggestion. We respectfully note that CARI is an architectural prerequisite for EGR in unaligned settings. Without CARI, EGR defaults to copying spatially aligned patches due to positional bias, rendering an isolated evaluation uninformative. Instead, we demonstrate their individual contributions through the marginal performance gains in our incremental ablation (Table 3). Furthermore, to provide deeper mechanistic insight, we have added a detailed attention visualization analysis for both CARI and EGR in Appendix A.4 (Figure 9).