acceptodds
Under review as a conference paper at ICLR 2027

RefRestore: Restoring Adverse Weather with Unaligned Clean References

Abstract

Severe adverse weather can remove the scene evidence needed for reliable restoration. Existing restoration models rely mainly on the degraded observation and learned priors; when the observation is weak, these priors cannot reliably recover details specific to the actual scene. A clean image of the same scene provides additional scene-specific information, but it is typically unaligned and may differ in viewpoint, framing, or appearance. Naively conditioning on such a reference can therefore copy content to incorrect locations or overwrite content supported by the degraded input. We introduce RefRestore, a reference-based restoration framework that exploits an unaligned same-scene image as supplementary visual information without treating it as a spatial template. RefRestore combines Content-Addressed Reference Injection, which removes explicit positional encoding from reference tokens so they are accessed by content, with Evidence-Gated Routing, which suppresses unreliable degraded features and allows other information sources to contribute where observation evidence is weak. We further introduce RefWeatherTrain-456 and RefWeatherTest-20, together with diagnostics for reference use and misplacement. On RefWeatherTest-20, RefRestore ranks first on five of six no-reference quality metrics and second on the remaining metric, while substantially reducing reference misplacement compared with naive reference conditioning. These results demonstrate the value of unaligned same-scene references as scene-specific information for adverse-weather restoration.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion models, image restoration, low-level vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LWsBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript presents RefRestore, a framework for adverse-weather image restoration that uses an unaligned clean image of the same scene as supplementary information. Its two main mechanisms are Content-Addressed Reference Injection (CARI), which removes explicit positional encoding from reference tokens, and Evidence-Gated Routing (EGR), which suppresses unreliable degraded-image keys based on estimated local degradation severity. Combined with severity-gated perceptual regularization and mismatch-calibrated training, these mechanisms aim to support selective reference use without explicit geometric alignment.

The manuscript also introduces RefWeatherTrain-456 and RefWeatherTest-20, alongside diagnostics for reference use and potential misplacement. On RefWeatherTest-20, RefRestore achieves the best scores among the compared methods on five of six no-reference quality metrics and the second-best score on the remaining metric. Ablations and qualitative comparisons indicate reduced reference-copying behavior relative to naive reference conditioning. These results support improved perceptual quality and more controlled reference use, while the absence of aligned clean ground truth limits conclusions about the factual correctness of reconstructed details.

Critical strengths

The manuscript proposes using unaligned reference images of the same scene to perform severe adverse weather restoration tasks, which has practical significance. The method provides a clear design approach for reference-assisted restoration from the perspectives of spatial bias and insufficient observation information: CARI is responsible for eliminating the explicit influence of reference coordinates, and EGR is responsible for suppressing the degenerate context that is estimated as unreliable. Furthermore, the manuscript is well-structured, logically sound, and of high quality.

Critical weaknesses

Some qualitative results altered the still recognizable background structure in the input, such as the sky in the fourth row of Figure 3. Therefore, the existing experimental results make it difficult to determine when the model uses a reference, when it relies on generative priors, and whether it can preserve recognizable background content in the input. This directly impacts the manuscript's core objective of selectively utilizing reference scene information for image restoration.

Questions for authors

1.The authors mentioned system prompt, user prompt, and optional region-consistency quarantine in Figure 2, but I observed that the authors did not mention these in the experimental setup. Please supplement the relevant information.

2.RefRestore recovers the main content well, but some backgrounds are different from the reference image and also change the structure that is still recognizable in the input. It is difficult to determine whether these changes come from the reference information or the generation prior. The discard and replacement strategies are insufficient to explain how the model selects information sources and whether it can retain visible content in the input. The authors are requested to address this issue in detail.

3.Increasing τ strengthens the suppression of degenerate keys, and RefUse and RefMisplace also become smaller compared to the no-reference baseline. However, this does not seem to determine whether the model more accurately transfers reference information or reduces the overall influence of reference information.

4.RefMisplace is marked as uparrow in Table 2 but as downarrow in Table 3. Please unify the direction. In addition, an increase in RefUse may come from incorrect copying, while correctly restoring occluded content may also increase RefMisplace. Please explain how to interpret RefUse and RefMisplace together, and how to distinguish between erroneous migrations and correct restoration. 5.Regarding the effectiveness of CARI and EGR, current ablation experiments only discuss the combined effect of the two and cannot demonstrate their individual contributions. It is suggested that the authors provide further clarification on this matter.

4 comments
AuthorsOpenReview ↗

Response to Questions from Reviewer

For GPT-Image-2.0, Nano Banana 2 and baseline Qwen-Image-Edit the system prompt is kept unchanged, the supplied user prompt is "remove falling precipitation such as rain streaks, snow, and restore the image to clear weather while maintaining the spatial layout of objects in the scene.".

For fine-tuned Qwen-Image-Edit, including RefRestore, the modified system prompt is "Describe the key features of picture 1 (color, shape, size, texture, objects, background), then explain how the user's text instruction should alter or modify the image. Generate a new image that meets the user's requirements while maintaining consistency with the original input where appropriate." and the supplied user prompt is "restore the image to clear weather.".

We thank reviewer for the feedback and commit to include these prompt engineering details in the revised Appendix (A.2). Region-Consistency Quarantine has been briefly discussed in the Appendix (Section A.3), we have further expanded this section based on feedback.

Measuring true 'accuracy of reference transfer' fundamentally requires a clean, spatially aligned Ground Truth (GT). In our real-world, unaligned setting, this is impossible because: (1) we lack aligned GT by definition, (2) reference-degraded viewpoint misalignment prevents the use of traditional pixel-wise metrics (PSNR, SSIM), and (3) feature-based metrics (like DINO) cannot distinguish between accurately retrieved features and plausible hallucinations from the generic prior. Because of these barriers, RefUse and RefMisplace are not designed to measure strict accuracy. Instead, they are proxies for controlled utilization: RefUse measures the necessity of retrieval in evidence-poor regions, while RefMisplace measures the safety of not overwriting observable input.

The τ ablation in Table 2 serves to show the RefUse/RefMisplace trade-off of controlled utilization:

  • Low τ (Under-suppression): The penalty is too weak to break the self-reinforcement of degraded features, leading to blind copying and high RefMisplace.
  • Moderate τ (Optimal Routing): The penalty is strong enough to suppress unreliable keys and route to the reference, while preserving observable input structure. This yields high RefUse with minimal RefMisplace.
  • High τ (Over-suppression): As you correctly noted, an overly aggressive penalty disrupts the degraded stream's structural role. The model effectively abandons the observation, falls back to the generic prior, and both metrics collapse toward the no-reference baseline.

We select the moderate τ because it balances structural anchoring with reference routing. While the τ ablation alone cannot prove accuracy, on top of proxy metrics RefUse and RefMisplace, we rely on visual inspection of qualitative results to confirm that this setting achieves controlled transfer.

We thank the reviewer for catching this typo and will correct RefMisplace to (lower is better) across all tables.

RefUse and RefMisplace are designed to be interpreted jointly to distinguish model behaviors:

  • High RefUse + High RefMisplace (Erroneous Migration): The model is blindly copying reference content, leading to high similarity with the reference but also inappropriately overwriting observable input content.
  • High RefUse + Low RefMisplace (Correct Restoration): The model is successfully retrieving useful, scene-specific appearance information from the reference without inappropriately overwriting observable input evidence.
  • Low RefUse + Low RefMisplace (Reference Ignored): The model fails to utilize the reference, behaving similarly to the no-reference baseline and missing out on scene-specific details.

We thank the reviewer for this suggestion. We respectfully note that CARI is an architectural prerequisite for EGR in unaligned settings. Without CARI, EGR defaults to copying spatially aligned patches due to positional bias, rendering an isolated evaluation uninformative. Instead, we demonstrate their individual contributions through the marginal performance gains in our incremental ablation (Table 3). Furthermore, to provide deeper mechanistic insight, we have added a detailed attention visualization analysis for both CARI and EGR in Appendix A.4 (Figure 9).

AuthorsOpenReview ↗

Response to Questions from Reviewer

We thank the reviewer for this observation. We understand the concern regarding background alterations and the ambiguity of whether these changes stem from the reference or the generation prior. We interpret the reviewer’s mention of 'discard and replacement' as referring to our Evidence-Gated Routing (EGR) mechanism.

EGR uses channel variance as a proxy for evidence strength to dynamically route attention:

  • Evidence-Rich (High Variance): Regions with sharp details route attention to the degraded input, preserving observable structures.
  • Evidence-Poor (Low Variance): Regions obscured by fog or smooth textures route attention to the reference to synthesize missing details.

To demonstrate that the model actively selects and rejects mismatched reference content rather than blindly replacing the input, we provide ablation with an idiosyncratic reference in Appendix A.4 Figure 10. The aim of the ablation is to prove reference utilization.

Reviewer_LWsBOpenReview ↗

Response to rebuttal

Thanks for the authors' rebuttal. Part of my concerns are addressed, but the evidence to support the main claim of this work is not yet sufficiently established, as mentioned in weakness "the existing experimental results make it difficult to determine when the model uses a reference, when it relies on generative priors, and whether it can preserve recognizable background content in the input. "

Thus, I keep the initial rating.

AuthorsOpenReview ↗

Response to Questions from Reviewer

We thank the reviewer for their continued engagement and for acknowledging that our rebuttal has addressed part of their concerns. While we respect their decision to maintain their initial rating, we hope the following explicit mapping of our experimental evidence to their core concern will clarify the model's behavior for the Area Chair and the broader community.

To make this determination completely transparent, we have summarize our ablation in the revised manuscript to explicitly isolate and visualize these three distinct behaviors:

  1. Identifying when the model uses the reference:

The model uses the reference when the degraded input is evidence-poor (low channel variance, e.g., heavy fog or snow) and the reference provides semantically compatible details to fill that void. The Idiosyncratic Reference Test (Figure 10) proves this condition. Unique reference objects (e.g., the balloon) appear in the output only when the corresponding input region is heavily degraded (evidence-poor), proving the model actively retrieves reference content specifically to resolve information loss. This is further quantified by the Attention Mass Visualizations (Figure 9), which provide mechanistic proof of degraded queries attending to reference keys.

  1. Identifying when the model relies on generative priors:

The model relies on the generative prior when the input is evidence-poor, but the reference is either unhelpful, globally mismatched, or actively blocked to prevent semantic leakage. The Region-Consistency Quarantine (RCQ) ablation (Figure 5) explicitly triggers this condition. When RCQ blocks the reference for the winter trees to prevent summer foliage leakage, the model is forced to rely on the prior, synthesizing plausible winter textures (Figure 5 panel f) . Similarly, in severe degradation scenarios (Figure 11), when the original pixels is destroyed and the reference cannot provide a perfect match, the model relies on the prior to synthesize plausible high-frequency textures to resolve the ill-posed problem.

  1. Identifying when the model preserves recognizable background content:

The model preserves the input's content when the degraded input retains high-frequency structural evidence (high channel variance). The EGR Severity Maps (Figure 5, panel c) show that sharp structural outlines (e.g., the dome, buildings) have lower pixel intensities (high variance). When this condition is met, EGR penalty to degraded keys is reduced, allowing the attention mechanism to naturally anchor to the degraded keys. We do not claim pixel-perfect preservation, as micro-level hallucinations are an inherent part of resolving ill-posed degradation. However, when the input has structural evidence, the model successfully preserves the macro-structural layout and semantic state of the background, using the generative prior only to synthesize plausible micro-textures where the exact original pixels are mathematically unrecoverable.

Reviewer_r5C2OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes RefRestore, an adverse weather restoration approach that uses scene-specific details from an unaligned reference image to restore a given image. The approach does not require any camera or pose information about the reference image. A naive approach would cause parts of the reference image to directly get copied during restoration, which affects the faithfulness of the resulting output. To mitigate this, the authors propose Content-Addressed Reference Injection (CARI), which removes positional information from the reference image's keys (during attention). This allows the transformer model to extract relevant information from anywhere in the reference image and does not bias it with incorrect positions relative to the query. Additionally, the authors introduce Evidence-Gated Routing (EGR) which finds reliable regions of the degraded image and suppresses attention to unreliable areas. This allows targeted contributions from the reference image.

Critical strengths
  1. The proposed modules are novel and intuitive. For instance, the idea of simply removing positional bias from the reference image keys is elegant, compared to using explicit geometric priors. Furthermore, the idea of using patch-level similarity as a metric to evaluate information transfer from the reference image is interesting.

  2. The adverse weather is strongly restored, with high scores in no-reference metrics.

Critical weaknesses
  1. It is unclear what needs to be transferred from the reference image: The approach does not seem to establish what aspects of the reference image needs to be leveraged for restoration. For example, in rows 2 and 3 of Fig. 3, there is a very significant change in the general appearance of the image (lighting, atmosphere, etc.), which may be arising from the reference image.

  2. Lack of evaluation of restoration fidelity: Image restoration requires outputs that are faithful to the input degraded image. The current paper only provides no-reference metrics which do not capture hallucinations or content changes from the generative backbone. Furthermore, many results from RefRestore have significant changes in overall appearance, in addition to weather removal, which raises major concerns about the restoration fidelity of the approach.

  3. Existing ablations lack visualizations of how the proposed components influence the attention mechanisms of the model. For example, attention maps can be visualized with CARI and EGR to qualify the impact of these modules.

Questions for authors
  1. What exactly needs to be transferred from the reference image?

  2. In row 3 of Fig. 3, why does the restored result seem to be from a completely different season (different tree leaves)?

  3. RefMisplace penalizes content in the restored image that appears in the reference image but is not in the degraded image. But doesn't this contradict the motivation of reference-based restoration as areas of degradations might have clean features in the reference image? Could the severity map in Eqn. 5 be used to improve this aspect of RefMisplace?

  4. Can the authors provide reference-based metrics (PSNR, SSIM, LPIPS) over a large-set of images? It should be possible as a similar synthetic degradation pipeline as mentioned in Sec. A.1 can be used.

  5. Can the authors provide some attention visualizations to establish the workings of the proposed mechanisms? For example, it would be very useful to visualize how EGR weights different regions of the degraded image, and to substantiate the claims in Line 249. Similarly, the effect of positional bias from RoPE on the key tokens (addressed by CARI) can also be visualized.

3 comments
AuthorsOpenReview ↗

Response to Questions from Reviewer

RefRestore is designed to retrieve low- and mid-level appearance cues (e.g., colors, textures, and materials) and instance-specific details from the reference, but strictly in regions where weather degradation has obscured the underlying structure. Crucially, in regions where the degraded input retains high-frequency structural evidence, the input's features take precedence and reference attention is suppressed. By prioritizing the input's observable evidence over the reference, the model is designed to preserve the high-level semantic state of the scene (e.g., time of day and overall layout), anchoring the restoration to the original input's context.

The specific example shows seasonal leakage. When heavy snow in the input eliminates the channel variance of the trees, our Evidence-Gated Routing (EGR) flags these areas as evidence-poor and pulls appearance from the reference. However, because the reference depicts a different season, the model inadvertently paints summer/autumn foliage over the winter scene. We have expanded the discussion section (Appendix A.6) of this limitation of EGR. To mitigate this issue, our framework includes an optional Region-Consistency Quarantine (detailed in Section A.3) to mask out protected semantic regions (e.g., vegetation or sky), forcing the model to synthesize those areas using its generic prior rather than retrieving mismatched content from the reference. We disabled this feature in our main experiments to isolate our core routing mechanisms.

We thank the reviewer for this insightful question and constructive suggestion.

Our primary intention of RefMisplace is strictly to quantify misplacement (i.e., inappropriate transfer or hallucination of reference content), rather than to measure overall restoration quality. In severely degraded regions where the input provides zero evidence, it is impossible to verify whether the content retrieved from an unaligned reference is 'correct' or a 'hallucination'. Because we cannot establish ground truth for unaligned, evidence-poor regions, RefMisplace is intentionally designed to be conservative. If the model pulls content from the reference into a region where the input has no corresponding evidence, the metric flags it as a potential misplacement, because we have no mathematical way to verify its correctness.

However, the reviewer makes an excellent point. If a region has absolutely zero evidence (maximal severity), 'misplacement' is indeed unmeasurable, and penalizing the model there is overly harsh. We agree that the severity map (Eq. 5) can be used to refine this aspect. Rather than using the severity map to reward restoration (which we cannot verify without GT), we will use it to mask out the RefMisplace penalty in zero-evidence regions. By weighting the intrusion term by the evidence strength, we ensure that RefMisplace only penalizes the model in regions where there was actually observable evidence in the input that was inappropriately overwritten. In regions where the evidence is zero, the metric will simply abstain from measuring misplacement. We will update the RefMisplace formulation in the final revised manuscript to sharpen the diagnostic rigor of our evaluation metric.

Following the reviewer's suggestion, we applied our synthetic weather degradation pipeline (Sec. A.1) to the clean images of the large-scale LMR benchmark to evaluate performance using full-reference metrics (PSNR, SSIM, and LPIPS). The detailed quantitative results are reported in Appendix A.4.

We thank the reviewer for this suggestion. To explicitly demonstrate the workings of EGR and CARI, we have added a new visualization (Appendix A.4, Figure 9) that tracks the attention dynamics across all transformer blocks for a representative test image. While spatial overlays show attention for a single layer, this block-wise analysis rigorously proves how our mechanisms govern the generation process from start to end.

Reviewer_r5C2OpenReview ↗

Response to author rebuttal

I thank the authors for their complete rebuttal. Please see my response:

  1. I thank the authors for clarifying this. Fig. 10 seems to provide some evidence to corroborate the precedence of structures in the input image over those in the reference. However, is the transfer or low-and mid-level features completely dependent on the generative backbone or are there any specific design choices which ensure that only low- and mid-level features are transferred? This is also important to understand why the balloon in Fig. 10 was placed in a particular location of the output instead of being placed somewhere else in the sky.

  2. I thank the authors for discussing this in the limitations. However, the introduction of Region-Consistency Quarantine still does not address the alteration of input scene details (for e.g., the trees changed in row 1 of Fig.5 (e)).

  3. My concern is addressed.

  4. I thank the authors for incorporating these results, however these further show that RefRestore does not preserve fidelity with the input during restoration (low perceptual and distortion scores; colors, trees, building structures have changed in Fig. 11). Its not clear if this is because of the sheer intensity of the degraded inputs or if the generative prior of the backbone is not well-constrained.

  5. The provided attention analysis is useful for CARI. However, for EGR I was wondering if a spatial visualization of the attention localization would be more effective to showcase its contribution. For instance, whether EGR identifies the correct severity maps and if those regions receive more attention from the reference image.

Given the rebuttal, my major concerns regarding what needs to be transferred from the reference and the restoration fidelity of the approach remain.

AuthorsOpenReview ↗

Response to Questions from Reviewer

We thank the reviewer for this insightful question. The selective transfer of low- and mid-level features is not solely dependent on the generative backbone, but governed by our design. In the degraded input (d), the sky region above the bridge is heavily obscured by fog (evidence-poor). EGR identifies this specific region as needing restoration, applies a penalty to degraded keys shifting attention mass to reference keys. CARI removes positional bias from the reference keys, further enabling the attention mechanism to search globally for feature-similar tokens. The clear sky region in the reference (which happens to contain the balloon) has the highest semantic similarity to the foggy sky queries in the input. Consequently, the generative backbone harmonizes the information into the scene naturally.

We acknowledge the reviewer's point and believe this highlights an important distinction between semantic alteration (which RCQ addresses) and textural synthesis (inherent to the restoration task).

Row 1 of Figure. 5(e) (w/o. RCQ) demonstrates severe semantic alteration: the trees incorrectly change from a winter state to an autumn state due to reference leakage. RCQ successfully resolves this, as seen in Figure. 5(f), where the trees correctly retain their winter state.

However, the original high-frequency details of the trees in the input are largely lost (therefore identified as evidence-poor by EGR via low channel variance). When RCQ blocks the reference to prevent semantic leakage, the backbone is forced to rely on its generic prior to synthesize plausible winter textures which inevitably 'alters' the specific degraded patterns of the input, replacing them with generated clean details.

The observed 'changed' details in restored outputs reflect the classic perception-distortion tradeoff inherent to severe image restoration, rather than an unconstrained generative prior. The backbone is well-constrained, operating at the macro-structural and semantic level rather than the micro-textural level.

In many of the test scenes (e.g., aerial cityscapes or continuous parks), the reference and the input represent a single, large semantic entity. When EGR identifies a heavily degraded region, the attention mechanism retrieves any semantically matching, high-quality texture from the reference. The reference here acts as constraint to ensure that the global layout, lighting, color palette, and stylistic priors for man-made objects remain perfectly coherent with the intended scene.

Because the exact original pixels of the fine details are unrecoverable from the degraded input, the backbone must rely on its generative prior to synthesize plausible high-frequency details in those evidence-poor regions. This is a standard and necessary behavior in perceptual restoration; attempting to strictly preserve pixel-level fidelity in such heavily degraded regions would result in the residual weather artifacts and blurring seen in the baselines Figure 11 (d, e, f). In contrast, by using the reference to anchor the macro-structure and the generative prior to synthesize sharp, plausible micro-textures in evidence-poor regions, RefRestore produces visually natural results.

We agree with the reviewer on this point and have included the severity map computed from EGR in Figure 5c. EGR is not a hard binary switch but rather, modulates the attention logits. In evidence-poor regions (bright pixels), EGR applies a penalty to suppress the degraded keys to route queries to the reference keys.

Reviewer_wvoMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses adverse-weather restoration. A clean image of the same scene can serve as a reference, but it is typically not spatially aligned with the degraded input. RefRestore removes explicit positional encoding from reference tokens so they are accessed by content (CARI), suppresses unreliable degraded keys in proportion to a severity score estimated from channel variance (EGR), and is trained with mismatch-calibrated training, i.e., reference dropout and random-scene references. The authors also introduce a small reference-paired training set and a real-weather benchmark for which pixel-aligned clean ground truth is unavailable, together with two DINOv2-based diagnostics for reference use and misplacement. RefRestore ranks first on five of six no-reference quality metrics and lowers the reference misplacement diagnostic (RefMisplace) compared with naive reference conditioning.

Critical strengths
  • The paper addresses a timely and practically relevant problem. When severe weather removes the scene evidence needed for restoration, learned priors alone cannot recover scene-specific details, and powerful generative priors can silently alter observed content (Appendix A.3). As the paper notes, images of the same location are often available from webcam archives, public image collections, or repeated captures, and the paper poses two clear research questions: whether such an unaligned reference is useful, and whether it can be integrated without reference copying.

  • The paper identifies and quantifies a concrete failure mode of naive reference conditioning: supplying the reference improves all six no-reference metrics but raises RefMisplace from 0.236 to 0.547 and copies content such as the crowd and graffiti bin (Table 3, Fig. 4).

  • The paper is clearly written and easy to follow. The problem, the failure mode (Eq. 2, Fig. 4), and each component are presented in a logical order.

Critical weaknesses
  • The qualitative results appear to contradict the claim that the observed scene is preserved. The paper claims RefRestore "restores the lost appearance from the reference while preserving the observed scene" (Fig. 1), and the Abstract warns that naive conditioning can "overwrite content supported by the degraded input." In Fig. 7 (2nd row), however, the input (e) shows bare, snow-covered trees and the reference (d) is an autumn scene, yet the output (f) shows orange foliage with snow still on the ground. In the 5th row, the trees in the input (b) are snow-covered on both sides, but the output (c) shows green foliage on the right. In both cases, the reference appears to override the state visible in the input, and the paper does not define what should be transferred versus preserved.

  • The diagnostics do not substantiate the claim. RefMisplace relies on DINOv2 features, which, as the paper notes, "capture semantic content rather than exact geometry" (Sec. 3.3), so changes within a category (e.g., bare vs. foliated trees) may go undetected; it is also validated on only four scene pairs (Fig. 6). Moreover, a low RefMisplace does not by itself show selective use: RefUse is also close to the no-reference baseline (+0.0103 at τ = 1.0, vs. +0.1203 for naive conditioning; Table 2), and no variance is reported, so the diagnostics cannot tell selective use from limited use of the reference.

  • The evaluation design is weak. The evaluation design is weak. Table 1 includes no reference-based methods, not even ReBaIR [4], which the paper cites, nor alternatives such as gating-based reference control [1, 2] or position-shifted reference tokens [3] on the same backbone. Moreover, data is small (456 training images from 52 scenes; 20 test scenes).

[1] Guo, H., Dai, T., Ouyang, Z., Zhang, T., Zha, Y., Chen, B., & Xia, S. T. (2024). Refir: Grounding large restoration models with retrieval augmentation. Advances in Neural Information Processing Systems, 37, 46593-46621.

[2] Y. Wang, Y. Wan, S. Zheng, B. Li, Q. Hou, and P.-T. Jiang. Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling. International Conference on Learning Representations (ICLR), 2026.

[3] Z. Tan, S. Liu, X. Yang, Q. Xue, and X. Wang. OminiControl: Minimal and Universal Control for Diffusion Transformer. IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 14940–14950.

Questions for authors
  • The paper states that RefRestore “restores the lost appearance from the reference while preserving the observed scene” (Fig. 1). What exactly should be transferred from the reference (e.g., color, texture), and what must be preserved from the input (e.g., season, foliage state, snow cover)? For example, two results in Fig. 7 appear inconsistent with this:

    • Second row (d)-(f): the input shows bare, snow-covered trees, and the reference is an autumn scene with orange foliage. The output shows orange foliage and orange reflections on the water, while snow remains on the ground.
    • Fifth row (a)-(c): the trees on both sides of the input are snow-covered, and the reference shows green foliage. The output keeps the left tree snow-covered but shows green foliage on the right tree.
  • For each of the 20 scenes in RefWeatherTest-20, please report (a) how the reference differs from the input (season, time of day, weather, viewpoint), (b) RefUse and RefMisplace, and (c) whether the output contains content that conflicts with the degraded input. Please also include the reference for each scene in Fig. 3, and release the output images so these values can be verified.

  • To separate the effect of the reference from that of fine-tuning, please report results for (a) the fine-tuned model without any reference, (b) the final model with the reference removed, (c) the final model with a random-scene reference, and (d) the final model with a same-scene reference in a different state (e.g., a different season or time of day). Currently, the gain over the backbone (e.g., TReS 66.85 → 78.22) cannot be attributed to the reference, and a low RefMisplace cannot be distinguished from limited reference use.

  • In Fig. 3, the compared methods, which are designed for multiple adverse weather types, appear nearly identical to the input. Could the authors explain why, and add an Input row to Table 1 and specify the checkpoints, training data, and inference settings used.

  • Could the authors ablate CARI and EGR separately? Table 3 reports only the combined “+CARI-EGR.”.

3 comments
AuthorsOpenReview ↗

Response to Questions from Reviewer

To clarify the intended scope of our design: low- and mid-level appearance (e.g., colors, textures, and materials) destroyed by degradation, along with instance-specific priors (e.g., transient objects), should be transferred from the reference specifically where the degraded observation provides weak evidence. Crucially, if the available evidence in the degraded input contradicts the reference, the reference prior is suppressed and not carried over. Conversely, high-level semantic state (e.g., season, scene layout, and the existence of observable objects) must be strictly preserved from the input.

We agree that the inappropriate transfer of summer/autumn foliage onto snow-covered trees represents a failure to preserve the input's semantic state. When heavy snow or fog destroys the visual evidence of the trees (driving channel variance to near zero), our Evidence-Gated Routing (EGR) identifies the region as evidence-poor and routes it to the reference. Without an explicit semantic constraint, the model retrieves the high-frequency foliage texture from the reference, causing a seasonal mismatch. The optional Region-Consistency Quarantine (Appendix A.3) is designed to mitigate this failure mode. By masking protected semantic regions (e.g., sky, vegetation), it blocks direct reference retrieval and forces the model to synthesize appearance from its generic prior. We disabled this feature in our main experiments to isolate the automatic routing mechanisms, but acknowledge it is necessary when references contain local state mismatches.

We thank the reviewer for this request. Due to space constraints, our full exhaustive results are in Appendix A.5 for verification.

(a) Our references are clean, daytime images, but span various seasons, weather conditions, and slight viewpoint shifts. Nighttime references are excluded due to insufficient structural detail. The reference images used for restoration of each scene are detailed in Appendix A.5.

(b) We have expanded A.5 to include per-image RefUse and RefMisplace scores for all 20 scenes.

(c) In the absence of aligned ground truth, we manually evaluate outputs for structural conflicts (e.g., added/removed objects). We confirm that while appearance transfer occurs (e.g., foliage color), most outputs contain no structural hallucinations that contradict the evidence in the observed degraded input with exception of display panel (Figure 9, row 4) and transient bird (Figure 10, row 4).

The compared multi-weather restoration methods appear nearly identical to the input because they severely under-remove real-world degradation. These models are trained exclusively on synthetic degradation pairs, which fail to capture the complex physics of real adverse weather (e.g., rain-induced mist, snow-scatter fog, and mixed atmospheric conditions). Faced with a severe domain gap between their synthetic training data and our real-world test set, the models default to highly conservative outputs that retain most of the atmospheric veil.

As requested, we have added an 'Input' row to Table 1 in the revised manuscript so that the gains in no-reference image quality metrics can be directly compared against the degraded inputs.

Regarding the evaluation setup, all baseline models were evaluated using their official, default inference scripts and were tested using their officially released checkpoints (trained on their respective original synthetic datasets). The specific inference settings are detailed below:

  • TKL-MCR: Default checkpoint (Setting 1) and network configuration.
  • WeatherDiff: Default WeatherDiff64 checkpoint and allweather.yml configuration. Inference uses patch-based sampling size 64x64 with an overlapping stride of 16 and 25 sampling time-steps.
  • MWDT: Default checkpoint and default network configuration.
  • T³-DiffWeather: Default checkpoint with the default network configuration specified in allweather_demo.yml.
  • SSGFormer: Default single checkpoint and configuration file.
  • MODEM: Default single checkpoint and network configuration.

Thank you for the authors' response.

  1. I appreciate the acknowledgment that the foliage transfer is a failure to preserve the input’s semantic state. However, EGR uses channel variance only as a relative evidence proxy, not as a measure of contradiction. The claim that “the reference prior is suppressed” when the input contradicts the reference is therefore not supported. In row 3 of Fig. 3 and row 3 of Fig. 9, the snow-covered trees remain visible in the input. RCQ is disabled in the main experiments and evaluated only qualitatively. The response states that RCQ “is designed to mitigate this failure mode.” A.3, however, states that RCQ “can degrade performance when used with globally mismatched reference (e.g., summer reference for winter scene)” and is designed for “transient objects” rather than “global scene mismatches.” When should a user enable RCQ or omit the reference?

  2. The golden-dome output without RCQ inappropriately transfers autumn/summer foliage (Fig. 5). Yet the sanity test presents the same output as (d), which preserves it (Fig. 7). Its RefMisplace is 0.218 in Fig. 7 and 0.293 in Fig. 9. The Fig. 7 score equals that of the last row in Fig. 10; so please check the Fig. 7-10 scores. Either value lies within the range of (d) [0.14, 0.31], well below that of (c) [0.38, 0.70], which visibly copies reference content. RefMisplace thus scores this acknowledged failure like a faithful output, so the RefMisplace results in Table 2 do not rule out such foliage transfer.

  3. Thank you for adding the Input row. The response attributes the baselines’ weakness to “a severe domain gap between their synthetic training data and our real-world test set,” yet they were not retrained on RefWeatherTrain-456, on which RefRestore is trained with synthetic degradations. The Qwen-Image-Edit backbone alone, without a reference, already outperforms all baselines on NIQE, MUSIQ, CLIPIQA+, and Q-Align. RefRestore improves further over the backbone, but this gain combines fine-tuning on RefWeatherTrain-456 and the reference. The claim in Fig. 3 that RefRestore “uses an unaligned clean reference to produce clear and vivid results” therefore requires Qwen-Image-Edit fine-tuned on RefWeatherTrain-456 without a reference (Q3).

Unaddressed: Q3 (reference dropped or replaced at inference), Q5 (separate ablation of CARI and EGR), and the comparison with reference-based methods (Weakness 3).

My main concerns remain.

AuthorsOpenReview ↗

Response to Questions from Reviewer

We thank the reviewer and appreciate the opportunity to clarify these mechanisms and provide explicit guidelines for users. Our previous phrasing was imprecise. EGR measures evidence strength (via channel variance), not semantic contradiction.

The reviewer noted an apparent contradiction: we stated RCQ mitigates foliage transfer, yet the Appendix notes it degrades performance for globally mismatched references. This distinction hinges entirely on the scope of the mismatch.

  • Omit the reference: For globally mismatched references (e.g., different season or scene). In this case, RCQ degrades unprotected regions, making the no-reference baseline the optimal choice.
  • Enable RCQ: For largely aligned references with localized, transient mismatches (e.g., a passing crowd). RCQ acts as a targeted guardrail to mask these specific regions while preserving the reference's benefits elsewhere.

We apologize for the typographical errors in the figure captions and tables. We have carefully re-verified the data and will correct the inconsistent RefUse/RefMisplace scores in the revised Appendix.

The reviewer correctly notes RefMisplace is low for the foliage transfer. This is expected, as the metric (averaged intrusion + overwrite) is insensitive to localized appearance changes. RefMisplace operates in the DINO feature space, which prioritizes object shape and semantic over surface appearance. A snow-covered tree and an autumn tree share the same underlying structure, so their DINO embeddings are close. Consequently, the foliage transfer neither registers as strong "intrusion" (the tree object already exists in the input) nor as strong "overwrite" (the patch's structure is preserved).

We wish to clarify that these specific RefMisplace ranges for column (c) and (d) are derived exclusively from the scenes depicted in the sanity check figure, rather than the entire dataset. Furthermore, these values do not represent universal pass/fail thresholds. The sanity check for RefMisplace in the Appendix is designed to verify relative sensitivity within a controlled comparison: that RefMisplace rises sharply when the model performs aggressive naive copying (column c), which produces the gross spatial misplacement the metric is built to detect versus controlled generation (column d).

Because τ\tau is an inference-time hyperparameter governing EGR's gating strength, the reviewer is correct that Table 2 is not a guarantee that all semantic mismatches are eliminated. It illustrates the relative trade-off between RefUse and RefMisplace as τ\tau shifts the model along the spectrum of reference utilization, and it should be read alongside the visual evidence rather than as an absolute correctness guarantee.

Questions from previous comments:

We thank the reviewer for the request. We have added new experiments in Table 3. Specifically, results for:

  • a) the fine-tuned model without any reference (row c).
  • b) the final model with the reference removed (row d).
  • c) the final model with a random-scene reference (row i).
  • d) the final model with a same-scene reference in a different state (e.g., a different season or time of day). Our results in (row h) uses clean unaligned references, some in a different state (typically for snow degraded input, clean summer/autumn references). We have expanded the Appendix A.4 to provide qualitative results using same-state references for snow-degraded scenes.

To address the results from new experiments, we have expanded the ablation (Section 4.2) in the main paper.

We thank the reviewer for this suggestion. CARI is an architectural prerequisite for EGR in an unaligned setting. Without CARI, EGR simply copies spatially aligned patches due to positional bias and therefore cannot be meaningfully evaluated in isolation. We instead demonstrate their individual contributions via the marginal gains in our incremental ablation (Table 3).

Weakness 3.

While ReBaIR is seminal in unaligned reference restoration, a direct quantitative comparison is not a fair evaluation. ReBaIR is explicitly trained for super-resolution and mild deblurring, whereas adverse weather restoration involves complex, non-linear scattering, occlusion, and illumination shifts. Our work specifically adapts unaligned reference mechanisms to disentangle these severe weather artifacts from scene structure—a capability ReBaIR and similar reference-based SR methods were neither designed nor trained to handle.

We acknowledge that 456 training images from 52 scenes is modest. However, constructing real-world datasets with unaligned clear/adverse weather pairs of the same scene is logistically difficult, making such data inherently scarce. More importantly, the modest training size does not indicate overfitting. The effectiveness of our mechanisms is proven by their strong generalization to unseen real weather scenes demonstrated in the Appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.