RefRestore: Restoring Adverse Weather with Unaligned Clean References

Severe adverse weather can remove the scene evidence needed for reliable restoration. Existing restoration models rely mainly on the degraded observation and learned priors; when the observation is weak, these priors cannot reliably recover details specific to the actual scene. A clean image of the same scene provides additional scene-specific information, but it is typically unaligned and may differ in viewpoint, framing, or appearance. Naively conditioning on such a reference can therefore copy content to incorrect locations or overwrite content supported by the degraded input. We introduce RefRestore, a reference-based restoration framework that exploits an unaligned same-scene image as supplementary visual information without treating it as a spatial template. RefRestore combines Content-Addressed Reference Injection, which removes explicit positional encoding from reference tokens so they are accessed by content, with Evidence-Gated Routing, which suppresses unreliable degraded features and allows other information sources to contribute where observation evidence is weak. We further introduce RefWeatherTrain-456 and RefWeatherTest-20, together with diagnostics for reference use and misplacement. On RefWeatherTest-20, RefRestore ranks first on five of six no-reference quality metrics and second on the remaining metric, while substantially reducing reference misplacement compared with naive reference conditioning. These results demonstrate the value of unaligned same-scene references as scene-specific information for adverse-weather restoration.