acceptodds
Under review as a conference paper at ICLR 2027

UENR-600K: A Large-Scale Physically Grounded Dataset for Nighttime Video Deraining

Abstract

Nighttime video deraining is uniquely challenging because raindrops interact with artificial lighting. Unlike daytime white rain, nighttime rain takes on various colors and appears locally illuminated. Existing small-scale synthetic datasets rely on 2D rain overlays and fail to capture these physical properties, causing models to generalize poorly to real-world night rain. Meanwhile, capturing real paired nighttime videos remains impractical because rain effects cannot be isolated from other degradations like sensor noise. To bridge this gap, we introduce UENR-600K, a large-scale, physically grounded dataset containing 600,000 1080p frame pairs. We utilize Unreal Engine to simulate rain as 3D particles within virtual environments. This approach guarantees photorealism and physically real raindrops, capturing correct details like color refractions, scene occlusions, rain curtains. Leveraging this high-quality data, we establish a new state-of-the-art baseline by adapting the Wan 2.2 video generation model. Our baseline treat deraining as a video-to-video generation task, exploiting strong generative priors to almost entirely bridge the sim-to-real gap. Extensive benchmarking demonstrates that models trained on our dataset generalize significantly better to real-world videos. 1. We provide video comparisons in the Supplementary Material. 2. We will release the complete dataset upon acceptance, in strict adherence to the ECCV 2026 requirement.

Primary area:
datasets and benchmarks
Keywords:
Nighttime video deraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4JfwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UENR-600K, a synthetic paired dataset for nighttime video deraining. The authors model rain as 3D particles inside Unreal Engine 5 scenes, producing nighttime rain effects with chromaticity, localized illumination, glimmer, rain curtains, and scene occlusion. The dataset contains 600,000 paired 1080p rainy and clean frames. The authors also adapt Wan 2.2-TI2V-5B to video-to-video deraining using condition tokens, generation tokens, shared positional embeddings, an attention mask, and flow matching. Six existing methods and the Wan baseline are evaluated using PSNR/SSIM on synthetic data, VLM and human preferences on real nighttime rain videos, and temporal-consistency metrics. The paper claims that training on UENR improves real-video deraining and that the Wan baseline performs best under its evaluation protocol.

Critical strengths

The dataset addresses a specific limitation of nighttime deraining data: rain visibility and appearance depend on nearby lights and scene geometry. Its paired videos capture effects that global rain overlays represent poorly. This makes the dataset a useful resource for studying nighttime restoration.

The experiments test the data across several model families, rather than relying solely on the proposed Wan baseline. Synthetic metrics, real-video judgments, and the additional WeatherWeaver comparison provide evidence of its practical utility. The Wan adaptation also supplies a strong baseline, with training details and inference costs reported.

Critical weaknesses

The experiments do not fully isolate the source of the gains. UENR has about 42 times as many training frames as SynNightRain. Appendix F addresses scale through an equal-frame WeatherWeaver comparison, but illumination and restoration targets also differ. Appendix A does not establish scene or trajectory independence between training and testing. Likewise, Wan’s pretraining, full-resolution input, and 90-frame context complicate attributing its advantage to the architectural adaptation. These limitations weaken the causal explanations more than the dataset’s demonstrated utility.

The real-video evaluation needs clearer documentation. Annotator and sample counts, presentation format, and aggregation are missing from the human-study description. Table 3 reports 95.2% human preference and a 95.5% evaluator average, while the supplementary project page reports 97.6% and 96.3%. The temporal VLM examines only four midpoint frames, so its score supports local consistency rather than consistency throughout the clip. These issues limit confidence in the precise preference rate and the breadth of the temporal claims.

Questions for authors

How many annotators and evaluation items were used, what did annotators view, and how were judgments randomized and aggregated? Please reconcile the PDF and project-page preference rates and provide the underlying counts.

Are the training and test frames independent in scene and camera trajectory? Why was Kyoto excluded? Beyond the equal-frame WeatherWeaver comparison, what evidence separates the effects of rain rendering from scale, illumination, and target differences?

Which Qwen model and evaluation settings were used? How representative are the four midpoint frames of temporal behavior across the full clip?

How much of Wan’s advantage can be attributed to the adaptation beyond pretraining and larger spatial and temporal context? Why were NightRain (Lin et al., AAAI 2024) and ASF-Net (Xue et al., Pattern Recognition, 2025) excluded from the comparisons?

6 comments

Thank you for the constructive review. We address each concern below.

 

W1: The dataset comparison confounds rendering realism with dataset scale and synthetic train/test split design.

  • Scale: Dataset scale is an important part of UENR's contribution.

Regarding dataset scale, we added a matched-frames experiment, training Our Baseline on 11,880 UENR frames, close to SynNightRain's 11,915. The remaining training settings match those in the paper.

We evaluated 372 frame pairs from 124 Pexels clips with Qwen3-VL-32B-Instruct (Appendix I.6). The UENR-trained model is preferred in 301 comparisons (80.9%), the SynNightRain-trained model in 24 (6.5%), with 47 ties (12.6%).

  • Split: Table 2 compares training datasets on real Pexels videos. The 1,800-frame split listed in Appendix A, Table 5 is used for synthetic validation, not that real-video comparison. Specifically:
    • UENR CitySample 500K (synthetic): 498,200 training frames and 1,800 in-distribution validation frames. Table 9 uses these validation frames to measure reconstruction within the training environment; this split does not establish scene- or trajectory-level independence.
    • UENR Kyoto 100K (synthetic): additional resources for future research.
    • Pexels videos (real): 11,160 test frames, used for the dataset comparison in Table 2.

 

W2: Human-evaluation protocol, temporal ratings and PDF/project-page consistency.

We address the count and reporting questions in Q1.1–Q1.2 below, and the protocol and temporal-evaluation questions in Q2.1–Q2.3.

 

W3.1: The method comparison is also confounded; Table 3 does not isolate the Wan architecture or UENR itself.

Role of pretraining:

We trained Our Baseline without its pretrained denoiser weights, using the same LoRA adaptation procedure. The VAE and text encoder remain pretrained. Both the SynNightRain and UENR runs produce noise-like outputs without recognizable scene structure on the evaluated Pexels clip (Figure 20; Appendix I.5). These short adaptation runs do not isolate pretraining under matched training budgets.

Isolating UENR's contribution: This is already provided by the dataset comparison in Table 2, specifically the "Our Baseline" row, rather than the method comparison in Table 3.

 

W3.2: The related-work positioning is too broad.

Regarding the positioning, UENR's novelty is the dataset as a whole, including UE5 in-scene lighting and occlusion, scale, and its benchmark. This is why the paper is titled "UENR-600K". Your summary is accurate and matches the contribution claimed in our paper.

Regarding the related works you mentioned:

  • GTAV-NightRain contains in-scene rain images, but no video sequences for training video models.
  • LARA, introduced with ASF-Net, overlays Direct3D10-rendered rain streaks globally onto mostly daytime videos. In comparison, UENR provides localized nighttime rain with scene-dependent occlusion, such as rain curtains behind buildings. LARA is included in Table 1.
  • Neither provides the same paired nighttime-video resource covering rain-induced effects such as rain curtains and glimmer (Figure 2).

Q1.1: Please provide selection counts for each evaluator and method in Table 3, win/tie/loss counts for Table 2, and the denominator and tie-handling rules.

Appendix H: Raw Results provides the counts and individual human responses. Qwen multi-way counts are reconstructed from the archived table and its 124-comparison protocol.

  • Table 2 denominator: wins + losses + ties.
  • Table 3 denominator: the sum of selections across methods, with no tie category.
  • Ties: converted directly to percentages, like wins and losses, without further processing.

 

Q1.2: Why do the PDF values of 95.2%/95.5% differ from the project-page values of 97.6%/96.3%?

95.2% in Table 3 is the final human-evaluation result. Thank you for pointing this out. We have revised the paper to quote this human result explicitly and will revise the project page as well.

 

Q2.1: Please describe the complete human-evaluation protocol.

  • Number of annotators: 21.
  • Videos or frames: videos.
  • Judgments per annotator: 8 dataset comparisons (two-way, including ties) and 2 method comparisons (multi-way), giving 168 + 42 = 210 selections overall.
  • Randomization: candidate order was randomized.
  • Blinding: method names were hidden and candidates were shown anonymously. Figures 9 and 10 illustrate the two video-based evaluation tasks.
  • Aggregation: selections were summed and converted to percentages, without further processing.
  • Inter-rater agreement: To maximize randomization, evaluators judged different video comparisons rather than repeated, identical comparisons. We therefore cannot estimate inter-rater agreement. Individual responses are now provided in Tables 18 and 19 in Appendix H for direct inspection.

 

Q2.2: Why does Appendix A describe the real set as VLM-only while Tables 2/3 report human results?

The VLM-only description in Appendix A was a reporting error. Tables 2 and 3 are correct, and Appendix A now states that the real set is used for both VLM and human evaluation. Thank you for pointing this out.

 

Q2.3: Please specify the Qwen checkpoint, decoding settings, prompt and repeated runs, and justify using four midpoint frames for temporal evaluation.

  • Qwen version and checkpoint: Qwen3-VL-32B-Instruct.
  • Decoding settings: temperature 0.7, top_p 0.8, top_k 20 and repetition_penalty 1.0.
  • Prompts: the same as Claude's, provided in Appendices C–D.
  • Repeated runs: one run per item.
  • Four midpoint frames: sparse sampling, such as one frame per second, leaves gaps too large to assess local flicker. We instead use four consecutive midpoint frames and three difference maps to evaluate sampled-window temporal consistency. Full-clip AFD complements this evaluation across all 89 adjacent frame pairs.

Q3.1: Was a matched-scale or matched-scene comparison performed?

Regarding dataset scale, we added a matched-frames experiment, training Our Baseline on 11,880 UENR frames, close to SynNightRain's 11,915. The remaining training settings match those in the paper.

We evaluated 372 frame pairs from 124 Pexels clips with Qwen3-VL-32B-Instruct (Appendix I.6). The UENR-trained model is preferred in 301 comparisons (80.9%), the SynNightRain-trained model in 24 (6.5%), with 47 ties (12.6%).

 

Q3.2: Are the 1,800 frames independent of training at the scene, virtual-environment and camera-trajectory levels?

As explained in W1, these are in-distribution validation frames, not the test frames used in the main real-video experiments. They share the training environment and continuous camera trajectory and measure reconstruction within that environment. The main experiments use the separate 11,160-frame Pexels real-video test set, with independent scenes and trajectories.

We have revised "1,800 test frames" to "1,800 validation frames" in the paper to avoid confusion.

 

Q3.3: Please explain why Kyoto is excluded from the benchmark.

We did not exclude Kyoto from the dataset. Its 100K paired frames are provided as additional resources for future research.

On two 90-frame Kyoto clips, the CitySample-trained baseline removes visible rain without further finetuning (Figure 19; Appendix I.4). Mean RGB PSNR is 22.57 dB, compared with 23.45 dB for the rainy inputs. Texture and brightness differences remain, so pixel-level fidelity does not improve on these clips.

Reviewer_mLKJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work introduces UENR-600K, a large-scale synthetic dataset for nighttime video deraining, generated by Unreal Engine 5. Unlike conventional datasets that overlay rain streaks onto existing images, the proposed dataset simulates rain particles in 3D scenes, aiming to capture nighttime-specific effects such as colored rain streaks, localized illumination, glimmer effects, and rain curtains.

Furthermore, the authors adapt the pretrained Wan 2.2 video generation model for video deraining using shared positional embeddings and a unidirectional attention mask. For validation, they train 7 deraining methods on their dataset and existing datasets, and compare their performance using PSNR/SSIM, VLM-based evaluation, and human preferences. The results suggest that the proposed dataset improves real-world generalization, and the Wan-based baseline achieves the best perceptual quality among the compared models.

Critical strengths
  1. A high-quality large-scale dataset addressing a limitation of existing methods.
  2. Results across multiple architectures and datasets that can inspire follow-up work.
Critical weaknesses
  1. UENR-600K differs from existing datasets in several respects including data scale, scene diversity, and camera configurations, rather than only rain synthesis techniques. The current experiments do not establish how much of the improvement comes specifically from the proposed 3D rain simulation.
  2. The proposed baseline is the only method initialized from a large pretrained video generation model, while the other 6 methods are trained from scratch. It is difficult to attribute the reported performance gap to the proposed architecture or dataset.
Questions for authors
  1. Can you provide more details about the physical rain simulation and rendering? What are the configuration parameters in UE5?
  2. How many people participated in the user study? How many samples did they evaluate?
3 comments

Thank you for the constructive review. We address each concern below.

 

W1: UENR differs in scale, scenes and camera configurations; the experiments do not establish the contribution of 3D rain simulation alone.

Our contribution is the paired nighttime video dataset as a whole; we do not claim that 3D rain alone is the contribution or causes the improvement. Rain particles alone, especially when overlaid globally onto footage, are insufficient. The scene interactions discussed in Related Works include:

  • Rain curtains behind a building.
  • Rain illuminated by a light source behind the camera.
  • Localized rain under a street lamp's light cone.

Regarding dataset scale, we added a matched-frames experiment, training Our Baseline on 11,880 UENR frames, close to SynNightRain's 11,915. The remaining training settings match those in the paper.

We evaluated 372 frame pairs from 124 Pexels clips with Qwen3-VL-32B-Instruct (Appendix I.6). The UENR-trained model is preferred in 301 comparisons (80.9%), the SynNightRain-trained model in 24 (6.5%), with 47 ties (12.6%).

 

W2: The proposed baseline is finetuned while the other six methods are trained from scratch, making it difficult to attribute the performance gap to the model or dataset.

The two main result tables clarify the attribution: Table 2 evaluates the dataset contribution, and Table 3 the model contribution.

  • Table 2 attributes the performance gap to the dataset. Each row compares different datasets using the same model. The Average row summarizes the dataset benefit across models.
  • Table 3 attributes the performance gap to the model. Each column compares different models trained on the same dataset. The Average column compares model performance across datasets.

 

Q3: Please provide the physical rain simulation, rendering and UE5 configuration details.

Rain is rendered in Unreal Engine 5.3 world space using Niagara particles and a translucent Default Lit sprite material with volumetric non-directional lighting. Gravity and wind drive particle motion; scene geometry determines collisions and visibility. Scene-color backlighting and a time-varying volume texture produce illuminated streaks and glimmer.

Relevant parameters: nominal rain emission: 0.2–1.5 million particles/s; camera-following sampling cylinder: 10 m radius and 3 m height (approximately 942 m³), with a 5 m vertical offset; particle lifetime: 2.08 s; initial rain-curtain sprite size: 5–45 m; rain-streak motion scaling: 0.5–1.5; gust contrast: 0.3–1; wind-strength coefficient: 0–9; wind direction: 45–210°; rain opacity: 0.1–0.5; camera focal length: 10–120 mm; camera aperture: f/2.8–f/12; camera white-balance temperature: 3,000–7,000 K; camera white-balance tint: −0.12–0.20; and fixed camera focus distance: 1,000 m.

Further implementation details, material equations and the parameter table are provided in Appendix G.

 

Q4: How many people participated in the user study, and how many samples did each evaluate?

We invited 21 human evaluators, each making 8 pairwise dataset-comparison selections and 2 multi-way method-comparison selections, totaling 168 + 42 = 210 judgments.

  • In two-way comparisons, evaluators choose the better output or a tie.
  • In multi-way comparisons, evaluators choose the best output.

Figures 9 and 10 illustrate the two video-based evaluation tasks.

Thank you for the effort! These clarifications have addressed most of my concerns, though some details regarding comparisons with baselines are still subtle. However, as a dataset work, it can definitely bring something useful into the community. I am glad to raise my score.

Thank you very much for the constructive feedback, and thanks for raising the score. Your feedback has helped us improve the paper.

Reviewer_NqefOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the problem of nighttime video deraining, emphasizing the challenges posed by complex interactions between rainfall and artificial lighting that are inadequately represented by conventional 2D synthetic rain overlays. To tackle this, the authors introduce UENR-600K, a large-scale dataset comprising 600K pairs of 1080p video frames generated using 3D particle-based rainfall simulation within Unreal Engine 5. In addition, they present a video-to-video deraining baseline by adapting the Wan 2.2 Video DiT model with unidirectional attention masking. Extensive benchmarking shows that models trained with UENR-600K achieve notable improvements in removing rain artifacts while preserving temporal consistency in challenging night scenes.

Critical strengths
  1. Physically Grounded 3D In-Scene Rain Simulation: Moving beyond simplistic 2D additive rain overlays, this physically grounded formulation effectively incorporates spatial depth occlusions, localized illumination near light cones, volumetric rain curtains, and chromatic refractions that are essential for nighttime scenes. Previous works do not carefully consider these effects.
  2. Consistent Cross-Architecture Real-World Generalization Gains: The paper conducts a thorough data-centric benchmark comparing models trained on UENR-600K against SynNightRain and WeatherWeaver under matched iteration budgets. Retraining diverse architectures spanning CNN, RNN, Transformer, and Diffusion backbones demonstrates that the proposed 3D simulation data universally improves rain removal and detail preservation on real-world nighttime videos.
Critical weaknesses
  1. Missing Rain Simulation and Rendering Details: Despite framing UENR-600K as a physically grounded dataset, the paper provides virtually no implementation details on the rendering pipeline. Section 3.2 contains only 11 lines of generic description, omitting the particle system, optical shader models, and numerical distributions of weather parameters.
  2. Discrepancy Between Claimed Dataset Scope and Actual Benchmark: The paper advertises UENR-600K as covering North American skylines (Matrix City) and Eastern alleyways, yet Appendix Table 5 reveals that 100,000 frames from the Kyoto subset were completely omitted from all benchmark experiments. Discarding one-sixth of the dataset without explanation weakens the claimed scene diversity. It also misses an ideal opportunity to test cross-scene generalization from wide avenues to densely lit alleyways.
  3. Generative Hallucination in Severe Conditions (Minor): Adapting a 5B video foundation model introduces generative hallucination under heavy rain and extreme low-light scenes. In challenging examples like the backlit silhouette in Figure 6 and supp video, the baseline invents detailed facial features absent in the input, causing subtle cross-frame inconsistencies. While generative priors help remove heavy rain, the paper lacks discussion of the trade-off between hallucination and restoration fidelity.
Questions for authors

Beyond deraining, could the authors discuss how UENR-600K can benefit adjacent domains, such as controllable rain synthesis for autonomous driving simulators or benchmarking downstream perception robustness (e.g., optical flow, object detection) under adverse nighttime weather?

3 comments

Thank you for the constructive review. We address each concern below.

 

W1: Rain Simulation and Rendering Details

Rain is rendered in Unreal Engine 5.3 world space using Niagara particles and a translucent Default Lit sprite material with volumetric non-directional lighting. Gravity and wind drive particle motion; scene geometry determines collisions and visibility. Scene-color backlighting and a time-varying volume texture produce illuminated streaks and glimmer.

Relevant parameters: nominal rain emission: 0.2–1.5 million particles/s; camera-following sampling cylinder: 10 m radius and 3 m height (approximately 942 m³), with a 5 m vertical offset; particle lifetime: 2.08 s; initial rain-curtain sprite size: 5–45 m; rain-streak motion scaling: 0.5–1.5; gust contrast: 0.3–1; wind-strength coefficient: 0–9; wind direction: 45–210°; rain opacity: 0.1–0.5; camera focal length: 10–120 mm; camera aperture: f/2.8–f/12; camera white-balance temperature: 3,000–7,000 K; camera white-balance tint: −0.12–0.20; and fixed camera focus distance: 1,000 m.

Further implementation details, material equations and the parameter table are provided in Appendix G.

 

W2: Why does the 600K dataset include Kyoto while the benchmark excludes it, and does the model generalize across scenes?

We are not "discarding one sixth of the dataset". UENR provides 500K CitySample frame pairs for the benchmark and 100K additional Kyoto pairs for further research. The main generalization experiment evaluates synthetic-trained models on real Pexels videos, testing sim-to-real generalization.

On two 90-frame Kyoto clips, the CitySample-trained baseline removes visible rain without further finetuning (Figure 19; Appendix I.4). Mean RGB PSNR is 22.57 dB, compared with 23.45 dB for the rainy inputs. Texture and brightness differences remain, so pixel-level fidelity does not improve on these clips.

 

W3: Hallucination under severe conditions and the trade-off with restoration fidelity.

Thank you for pointing this out. Generative restoration methods face a trade-off between hallucination and restoration fidelity.

  • Quantitatively, PSNR (on synthetic data with ground truth) measures the combined pixel error from incomplete rain removal and unwanted changes. Even including possible hallucinations, Our Baseline achieves 25.49 dB on UENR, exceeding the six-method mean by 2.89 dB and the strongest comparator, ESTINet, by 0.61 dB.
Training and evaluation datasetOur Baseline PSNR (dB)Six-method mean (dB)Best follow-up (dB)
UENR25.4922.6024.88 (ESTINet)
  • Qualitatively, we added an experiment that "derains" clean videos (Figure 15), so pixel changes directly expose unwanted modifications. Differences concentrate around edges and textured regions, while smoother backgrounds remain comparatively unchanged.

 

Q1: Can UENR support controllable rain synthesis and downstream perception research?

Yes! We added an experiment that trains Our Baseline with the inputs and outputs swapped: clean UENR videos as inputs and rainy videos as outputs. The rain-synthesis results in Figure 18 show:

  • Rain streaks and curtains illuminated by colored light.
  • Rain concentrated in localized regions.
  • Scene-dependent occlusion without requiring a 3D reconstruction of the input video.

This scene-aware synthesis could generate rain videos at scale for downstream research, such as autonomous driving and perception robustness studies.

In addition, UENR is rendered in Unreal Engine, which supports pairing the videos with the rain-control parameters listed in W1. These pairs can provide training data for controllable rain-synthesis models.

Thank you for the authors' detailed responses to my questions and for addressing my concerns. I also checked the updated draft. The additional content and the experiments are helpful.

Additional remarks:

  • "100K additional Kyoto pairs for further research": this should be stated earlier in the main paper. The dataset usage ablation experiments are based solely on CitySample rendering.
  • In the updated rendering details, the authors should better disclose the sources of the UE 3D assets.

My concerns are mostly addressed, and I can raise my score to the positive side of borderline.

This dataset contribution is valid and useful, but its application scope, in my opinion, is still limited.

Thank you very much for your reply, and thanks for raising the score.

We will clarify the CitySample/Kyoto usage earlier in the main paper, and add links to the UE assets we used. Thanks again for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.