acceptodds
Under review as a conference paper at ICLR 2027

WeatherShift: Transitional Weather Video Generative Modeling for Autonomous Driving Perception

Abstract

Robust perception for autonomous driving requires generalizing beyond discrete weather categories to the continuously evolving conditions encountered on the road. Existing autonomous-driving datasets and perception methods predominantly represent weather as fixed states such as clear, rain, fog, or snow, ignoring the transitions between them, yet real driving sequences pass through intermediate regimes, such as sunny to rainy or rainy to sunny, with continuously varying levels of adversity. Recent weather-transition generation methods primarily operate at the image level and typically assume a static background, limiting their ability to capture the temporal evolution of weather alongside the dynamic scene content and ego-motion inherent to real-world driving videos. To address these limitations, we propose WeatherShift, a video-based framework for generating temporally coherent weather transitions in autonomous-driving scenes. Starting from real-world driving videos in nuScenes, we construct extreme-weather counterparts, namely Rainy-nuScenes, Foggy-nuScenes, and Snowy-nuScenes, by applying weather-specific scene transformations and atmospheric effects. We then generate intermediate weather-transition videos in which weather attributes progressively evolve while scene content, ego-motion, and background elements dynamically change across frames. Building on this framework, we introduce nuScenes-WT (Nuscenes Weather Transition Video Dataset), comprising twenty directional transition categories spanning sunny, rainy, foggy, snowy, and cloudy conditions. We further evaluate the generated videos for downstream object detection and semantic segmentation, establishing a unified generative-to-perceptual benchmark for transitional weather. Experimental results demonstrate that WeatherShift generates visually coherent weather transitions while preserving temporal scene dynamics, and that nuScenes-WT provides a challenging testbed for evaluating perception under continuously changing weather conditions.

Primary area:
datasets and benchmarks
Keywords:
Autonomous driving, Transitional Weather, Video Generation, Scene Perception

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9rEpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

WeatherShift proposes a two-stage framework for generating transitional-weather driving videos. It first constructs frame-aligned extreme-weather references from nuScenes clips using depth-guided fog synthesis or prompt-driven weather editing with particle effects. It then estimates a weather-change mask and applies a temporal severity schedule to blend or transform the source and target weather states. Transitions between two non-sunny conditions are composed through an intermediate sunny state. The authors introduce nuScenes-WT, containing 20 directional transition categories generated from 1,000 front-camera nuScenes scenes, and evaluate both video preservation and downstream detection/segmentation behavior.

Critical strengths

None

Critical weaknesses
  1. The methodological novelty is limited. The framework mainly consists of two parts: extreme-weather generation, which applies several customized transformations for a small set of weather types, and transition synthesis, which primarily performs temporal weighted blending between weather references. The overall design therefore appears to combine task-specific weather effects with a simple frame-wise transition mechanism, without a sufficiently novel modeling contribution.

  2. The experimental comparison is insufficient. The quantitative metrics mainly measure scene and object preservation, but do not directly evaluate the quality or realism of the generated weather effects. The baselines are limited to general-purpose closed-source commercial video models, with no comparison against dedicated autonomous-driving generation or weather-simulation methods. In addition, the qualitative comparison figure is too low-resolution to support a reliable visual assessment.

Questions for authors
  1. How does the method compare with autonomous-driving generation and weather-editing approaches when the two stages are evaluated separately: extreme-weather generation and weather-transition synthesis?
  2. Can the authors report quantitative measures that directly assess the quality and realism of the generated weather effects, beyond scene and object preservation?
  3. What additional experimental results and evidence support the claimed improvements in downstream perception?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Comment: Methodological Novelty and Unified Framework Contributions of WeatherShift

We thank the reviewer for this observation. We would like to clarify that WeatherShift does not claim a novel standalone weather editor or a new mathematical blending operator. Rather, its contribution lies in integrating weather-specific reference synthesis, spatial weather-change localization, and temporally controlled transition composition into a unified framework for generating directional weather transitions in autonomous-driving videos.

  • Frame-Aligned Extreme-Weather Reference Synthesis: The first stage generates frame-aligned extreme-weather references from the same original driving sequence. Beyond appearance editing, rain and snow synthesis incorporates temporally evolved 4D Gaussian particle representations to model dynamic precipitation and scene-dependent occlusions. This ensures that the weather references retain the original scene configuration, camera viewpoint, and temporal evolution.
  • Spatially Selective & Direction-Controlled Transition Synthesis: The second stage goes beyond uniform frame-wise interpolation between complete weather videos. It estimates spatial weather masks from the original and corresponding weather-transformed frames, thereby localizing weather-induced changes. These masks are progressively modulated using an explicit temporal severity schedule, enabling spatially selective, direction-controlled transitions while preserving the underlying scene dynamics. Furthermore, transitions between two adverse weather conditions are composed through an intermediate Sunny state, allowing a common transition operator to support multiple weather directions.

Thus, the contribution is not the individual use of existing editing or blending operations, but their integration into a unified, severity-controlled framework for constructing spatially localized and temporally aligned weather transitions. The additional downstream perception experiments further demonstrate the practical necessity of these transitions: models trained on discrete weather conditions exhibit substantial performance degradation when evaluated on transitional-weather sequences, while nuScenes-WT provides a dedicated benchmark to train and evaluate perception under these evolving conditions.

Thank you for the authors' response. However, it did not provide additional useful information, so I will maintain my score.

AuthorsOpenReview ↗

Evaluation of Weather Realism, Generation Quality, and Temporal Consistency

We thank the reviewer for this important and insightful observation. We clarify that our initial focus on scene-preservation metrics was intended to establish a critical baseline ensuring that WeatherShift does not hallucinate objects or warp road geometry (a frequent failure mode of general-purpose video generators). However, to directly address the reviewer's concern regarding weather realism and temporal dynamics, we have expanded our evaluation across two dedicated axes:

  1. Temporal Coherence & Motion Consistency: To verify that our weather transitions evolve smoothly without flickering, warping, or ghosting artifacts, we incorporate RAFT Optical Flow Warping (OFW) Error (0.0370.037 vs. 0.0420.042), DINOv2 Feature Similarity (0.860.86 vs. 0.820.82), and the RAFT Flow Acceleration Ratio (0.540.54 vs. 0.520.52).
  2. Generation Quality & Realism Alignment: To rigorously assess whether our synthesized weather effects match real-world adverse conditions, we evaluate distributional alignment using Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and CLIP Maximum Mean Discrepancy (CMMD) against established real-world benchmarks (ACDC and BDD100K).

To make these enhancements explicit, we have integrated the following quantitative evaluations into the revised manuscript:

1. Temporal Coherence and Motion Consistency (Table 4)

Evaluates frame-to-frame stability and motion preservation across transitional sequences.

MetricsRAFT OFW Error ↓\downarrowDINOv2 Feature Similarity ↑\uparrowRAFT Flow Acceleration Ratio ↓\downarrow
Discrete Baseline0.0420.820.52
NuScenes-WT (Ours)0.0370.860.54

2. Generation Quality and Realism Alignment (Table 6)

Assesses distributional alignment across 100 scenes against real-world adverse-weather benchmarks.

Dataset ComparisonFID ↓\downarrowKID (×103\times 10^3) ↓\downarrowCMMD ↓\downarrow
nuScenes vs. ACDC65.3022.710.72
nuScenes vs. BDD100K75.1515.180.61
nuScenes-WT vs. ACDC90.5528.100.83
nuScenes-WT vs. BDD100K84.1220.710.70
nuScenes-WT vs. nuScenes8.148.000.002

By coupling these generation quality metrics with our content preservation scores, we demonstrate that WeatherShift achieves high weather realism without compromising structural scene integrity. We have updated Section 3 of the revised manuscript to include these analyses.

Reviewer_K9P9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WeatherShift generates weather transitions in driving videos by combining weather-specific reference generation with spatially masked, temporally scheduled blending. The paper also introduces nuScenes-WT for evaluating perception under changing weather conditions.

Critical strengths

nuScenes-WT contains 20,000 transition videos derived from 1,000 driving scenes, covering 20 directional transitions with evolving scene content, which is useful for AV. Table 3 reports substantially better object-preservation recall and matched-object IoU than the compared generators.

Critical weaknesses

(1) Weather-specific endpoints rely on existing editors, while the central transition operation is masked blending. Comparisons with general-purpose generators do not establish the advantage over simple blending or temporally varying weather-rendering parameters; component ablations are also missing. (2) The reported metrics primarily assess framewise content preservation, not temporal coherence or weather-transition realism. These scores can favor weaker editing, so preservation should be evaluated alongside weather fidelity at comparable severity levels.

Questions for authors

Please refer to weaknesses,  another question is as follows. Which CLIP-S definition is used: original–generated image similarity in Section 3.1, or alignment with the target weather description in Section 3.2? These measure different properties.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Comment: "Weather-specific endpoints rely on existing editors, while the central transition operation is masked blending."

We would like to clarify that WeatherShift is designed as a multi-stage framework for generating spatially and temporally controlled weather transitions, rather than proposing a new standalone weather editor or an entirely new blending operator. The framework consists of three interconnected stages: (i) weather-specific endpoint synthesis, which employs text-guided editing and temporally evolved 4D Gaussian particle representations for dynamic rain and snow effects; (ii) spatial weather-mask estimation, which localizes weather-induced visual changes while preserving scene content; and (iii) temporally controlled transition generation, which uses a frame-wise severity schedule to progressively introduce weather effects in a spatially constrained manner.

AuthorsOpenReview ↗

Technical Advantages Over Simple Blending and Procedural Weather Rendering

Response to Reviewer Comment: Technical Advantages Over Simple Blending and Procedural Weather Rendering

We thank the reviewer for this insightful critique. We acknowledge that simple 2D pixel-level alpha blending or procedural parameter variation (such as linearly ramping fog density or rain particle opacity over time) represent traditional heuristic alternatives. However, we respectfully clarify why such simplistic approaches fail to solve the unconstrained video editing problem in autonomous driving, and how WeatherShift provides fundamental technical advantages over them:

  • Inadequacy of Simple 2D Alpha-Blending: Naive pixel blending operates entirely in 2D space without understanding scene depth, geometry, or object boundaries. Consequently, it results in severe visual artifacts such as "ghosting" effects on moving vehicles, uniform atmospheric layers plastered indiscriminately over foreground and background objects, and a total failure to model realistic occlusion across depth gradients.
  • Limitations of Procedural Parameter-Tweaking: While traditional rendering engines can vary weather parameters over time, they require precise 3D asset reconstructions, accurate depth maps, and relighting controls that are unavailable in unconstrained real-world driving benchmarks like nuScenes. Applying parameter modulation to raw video frames without 3D awareness leads to unnatural contrast flattening and texture washing.
  • The Technical Solution of WeatherShift: To overcome these limitations, WeatherShift introduces a principled, physics-grounded architecture via 4D Gaussian particle representations and spatial weather-mask estimation. Instead of naive blending, our framework dynamically decouples atmospheric effects from scene structure, ensuring that weather phenomena interact correctly with surface normals and depth gradients while preserving fine-grained object configurations and bounding boxes.

To substantiate that WeatherShift achieves meaningful, physically grounded weather transformations rather than superficial overlays, we expanded our empirical evaluation beyond general-purpose video editors (OmniFlash, Seedance, and Kling) to incorporate three rigorous validation axes:

  • Generation Realism & Distribution Alignment (Table 6): Evaluated across 100 scenes using FID, KID, and CMMD, confirming that our generated transitions align closely with real-world adverse-weather distributions (ACDC and BDD100K) while remaining faithful to the natural driving domain (8.14 FID relative to nuScenes).
  • Temporal Coherence & Motion Consistency (Table 4): Evaluated via RAFT Optical Flow Warping Error (0.0370.037 vs. 0.0420.042), DINOv2 Feature Similarity (0.860.86 vs. 0.820.82), and RAFT Flow Acceleration Ratio (0.540.54 vs. 0.520.52), proving smooth, flicker-free temporal progression without the warping or ghosting artifacts inherent to naive blending.
  • Downstream Perception Utility (Table 5): Evaluated with RT-DETR and Mask2Former Swin-Large, showing substantial zero-shot degradation on nuScenes-WT (52.2% →\to 31.3% mAP and 68.0% →\to 41.0% mIoU) followed by strong recovery when trained on transitional sequences (45.0% mAP and 57.0% mIoU). This confirms that our synthesized transitions introduce authentic, challenging visual shifts that static datasets and simple overlays fail to capture.
AuthorsOpenReview ↗

Balancing Content Preservation with Temporal Coherence and Weather Fidelity

We completely agree with the reviewer’s observation. To address this concern and establish a rigorous, balanced evaluation, we clarify the distinct roles of our metrics and have extended our experimental analysis across three complementary axes:

  • Distinguishing Preservation from Realism: We clarify that scene-preservation metrics are designed exclusively to verify that structural layout and traffic participant bounding boxes are not catastrophically destroyed during editing, a critical failure mode of general-purpose video generators. However, preservation scores are evaluated in tandem with generation realism.
  • Temporal Coherence & Motion Consistency (Table 4): To ensure our transitions are smooth and physically plausible rather than artifact-ridden, we evaluate frame-to-frame stability using RAFT Optical Flow Warping (OFW) Error (0.0370.037 vs. 0.0420.042), DINOv2 Feature Similarity (0.860.86 vs. 0.820.82), and the RAFT Flow Acceleration Ratio (0.540.54 vs. 0.520.52).
  • Weather Fidelity & Distribution Alignment (Table 6): To prove that meaningful, high-fidelity weather phenomena are successfully synthesized, we incorporate distribution-level metrics, Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and CLIP Maximum Mean Discrepancy (CMMD), evaluated across 100 scenes against real-world adverse-weather benchmarks (ACDC and BDD100K).
  • Controlled Severity Progression: Furthermore, our framework utilizes an explicit frame-wise severity parameter. This enables evaluation across intermediate weather intensities rather than just discrete endpoints, ensuring that weather fidelity is rigorously tracked across the entire transition trajectory.

To make our response self-contained and immediately clear to the reviewers, we have included the corresponding quantitative tables directly below, summarizing our expanded evaluation across temporal consistency, distribution realism, and downstream perception utility.

1. Temporal Coherence and Motion Consistency (Table 4)

Evaluates frame-to-frame stability and motion preservation against discrete baselines.

MetricsRAFT OFW Error ↓\downarrowDINOv2 Feature Similarity ↑\uparrowRAFT Flow Acceleration Ratio ↓\downarrow
Discrete Baseline0.0420.820.52
NuScenes-WT (Ours)0.0370.860.54

2. Generation Quality and Realism Alignment (Table 6)

Assesses distributional alignment across 100 scenes against real-world adverse-weather benchmarks.

Dataset ComparisonFID ↓\downarrowKID (×103\times 10^3) ↓\downarrowCMMD ↓\downarrow
nuScenes vs. ACDC65.3022.710.72
nuScenes vs. BDD100K75.1515.180.61
nuScenes-WT vs. ACDC90.5528.100.83
nuScenes-WT vs. BDD100K84.1220.710.70
nuScenes-WT vs. nuScenes8.148.000.002

3. Downstream Perception Performance (Table 5)

Demonstrates that transitional weather data successfully bridges the generalization gap for perception models.

TaskModelnuScenes (Discrete)nuScenes →\to nuScenes-WTnuScenes-WT (Ours)
Object DetectionRT-DETR52.2%31.3%45.0%
Semantic SegmentationMask2Former Swin-Large68.0%41.0%57.0%

Summary of Revisions

By evaluating these metrics alongside our explicit frame-wise severity parameter, we confirm that WeatherShift achieves high weather fidelity without sacrificing structural integrity. We have integrated these tables and the accompanying analysis into Section 3 of the revised manuscript.

AuthorsOpenReview ↗

Clarification on CLIP-S Definition

We thank the reviewer for catching this ambiguity. To clarify, original–generated image similarity (CLIP-S(eorigi,egeni)\text{CLIP-S}(e_{\text{orig}}^i, e_{\text{gen}}^i)), as defined in Section 3.1, is the metric used for our quantitative evaluation. We have modified Section 3.2 in the revised manuscript to correct this description and ensure consistent, unambiguous terminology across the entire paper.

Reviewer_8Ab7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper argues that while existing autonomous driving datasets span different weather states such as clear, rain, or snow, they rarely contain transitions between such states. They introduce video (editing) generation models that can convert existing flagship datasets (e.g., NuScenes) to weathered variants that importantly, also contain weather transitions. They use their generated data to evaluate downstream perception stacks including object detection and semantic segmentation. They demonstrate that their generated videos are more similar to the original source videos (in terms of feature similarity and detected object overlap) than other video editing approaches.

Critical strengths
  • Calling out weather transitions, in addition to discrete weather states, is an interesting framing for autonomous vehicle perception that I have not seen. Different approaches are used for different weather conditions. Fog is generated using a physics-based scattering model (applied to a video depth reconstruction). For rain, snow, and cloudy weather synthesis, this approach uses an image editing model applied to frames of a video (rather than a video editing model, which could be viewed as a weakness).
Critical weaknesses
  • My biggest concern is evaluation. The gold standard evaluation would be showing that policies trained on such simulated data generalize to the real-world, but this is clearly difficult. As an alternative, I was expecting this work to argue that current perception stacks, while potentially generalizing to different weather conditions, struggle to generalize to different weather transitions (which I might still be skeptical of, more below). But this work doesn't make this argument either. Instead, Table 4 evaluates that the edited video are more similar to the original videos than baseline video editing approaches. This doesn't make sense to me; a trivial baseline that would maximize all metrics in Table 3 is simply to not change the video at all (or change it very little). Clearly that is not a demonstration that the generated edits are meaningful. Am I missing something fundamental?
  • This work doesn't actually mention what detectors and semantic segmentors are used. To make a compelling argument about transitions, I would think that such models should be trained on different weather conditions, but not transitions. One could then evaluate the hypothesis that current multi-weather perception stacks fail to generalize to transitions.
  • Finally, I would be skeptical of the hypothesis above. I would suspect that perception stacks trained for different weather conditions would generalize to transitions since they could be regarded as "interpolations" between the discrete weather conditions observed during training. In fact, the generated transitions videos seem to be constructed by (literal) alpha bending of different-weathered videos (Eq 3).
Questions for authors

Am I missing something fundamental regarding evaluation?

4 comments
AuthorsOpenReview ↗

Generalization to Weather Transitions and Transition Formulation (Eq. 3)

We thank the reviewer for this insightful critique. While transitional weather may appear to be an interpolation between discrete weather conditions, such interpolation does not necessarily imply that perception models trained on discrete weather conditions will generalize reliably to temporally evolving weather. As weather conditions progressively change, variations in pixel intensities, contrast, visibility, and local appearance can introduce intermediate visual patterns and blending artifacts that are not necessarily present in either discrete weather endpoint. These pixel-level variations can alter object boundaries, reduce feature consistency, and affect the visual cues used by perception models, potentially leading to detection and segmentation errors. Therefore, interpolation in image space does not necessarily translate into reliable interpolation in the feature space of perception models.

Clarification on Eq. 3 and Transition Mechanics

We clarify that Eq. 3 is not a naive, indiscriminate alpha-blending of two full videos. The transition pipeline incorporates several structured components:

  • Dynamic Weather Synthesis: For rain and snow, we employ temporally evolved 4D Gaussian particle representations to simulate realistic precipitation physics and scene-dependent occlusions, rather than static image cross-fading.
  • Spatially Localized Weather Masks: We compute explicit spatial masks by comparing original frames against aligned extreme-weather counterparts, ensuring that weather modifications are strictly isolated while background scene geometry and moving traffic participants remain preserved.
  • Severity-Controlled Modulation: The transition operator in Eq. 3 applies weighted blending exclusively to these localized weather residuals guided by a temporal severity schedule, rather than cross-fading whole frames.

This formulation enables directional transitions and precise frame-wise control over atmospheric intensity.

Empirical Verification

To directly test the reviewer's hypothesis, we evaluated RT-DETR (object detection) and Mask2Former (segmentation) models trained solely on discrete weather conditions against our continuous nuScenes-WT transitional sequences. The resulting performance drops in Table 6 of the revised manuscript confirm that static multi-domain training fails to generalize across temporal weather progression, validating the necessity of transitional training data.

AuthorsOpenReview ↗

Downstream Perception Generalization and Extended Quantitative Evaluation

We thank the reviewer for this insightful critique, which motivated us to investigate the downstream impact of transitional weather on autonomous-driving perception models.

While real-world adaptation from synthetic to real transitional driving conditions would provide the most comprehensive validation, such an evaluation is currently constrained by the scarcity of real-world transitional-weather sequences with dense annotations. Following the alternative evaluation suggested by the reviewer, we investigate a fundamental question: Do perception models trained on conventional discrete weather conditions generalize effectively to continuously evolving weather transitions?

To answer this, we conducted rigorous downstream perception experiments using RT-DETR (for object detection) and Mask2Former Swin-Large (for semantic segmentation) across three distinct training and testing configurations:

  1. (i) Training & testing on original nuScenes (discrete baseline conditions).
  2. (ii) Training on nuScenes and testing on nuScenes-WT (zero-shot evaluation without exposure to transitional sequences).
  3. (iii) Training and testing on nuScenes-WT (evaluating models trained directly on transitional-weather data).

Key Findings and Empirical Evidence

Our experiments reveal substantial performance degradation when models trained on conventional discrete weather data are evaluated on continuously evolving transitional-weather sequences:

  • Object Detection (RT-DETR): Performance drops significantly from 52.2% mAP on standard nuScenes down to 31.3% mAP under zero-shot transfer to nuScenes-WT transitional sequences. Training directly on nuScenes-WT recovers performance to 45.0% mAP.
  • Semantic Segmentation (Mask2Former Swin-Large): Performance similarly drops from 68.0% mIoU down to 41.0% mIoU, recovering to 57.0% mIoU when exposed to transitional-weather training data.

These additional experiments and quantitative results have been incorporated into Section 3.2 and Table 5 of the revised manuscript. We believe this evaluation directly addresses the reviewer's suggested alternative and provides empirical evidence of the downstream relevance of nuScenes-WT for autonomous-driving perception research. These findings conclusively indicate that exposure to multiple discrete weather conditions does not guarantee robust generalization to continuously evolving weather transitions, validating the critical necessity of dedicated transitional training data.

AuthorsOpenReview ↗

Validity of Video Generation Evaluation Metrics

We thank the reviewer for this critical observation. Table 3 must be interpreted in conjunction with our broader evaluation framework, as autonomous-driving video editing involves a fundamental trade-off: achieving substantial target weather transformation while avoiding catastrophic structural drift from the source content. General-purpose generative baselines (such as OmniFlash, Seedance, and Kling) frequently fail along this axis by heavily distorting background geometries, hallucinating or deleting moving vehicles, or drifting away from the original camera trajectory—thereby invalidating ground-truth annotations. Thus, Table 3 demonstrates that WeatherShift preserves underlying geometric and semantic layouts where existing editors fail, while our complementary evaluations verify that the injected weather effects are genuinely transformative rather than negligible.

To directly address the reviewer's concern, we have restructured our evaluation in the revised manuscript to explicitly separate these four complementary objectives:

  1. Scene & Label Preservation (Table 3): Evaluated via CLIP-S, DINO-S, Object Recall, and Matched Object IoU against the unmodified source video to prove that structural integrity and traffic participant layouts are strictly retained.
  2. Weather-Generation Quality & Realism (Table 6): Evaluated via distribution-level metrics (FID, KID, and CMMD) comparing our generated sequences against real-world adverse-weather benchmarks (ACDC and BDD100K) to confirm meaningful distributional alignment with target weather conditions.
  3. Temporal Coherence & Motion Consistency (Table 4): Evaluated via RAFT Optical Flow Warping (OFW) Error, RAFT Flow Acceleration Ratio, and DINOv2 Feature Similarity to ensure smooth, physically plausible weather evolution across consecutive frames without flickering artifacts.
  4. Downstream Perception Utility (Table 5): Evaluated by testing discrete-trained RT-DETR and Mask2Former models on nuScenes-WT. A trivial identity baseline would leave perception accuracy virtually unchanged; instead, performance drops substantially (52.2% 31.3% mAP and 68.0% 41.0% mIoU), proving that our generated transitions introduce visually impactful atmospheric shifts that challenge static perception stacks.

Together, these four evaluation axes establish that WeatherShift resolves the core tension between strong atmospheric synthesis and strict spatio-temporal label preservation succeeding where both a trivial identity mapping (no weather transformation) and existing video editors (broken scene consistency) fall short. We have updated Section 3.2 of the revised manuscript to make these distinct evaluation axes explicit.

AuthorsOpenReview ↗

Specification of Perception Architectures and Transition Hypothesis

To clarify our experimental setup and rigorously evaluate this hypothesis, we explicitly detail the perception architectures and training protocols used in our study:

  • Perception Architectures: We utilized two representative, state-of-the-art models: RT-DETR for object detection and Mask2Former Swin-Large for semantic segmentation.
  • Training and Evaluation Protocol: To test whether standard multi-weather perception stacks fail to generalize to transitions, we evaluated models trained exclusively on conventional static, discrete weather conditions (e.g., standard nuScenes) against models trained on our continuously evolving nuScenes-WT transitional data.
  • Empirical Confirmation: Our experiments confirm the reviewer's hypothesis. Models trained only on static, discrete weather states suffer severe performance degradation during intermediate transition phases (e.g., RT-DETR mAP dropping from 52.2% to 31.3% under zero-shot transfer). Conversely, incorporating our generated transitional-weather sequences effectively bridges this generalization gap, proving that exposure to dynamic temporal transitions is vital for robust perception stacks.

We have updated Section 3.2 and Table 5 of the revised manuscript to explicitly state these detector and segmentor architectures, clarify the training configurations, and incorporate these comparative quantitative results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.