acceptodds
Withdrawn from ICLR 2027

TCWeatherGen: Target-Consistent Weather Augmentation for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion editors can convert labeled clear-weather driving images into adverse-weather training data, but often deform task-critical targets such as traffic signs, lights, pedestrians, and vehicles, invalidating their annotations despite plausible weather appearance. Our analysis of FLUX.2 identifies three causes: aligned source attention, which carries a critical source constraint, weakens during denoising; background condition keys divert attention from target-specific support; and the global flow-matching loss gives small regions little spatial weight. We therefore introduce **TCWeatherGen**, which combines **SegRGBHeat** for sparse semantic anchoring, **Source–Condition LoRA (SC-LoRA)** for object-aware pathway adaptation, and **Content-Aware Token Pruning (CATP)** for suppressing uninformative condition tokens. On ACDC weather transfer, our fine-tuned 4B student improves target ROI SSIM from to and FID from to against matched FLUX.2-32B-dev pseudo-targets. CATP reduces peak training and inference memory by 10 and 3 GB and inference time by . TCWeatherGen augmentation further improves downstream mIoU and mAP from and to and .

Primary area:
generative models
Keywords:
Diffusion Models, Autonomous Driving, Adverse-Weather Data Augmentation, Target-Consistent Image Editing, Parameter-Efficient Fine-Tuning, Token Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_eAveOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TCWeatherGen to mitigate small-target drift during weather augmentation while preserving the structural and semantic integrity of road scenes. The proposed framework addresses three major challenges: weakening of source-image evidence, competition among condition tokens, and area-dominated supervision. To overcome these limitations, TCWeatherGen integrates three complementary components: SegRGBHeat, SC-LoRA, and CATP. SegRGBHeat strengthens source-image guidance to preserve small-target details, while SC-LoRA minimizes interference among weather-conditioning signals. CATP addresses area-dominated supervision by emphasizing small yet critical objects during the generation process. Together, these components enable realistic weather augmentation while maintaining spatial consistency and improving the preservation of small targets under diverse weather conditions.

Critical strengths
  • The paper investigates target degradation during Diffusion Transformer (DiT) based weather editing, where critical objects such as vehicles, pedestrians, and traffic signs can be distorted even when the generated weather looks realistic. This is an important problem because such distortions can invalidate the original annotations and reduce the usefulness of synthetic data.

  • By emphasizing object-aware supervision and source-content preservation, the method targets a practical limitation of synthetic driving datasets: visually plausible images may still contain incorrect object geometry or semantics. Improving target consistency could make generated data more reliable for downstream detection and segmentation. Further, Achieved computational efficiency via Context AwareToken Pruning.

Critical weaknesses
  • The proposed method combines existing ideas. The paper needs to clarify what is the benefit of combining these methods over straight forward integration. Is there any theoretical guarantee that combining these methods would make annotations remain valid.

  • The qualitative figures shown, especially in figure 9 the traffic sign has no fog, rain, or snow. But realistic adverse-weather images should show weather effects on traffic signs, vehicles, pedestrians, and other objects when physically appropriate. The proposed generative approach may produce unrealistic images.

  • Sparse semantic anchors may fail to preserve small objects, such as distant traffic signs and pedestrians, and their precise boundaries, potentially compromising annotation validity and downstream perception.

  • Condition-token pruning may inadvertently remove tokens representing small or distant objects, such as traffic signs and pedestrians, potentially degrading image quality and annotation consistency.

  • The evaluation is limited to the ACDC dataset. Evaluating the method on additional autonomous-driving datasets would help establish its generalizability across different environments and adverse-weather conditions.

  • Although the authors report downstream detection and segmentation results, it remains unclear whether the original annotations are independently verified after weather editing. Can the authors provide direct evidence of annotation consistency, particularly for small and distant objects.

  • There are typographical mistakes Line 470, 374.

Questions for authors
  • The metrics used for evaluation FID, LIPSIS, SSIM measure the visual comparison. Further downstream metrics show perception performance. How can we verify that weather editing doesn’t distort annotated critical objects?

  • How can we ensure that preserving object structure does not suppress realistic weather effects?

  • Although downstream detection and segmentation results are reported, it remains unclear whether the original annotations are independently verified after weather editing. Provide direct evidence of annotation consistency, particularly for small and distant objects.

Reviewer_iypHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TCWeatherGen addresses object preservation during weather editing through sparse RGB conditioning, separate source–condition LoRA adapters, target-aware supervision, and condition-token pruning. The downstream gains are promising, but teacher-dependent generation evaluation, insufficiently controlled comparisons, and unclear data protocols limit confidence in the method’s contribution and generalization.

Critical strengths

On real adverse-weather images, the proposed augmentation improves four-class mIoU over base-model augmentation. CATP reduces training memory by 10 GB and inference time by 35.8%, with slightly improved generation metrics.

Critical weaknesses

(1) Vanilla LoRA uses half the adapter parameters of the full model, while the target-aware loss and pseudo-target quality-control pipeline lack separate ablations. A parameter-matched shared-adapter baseline, using identical conditions and supervision, would better establish the benefit of separating the two pathways. (2) Although downstream augmentation is restricted to training images, the generator’s fine-tuning set size and its separation from both evaluation sets remain unclear. The provenance and alignment of the “Cityscapes-compatible” annotations on ACDC clear images also need explanation. (3) Removing aligned source attention demonstrates its importance, but does not establish that its natural decline during denoising causes target corruption. Selective interventions at different denoising stages, paired with direct preservation measurements, would better support this claim.

Questions for authors

How do downstream results compare with augmentation from the 32B teacher and a same-data fine-tuned Vanilla LoRA model, rather than only the base editor?

Reviewer_Um4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper address object preservation during weather editing through sparse RGB conditioning, separate source–condition LoRA adapters, target-aware supervision, and condition-token pruning. It uses three components: SegRGBHeat, SC-LoRA, and CATP. SegRGBHeat makes guidance stronger for preserving small details, SC-LoRA aims to reduce interference among weather signals. Finally CATP provides supervision by emphasizing small and critical objects. These components enable spatial consistency while allowing weather augmentation.

Critical strengths

It addresses important problem as generating weather augmented data often leads to distortion in objects such as vehicles, pedestrians, and traffic signs. The proposed approach improves class mIoU, maintains competitive results in generation and reduces inference time.

Object aware supervision and content preservation is a sensible approach and have demonstrated to work well and achives computational efficiency.

Critical weaknesses

The primary quality metrics: teacher-relative ROI SSIM, FID, and KID, all measure agreement with FLUX.2-32B-dev pseudo-targets, the same model used to generate training supervision. The authors note this in the limitations, but the concern runs deeper: the teacher itself may degrade targets and the paper provides no independent ground-truth measure of target structural fidelity across the full test set, human evaluation, source-relative ROI SSIM is only briefly mentioned in Appendix A.1.

Table A.1 shows night ROI SSIM of 0.392, substantially below snow (0.727), fog (0.837), and rain (0.639). This is not discussed and would be helpful to discuss about this result. Night is more practically important condition similar to fog.

"Source-LoRA only, no Cond-LoRA" ablation would be helpful as the ablation “w/o SC-LoRA” replaces the dual-branch LoRA with vanilla single-branch LoRA, while “w/o Source-LoRA” removes only the source branch. These reveal that source branch drives most of the improvement, but it remains unclear whether the decoupling is what matters or simply adding more source-side capacity.

the Gaussian blur radius σ is unspecified in the main text.

CATP efficiency claim needs stronger evidence, latency, FLOPs for each component on various seeds to get a clear picture of the efficiency gains.

Questions for authors

How does source-relative ROI SSIM (the complementary measure mentioned in B.2) behave across all conditions? Does optimizing for teacher agreement ever conflict with preserving the original clear-weather annotation geometry?

The “h = max(M, G_σ(M))” formulation preserves a decaying halo, but why max rather than a simple dilation?

Provide ablation for "Source-LoRA only, no Cond-LoRA" to isolate the contribution

The paper should compare against at least one domain-specific weather-transfer baseline like "Controllable Weather Synthesis and Removal with Video Diffusion Models"[1], Syndiff-ad[2], Controluda[3]

Does optimizing for teacher agreement ever conflict with preserving the original clear-weather annotation geometry?

[1] Lin, Chih-Hao, et al. "Controllable weather synthesis and removal with video diffusion models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[2] Goel, Harsh, et al. "Syndiff-ad: Improving semantic segmentation and end-to-end autonomous driving with synthetic data from latent diffusion models." arXiv preprint arXiv:2411.16776 (2024).

[3] Shen, Fengyi, et al. "Controluda: Controllable diffusion-assisted unsupervised domain adaptation for cross-weather semantic segmentation." arXiv preprint arXiv:2402.06446 (2024).

Reviewer_njTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies preservation of annotated traffic objects when diffusion editors transform clear driving images into adverse-weather training examples. It analyzes aligned source attention, background condition keys, and the weighting of training objectives. Building upon the findings through analysis, the paper introduces TCWeatherGen that combines a sparse RGB condition for sparse semantic anchoring, object-aware supervision, and conditon-aware pruning. The results are reported with better accuracy and lower memory/runtime on the primary snow evaluation.

Critical strengths

The paper addresses annotation validity directly and supports its motivation with a controlled intervention. Removing aligned source-support attention produces large output changes than equal-cardinality shifted or random removals. This motivates preserving spatial correspondence for small traffic objects, where visually plausible editing can nevertheless invalidate supervision. Besides, the reported gains extend beyond image similarity, with satisfactory efficiency improvement.

Critical weaknesses

The major issue is that Tables 1 and A.1 measure similarity to the same teacher used to supply training targets, which establishes teacher agreement rather than directly validating label fidelity. Appendix A discusses source-relative ROI SSIM but does not actually report that column. Although teacher targets undergo quality screening, this does not directly establish label preservation in held-out student outputs.

Besides, the experiments do not sufficiently separate the proposed routing mechanism from adaptation capacity and target-weighted supervision. Meanwhile, Tab. 2 does not isolate the target-aware loss or the two-level target-quality screening. Consequently, improvements may partly reflect additional adaptation capacity or the supervision recipe rather than the proposed branch separation.

Questions for authors

Please clarify the data protocol to see if teacher-relative test frames and the downstream augmented frames dinstinct sets.

An independent audit of annotation validity could be presented, and the baseline ROI SSIM/FID/LPIPS for rain, fog, and night should be provided that best distinguish teacher imitation from preservation of critical labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.