acceptodds
Withdrawn from ICLR 2027

The Domain Is a Residue: Adapting Self-Supervised Features, Not Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Clearing fog, rain or snow from footage, or turning renders into photographs, must remove the source domain and keep the scene. Unpaired translators carry it through because their generator sees the source appearance (pixels, a near-invertible latent or a control map) and keeps it. A DINO feature map fixes what is in the scene and carries weather, lighting and rendering style as a residue of 13 to 14% of the feature norm. We propose the Representation Feature Adapter (RFA), a 2.9 M-parameter network that moves this residue. We train only the adapter and its discriminators; the encoder and a feature-conditioned decoder, trained once for all conditions, stay frozen. Against CycleGAN-Turbo it is ahead on both metrics on fog and on KID on night, and level within noise on snow, rain and haze. On sim-to-real it leads REGEN and HyPER-GAN on both metrics. Only the RFA removes the rain while keeping the scene. The removal costs scene structure: CycleGAN-Turbo keeps more on every condition but fog. On VAE latents the identical adapter collapses to the identity, and decoders from other groups that never saw it render its output. The RFA has about 160 times fewer trainable parameters than CycleGAN-Turbo and under a fifth of its per-condition training time.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unpaired image-to-image translation, domain adaptation, self-supervised features, DINO, feature-space adapter, frozen generators, one-step diffusion, ControlNet, adverse-weather restoration, sim-to-real, decoder portability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PCwGOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Representation Feature Adapter (RFA), a 2.9M-parameter network for unpaired image translation inside a frozen DINOv2 feature space. A frozen DINOv2 encoder maps the input to a feature map, the RFA maps it to a target-domain feature map, and a frozen one-step decoder (a DINO ControlNet on Sana-Sprint) renders the result. The adapter is trained with CycleGAN-style cycle and identity losses on features and with adversarial losses on the rendered image, whose gradients pass through the frozen decoder. The authors argue that the domain is a small residue of the feature map (13-14% of its norm) and show that the same adapter collapses to the identity on VAE latents. On adverse-weather removal, the RFA is ahead of CycleGAN-Turbo on fog and on night KID and level on snow, rain, and haze. On sim-to-real, it improves KID and FID over REGEN and HyPER-GAN, at a substantial cost in scene structure.

Critical strengths

S1. Clear diagnostic experiment. The comparison of the identical adapter on DINOv2 features and on two VAE latents (Table 1) is a clean and informative experiment, and it supports the claim that the choice of conditioning space determines what the adapter can change.

S2. Honest reporting of costs. The paper reports results that work against it, including halved detection mAP (App. G), lower structure retention, jackknife standard errors (App. H), and a decoder-only control (App. E).

S3. Efficiency and portability. A new domain pair requires training only a small adapter, which reduces per-condition training time to under a fifth of CycleGAN-Turbo's. Adapted features can also be rendered by decoders that never saw the adapter (PiD, RAE).

Critical weaknesses

W1. Limited technical contribution. Domain adaptation in feature space has been studied before (e.g., CyCADA), and decoders conditioned on frozen DINO features already exist (Control-DINO, RAE). The method mainly combines these with standard CycleGAN objectives and vision-aided discriminators. The "residue" is also measured as the adapter's own displacement, ||G(f) - f|| / ||f||, so it describes what the trained adapter does rather than an intrinsic property of the domain.

W2. Missing comparisons with modern editing and restoration models. The baselines are unpaired translators (CycleGAN, CUT, CycleGAN-Turbo) and Cosmos-Transfer. Instruction-based image editing models (e.g., FLUX.1 Kontext, Qwen-Image-Edit) can remove fog, rain, or snow and turn night into day zero-shot, and are known for strong structure preservation. Adverse-weather restoration models are also excluded on the grounds that their usual protocol is paired, although they could be run on the same crops. Without these comparisons, it is unclear whether the proposed approach has a practical advantage over currently available tools. I would be willing to raise my rating if the authors include recent image editing models as baselines and show that the RFA offers a clear advantage over them on the same crops, in particular on structure preservation and downstream detection.

W3. The structure loss undermines the target applications. On BDD100K snow and rain, a fixed detector's mAP on RFA outputs drops to about half of that on the untouched frames (0.148 vs. 0.276 and 0.146 vs. 0.254), whereas CycleGAN-Turbo keeps it (0.275 and 0.224; Table 12). On sim-to-real, only about a sixth of the detectable objects survive, compared with two thirds to three quarters for REGEN and HyPER-GAN. For driving applications, where restored frames are typically consumed by perception models, an output that is worse than doing nothing is a fundamental problem. The method appears to regenerate the scene rather than remove the weather from it, and the improved KID may partly reflect this.

W4. Evaluation protocol. KID and FID are computed on 50 crops per condition (25 for haze). Only fog and sim-to-real are separated by more than two standard errors (App. H), and FID on 50 images is rank-deficient, as the authors note. The "is the weather gone" classifier uses the same DINOv2 features that the adapter moves (App. F), so it is not an independent measure. In addition, the decoder is trained on Mapillary, which is also the sim-to-real target domain. Table 3 does not include a decoder-only row, so part of the sim-to-real KID gain may come from the decoder rather than the adapter.

Questions for authors

Q1. How does the RFA compare with instruction-based editing models (e.g., FLUX.1 Kontext, Qwen-Image-Edit) and adverse-weather restoration models on the same crops, in terms of KID, structure distance, and detection mAP?

Q2. What are the KID, FID, and structure distance of the decoder-only control on sim-to-real (Table 3)?

Q3. Can the authors evaluate weather removal with a classifier or metric that does not read DINOv2 features?

Q4. In which practical setting is the trade-off of the RFA preferable to that of CycleGAN-Turbo, given that detection mAP falls below that of the untouched input?

Reviewer_uBRrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Representation Feature Adapters (RFA), 2.9 M-parameter residual networks that move frozen DINOv2 feature maps to a target domain, rendered by a frozen diffusion decoder. Training is CycleGAN-style: feature-space cycle and identity losses plus image-space adversarial gradients through the decoder. Experiments cover ACDC fog/snow/rain, HIVIS haze, BDD100K night-to-day and PreSIL-to-Mapillary sim-to-real, with decoder-only controls, a same-backbone port, paired errors and a fixed-detector evaluation. The paper argues that adaptation is better done by moving a small "residue" in a frozen self-supervised feature space than by adapting a generator, and reports a scene-structure cost.

Critical strengths
  1. The design is straightforward: train an adapter between a frozen encoder and decoder. Decoder-only, same-backbone, and cross-decoder controls make the results easier to interpret.

  2. The paper documents how pixel-fed generators can retain source appearance and reports RFA's scene-structure cost openly. These negative results help explain the limits of both approaches.

  3. The 2.9M-parameter adapter and shared decoder reduce training cost across domain pairs, making the design practically appealing if scene preservation improves.

Critical weaknesses
  1. The detector results conflict with the claim of preserving the scene. In Table 12, snow and rain mAP fall to 54% and 58% of the unmodified-input scores, versus 100% and 88% for CycleGAN-Turbo. On sim-to-real, RFA reaches 0.153 mAP using detector-generated source boxes, versus 0.661–0.754 for REGEN and HyPER-GAN. This substantial preservation cost weakens the claimed advantage over generator adaptation.

  2. The residue measure does not independently identify a domain component. It is the adapter's relative feature displacement, measured under a chosen normalization and training objective. A small displacement shows how much this adapter changes its input; it does not establish that the changed component is the domain itself.

  3. The comparisons do not establish an advantage at matched scene preservation. Appendix H reports jackknife errors and finds snow, rain, and haze largely tied; variation across training seeds is not measured. Even the same-backbone port does not show that RFA improves distribution metrics when structure distance or detector mAP is held comparable.

  4. The detail loss limits downstream use. Small objects lose shape and text becomes unreadable (Section 8), which can impair detection and applications that depend on image overlays. The paper acknowledges this cost but leaves the proposed structure-preservation remedies untested.

  5. Decoder and normalization choices remain alternative explanations for the proposed mechanism. The latent-versus-feature and decoder-only controls are useful, but Appendix D.4 shows that changing the decoder shifts the trade-off between distribution and structure. The experiments do not establish how much of the advantage comes from feature-space adaptation itself.

The paper claims to remove the source domain while preserving the scene, yet its detector results show substantial content loss and underperform CycleGAN-Turbo. The remaining comparisons rely on limited single-run estimates, with improvements deferred to future work. Moreover, the reported output displacement depends on specific normalization, loss, and decoder choices, rather than identifying an intrinsic domain component. Thus, while the adapter and controls are valuable, the evidence does not support the paper’s two central claims.

Questions for authors
  1. At comparable structure distance or fixed-detector mAP, does RFA still improve weather removal or KID/FID over the generator baselines?

  2. Can the residue be measured independently of a trained adapter, or shown to track domain changes consistently across adapters and decoders?

  3. Beyond Appendix H's crop-level jackknife errors, how stable are the main results across training seeds? Have you tested a remedy for the snow, rain, and sim-to-real preservation losses within the current recipe?

Reviewer_wmwwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Representation Feature Adapter (RFA) for unpaired image-to-image translation. RFA learns a residual mapping within a frozen self-supervised feature space, using image-space adversarial supervision propagated through a frozen feature-conditioned decoder. The central idea is to adapt domain-dependent feature components while reusing the encoder and decoder across domain pairs.

Critical strengths
  1. Supervising the adapter in tasks such as image restoration or image transfer is novel, and experiments have demonstrated its effectiveness.
  2. Compared to similar methods, the training cost is cheap (trainable parameters and training time).
Critical weaknesses
  1. RFA yields good results with DINO, but performs poorly with VAE; while the authors offer an explanation (DINO features are lossy and semantic/ VAE near-invertible), they do not provide an in-depth analysis of this phenomenon.
  2. This paper employs a frozen DINO encoder, which leads to information loss. Although experiments demonstrate that a superior decoder can yield better empirical results, the issue of information loss inherent to the encoder remains unresolved.
Questions for authors
  1. The paper evaluates several DINOv2/v3 configurations, but does not report results with non-DINO representation encoders. Have the authors explored alternatives such as CLIP?
  2. Could the authors add a decoder-only baseline to Table 12 to quantify the additional degradation introduced by RFA, and clarify what evidence supports attributing the remaining loss to irrecoverable information loss in the encoder?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.