acceptodds
Withdrawn from ICLR 2027

Source Gains, Target Losses: Auditing Geometry-Distilled Visual Features

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Geometric supervision promises visual representations that better capture scene structure, yet gains on source geometry tasks can hide a loss of boundary performance on other datasets. This creates a selection problem: an adapter can score better on the task it learns while becoming less useful for the task it is meant to support. We study this mismatch by adapting DINOv2 with depth and multi-view supervision from ScanNet++ and fitting new linear boundary probes on three labeled target datasets. With adapter capacity, training objective and budget held fixed, adaptation depth separates the outcomes: early-layer updates transfer better than late-layer updates in all three training runs, under five evaluation metrics and two dataset weightings. Source boundary F1 nevertheless favors a later layer group in every run, and the chosen adapters score below the unadapted backbone. We then test whether geometric learning can keep more of the backbone's boundary performance by penalizing changes to pretrained patch features. With a penalty weight chosen using target results, the mean target F1 loss falls from 0.015–0.020 in three unconstrained training runs to 0.0010 on average over three constrained runs, while source coplanarity gains remain. A preregistered test on COCO, a target set the project had never used, repeats the layer result: early windows beat both late windows and the unadapted backbone in all three runs, and source boundary F1 again picks a worse window. Adapting a backbone well therefore depends on where it learns geometry and on which pretrained features it keeps for transfer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
visual representation learning, geometric supervision, transfer learning, parameter-efficient fine-tuning, boundary detection, model selection, feature preservation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wastOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Please see "Critical Weaknesses".

Critical strengths

None

Critical weaknesses

The sentences are often awkwardly phrased, and the ideas are not presented coherently. The abstract and introduction do not follow the conventional structure expected in an academic paper. The paper does not include a single real-world image, for example, from ScanNet++, despite this being the dataset used in the study. Moreover, the paper is 43 pages long, yet its main contributions remain unclear. Throughout my academic career, I have never encountered a paper of this nature.

Questions for authors

None

Reviewer_7ewMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper checks if geometry distillation that helps source tasks also makes features for downstream boundary prediction. It uses DINOv2 with four objectives looks at equal size layer windows and trains new target probes. In trials adapting early layers transfers more reliably than adapting late layers while source validation scores often pick worse target representations. A penalty that matches adapted features to the backbone gets back much of the lost target performance while keeping the source improvements. Experiments use three target datasets and a preregistered COCO evaluation with conclusions about patch-level boundaries and mostly linear probes.

Critical strengths
  1. The controlled layer comparison supports a model-selection finding: Equal-capacity windows, three full adapter/probe/selection repeats and agreement across metrics and dataset weightings make the source-to-target ranking mismatch convincing. Boundary-based source selection picks blocks 12–17 in all three runs; the new COCO evaluation shows that this choice misses early windows (Table 1; Appendix L).

  2. The preservation intervention gives evidence beyond just correlation: Matched runs get 0.0174–0.0197 in held-out target F1 with paired scene-bootstrap intervals above zero while keeping coplanarity gains. The paper also reports target losses separates target-informed tuning from source-only selection and says that the COCO’s preregistered recovery criterion is not met in every run (Appendices L–M).

Critical weaknesses
  1. The contribution needs positioning against similar preservation results: FiT3D already combines geometry-adapted features to keep generalization and says that longer fine-tuning can hurt performance. The paper does not talk enough about this related finding. Layer-selective adaptation and regularization toward pretrained feature maps also have examples in Surgical Fine-Tuning and DELTA. So the main contribution is the controlled boundary-transfer and source-selection audit; the preservation principle itself is not very new.

  2. The comparison with reducing adaptation strength is not as clear as the recovery result: Each reduced learning rate has one adapter run and it is not clear if preservation is better than the best reduced rate under dataset-balanced held-out F1 (Appendix M). Source-gain intervals also resample frames from nine scenes. These results show that preservation is useful. Do not prove it is consistently better than weaker updates.

Questions for authors
  1. What benefit does training-time preservation offer beyond FiT3D-style concatenation of adapted features? Please explain the difference in ideas and the tradeoff in processing cost. One example comparison using features considering the extra encoder and probe dimensionality would help make this point stronger.

  2. Could you do the reduced-learning-rate setting again for adapter seeds 43 and 44 and report paired held-out target F1 along with source uncertainty resampled at the scene level? This would directly check if the comparative benefit still works with training and source-scene variation.

Reviewer_EfTuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how geometry-aware adaptation of pretrained visual representations affects their transferability to downstream boundary prediction. Specifically, the authors adapt DINOv2 using four geometric supervision signals from ScanNet++ and evaluate the resulting representations through linear boundary probes on several target datasets. The paper examines the effects of adaptation depth, the reliability of source-task scores for model selection, and a feature-preservation regularizer intended to mitigate negative transfer.

The main findings are that early-layer adaptation generally transfers better than late-layer adaptation, source-task boundary and coplanarity scores do not reliably identify the best-performing adapters on the target datasets, and feature-preservation regularization can substantially reduce the observed transfer degradation.

Critical strengths

S1. Detailed empirical analysis of layer-wise adaptation. The paper provides a relatively thorough investigation of how adapting different transformer layers affects downstream boundary probing. By comparing four non-overlapping adaptation windows with matched LoRA capacity and training configurations, the authors offer useful empirical observations about the relationship between adaptation depth and representation transferability.

S2. Extensive experiments with relatively consistent findings. The experimental evaluation is comprehensive, covering multiple target datasets, three independent adapter training runs, several evaluation metrics, and different dataset-weighting strategies. The early-versus-late adaptation trend is generally consistent across these settings, which strengthens the credibility of the empirical observations.

Critical weaknesses

1. Limited novelty and scientific contributions.

The main finding that source-task improvements do not necessarily translate into target-task gains seems relatively unsurprising, especially given the differences in task definitions and data distributions. The source-based selection analysis mainly demonstrates the failure of two simple heuristics, without sufficiently investigating stronger alternatives. Moreover, the layer-wise LoRA adaptation and feature-preservation regularization are largely based on existing techniques, while the contributions of individual geometry objectives remain insufficiently characterized. Overall, despite extensive experiments, I find the new scientific insights rather limited, which is my primary concern regarding acceptance.

2. Poor clarity and organization of the experimental presentation.

The paper is difficult to follow. The historical candidate pool, different training and evaluation protocols, and extensive implementation details are presented in a confusing manner. The figures also fail to clearly communicate the main findings. This makes it difficult to distinguish the central experimental evidence from supplementary analyses and to understand precisely what each experiment establishes. Given that the paper's main contribution is empirical, the lack of a clear experimental narrative substantially weakens its presentation and overall impact.

Questions for authors
  1. Layer placement versus adaptation strength. Have the authors compared different adaptation windows at matched source-task performance or matched final-layer feature drift? Could the advantage of early-layer adaptation be primarily explained by weaker perturbations of the pretrained representation?

  2. Source-only model selection. Why are source boundary and coplanarity F1 selected as the two primary model-selection criteria, rather than a composite score across the four geometric supervision tasks? Have stronger source-only selection baselines been considered?

  3. Practical downstream transfer. Do the observed improvements persist when using more expressive, fully trained downstream prediction heads rather than patch-level linear probes? How much practical benefit does early-layer geometry adaptation provide over the original pretrained backbone?

  4. Geometric supervision contributions. What are the individual contributions of the four geometric supervision objectives? In particular, how should the limited improvements in ordinal depth and visibility be interpreted?

  5. Source probe evaluation protocol. Could the authors clarify which images are used to fit the newly trained source probes and whether the probe-training and source-scoring sets are strictly disjoint?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.