acceptodds
Under review as a conference paper at ICLR 2027

What Classifier Recovery Reveals about Cross-View Generalization

Abstract

A visual recognition system can fail after a change in camera viewpoint yet recover substantial accuracy by refitting its classifier on labelled target data while keeping the feature extractor frozen. Such recovery establishes accessible class information, but does not establish preserved representation geometry or a portable decision rule. We investigate this distinction through a diagnostic decomposition of local linear recoverability, class-relational organization, and cross-view decision compatibility across five task-trained convolutional architectures and frozen DINOv2 representations. Under cross-dataset transfer, convolutional classifiers retain 48.1–57.8% accuracy for a broadly comparable side-right view but fall to 6.9–12.1% for a frontal view; classifier-only frontal refitting recovers 47.1–59.4%, while DINOv2 rises from 11.8% to 94.1% under its reference protocol. Direct frontal–side-right geometry within 100-Driver shows a distinct structural regime: DINOv2 crop features retain substantially greater class-relational correspondence than the source-to-frontal cross-dataset comparison, with sensitivity to representation, input treatment, and covariance reference. Shared classifiers nevertheless remain view-asymmetric, and recovery decreases under driver-disjoint evaluation. The resulting diagnosis separates recoverable class information from class-relational change and from the functional compatibility of a fitted decision rule. These properties require separate measurements under explicit population contracts. This provides a diagnostic basis for evaluating single-source, closed-set cross-view methods beyond their target recovery score.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
cross-view generalization, classifier calibration, frozen representations, driver-disjoint evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a2X4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a framework to evaluate cross-view generalization in fine-grained driver-behavior recognition. By decoupling the evaluation into three distinct properties: local linear recoverability, class-relational organization (RDM), and cross-view decision compatibility, the authors reveal that successful classifier recovery (refitting the head on frozen features) merely establishes accessible class information, but does not guarantee preserved representation geometry or portable decision rules.

Critical strengths

The main strength of this work is its rigorous and exhaustive empirical validation across both general foundation models and task-specific models. This comprehensive analysis yields several key insights:

  1. The Illusion of Head-only Fine-tuning: High recovery accuracy (e.g., 94.1% for DINOv2) often masks severe overfitting to specific populations. Under strict driver-disjoint evaluation, performance drops precipitously, proving that the model merely "re-draws boundaries" for seen drivers rather than learning truly view-invariant features.

  2. Geometry Does Not Equal Decision Compatibility: Even when the class-relational geometry is highly preserved across views (e.g., high RDM similarity within the same dataset), a shared classifier still exhibits severe view-asymmetry. This insight fundamentally clarifies that preserving structural distances in the feature space does not equate to aligned decision boundaries.

Critical weaknesses

None

Questions for authors

Q1 (Algorithmic Improvements): since the paper demonstrates that frozen features neither preserve portable decision rules nor maintain class-relational geometry, what is the most promising direction for algorithmic improvement?

Q2 (Practical Evaluation Protocol): how to translate the proposed comprehensive diagnostic framework into a lightweight and standardized evaluation metric for routine use?

1 comment
AuthorsOpenReview ↗

Algorithmic directions and practical evaluation

We thank the reviewer for recognizing the rigor of the empirical comparisons and for raising two constructive questions. The revision treats the findings as an exploratory investigation of when class-discriminative information can be recovered locally but fails to support an unchanged decision at a different camera or driver population.

Q1. What algorithmic improvement follows? We do not claim that our experiments establish a new adaptation algorithm or identify the unique cause of failure. They identify where an algorithm would need additional evidence: T4-local seen-support recovery averages 73.46% while heads transported from the other cameras average 15.22% on that same receiver; source-fitted heads reach 27.55% there under a different fitting history. Combined with population-conditioned recovery, this suggests testing action-relevant versus driver/context-dependent cues and learning decisions robust to mounting-position changes. A discriminating next experiment would hold the receiver, support budget, driver breadth and preprocessing fixed while comparing pretrained and source-task-trained representations and targeted driver/context interventions. These are proposed tests, not completed improvements.

Q2. Can the diagnostic analysis become lightweight and standardized? The revised Discussion gives a practical minimum evaluation sequence, not an unvalidated universal score: (i) evaluate the unchanged source decision at the intended installation camera; (ii) with a declared small labeled budget, evaluate camera-local fitting and the unchanged fitted head/scaler on other receiving views; and (iii) repeat on drivers outside the fitting support, recording whether target statistics were used. Report the source, local and transported accuracies on named receiving cohorts, alongside labeled images per class (k), contributing drivers (m) and normalization access. Geometry and more expensive diagnostics can then be added when the functional results warrant explanation. We cannot yet establish a single predictive metric or optimal operating threshold.

The revised Results foreground the directional T4/SF3D observation, the local-versus-transport separation and the support-population results. Discussion 6.4-6.5 now explain evaluation usefulness and testable algorithmic directions. We appreciate the reviewer prompting this clearer distinction between empirical discovery, prospective mechanism studies and practical use.

Reviewer_6pgDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what classifier-only recovery reveals about cross-view generalization in visual recognition models. The authors argue that recovering recognition accuracy by retraining a linear classifier on frozen features does not necessarily imply that the underlying representation preserves its class-relational geometry or supports a transferable decision boundary.

To investigate this issue, the paper proposes a diagnostic framework comprising three complementary measurements: local linear recoverability, class-relational organization, and cross-view decision compatibility. Experiments are conducted on fine-grained driver-behavior recognition using the State Farm Distracted Driver Detection and 100-Driver datasets, involving five CNN architectures, DINOv2 representations, and additional DBMNet features.

The results show that substantial recognition accuracy can be recovered through classifier refitting even when direct cross-view transfer performs poorly. However, such recovery does not necessarily indicate preserved class-relational structure or transferable classification rules. The authors further observe that recovery performance is highly sensitive to the support-query population and that within-dataset cross-view representation similarity can differ considerably from cross-dataset comparisons.

Critical strengths
  1. The paper jointly examines local linear recoverability, class-relational geometry, and cross-view decision compatibility instead of relying exclusively on classification accuracy. These measurements capture different properties of a frozen representation, and their combination helps identify situations where classifier refitting succeeds without producing a decision rule that generalizes to another viewpoint. The separation between structural and functional measurements is a useful aspect of the experimental design.

  2. The comparison between image-disjoint and driver-disjoint evaluation is informative. For example, at 100 labeled samples per class, ResNet50 frontal recovery decreases from 94.3% under image-disjoint evaluation to 30.8% under driver-disjoint evaluation. This substantial difference demonstrates that reported classifier-recovery performance can depend strongly on the evaluation population. The authors also explicitly recognize the confounding effects of cross-dataset comparisons, which makes their interpretation more cautious than studies that directly attribute such differences to viewpoint changes.

Critical weaknesses
  1. The central claim of the paper is that classifier-only recovery does not imply preserved representation geometry or transferable decision boundaries. However, these distinctions are already well established conceptually. Linear probing measures whether discriminative information can be extracted from frozen features, while representational similarity measures structural relationships, and cross-domain evaluation measures the portability of learned decision rules. These measurements characterize different properties by construction, and agreement among them is not theoretically guaranteed.

The authors explicitly acknowledge that the paper does not introduce a new probing method or representational similarity estimator. Consequently, the main contribution is the combination of existing diagnostic tools and their empirical application to cross-view driver-behavior recognition.

The critical question is what fundamentally new understanding this combination provides beyond confirming that linear separability, geometric similarity, and classifier transferability are not equivalent. Although the reported observations are informative, the paper does not establish a new theoretical relationship among these properties or identify a previously unknown mechanism underlying cross-view generalization failure.

I would encourage the authors to clarify what genuinely new scientific insight is obtained from the proposed diagnostic decomposition, beyond the expected non-equivalence of three established representation properties.

  1. One of the most striking results is the substantial performance gap between image-disjoint and driver-disjoint evaluation. As shown in Figure 3, at 100 labeled samples per class, ResNet50 achieves 94.3% frontal accuracy under image-disjoint evaluation but only 30.8% when fitting and evaluation drivers are separated. DINOv2 exhibits a similarly large decrease from 92.3% to 51.4%.

These results raise a fundamental question about the interpretation of classifier recovery. High recovery accuracy under image-disjoint evaluation may reflect adaptation to subject-specific appearance, cabin configuration, or other contextual correlations rather than the recovery of action-discriminative information that generalizes across individuals.

The authors correctly describe this difference as population sensitivity rather than direct evidence of identity leakage. Nevertheless, the large performance degradation suggests that the distinction between subject-dependent adaptation and transferable action information is central to the paper's conclusions.

The current framework does not sufficiently characterize which information is being recovered by the classifier. A more informative investigation would examine whether recovery is primarily associated with action-related representations or subject- and context-dependent features. Furthermore, evaluating classifier recovery using fixed held-out drivers across different support budgets would help distinguish the effect of additional labeled examples from changes in the evaluation population.

Questions for authors

see the weakness

1 comment
AuthorsOpenReview ↗

New empirical knowledge and population-specific recovery

We thank the reviewer for identifying the essential novelty and interpretation question. We agree that linear probing, representation similarity and measuring transfer are established methods, and their non-equivalence by construction is not itself a new scientific result. We have revised the paper to distinguish these tools from the empirical relationships discovered when independent collection, camera view, receiving population and input information change together.

The expanded four-camera study exposes a directional exception that was not conveyed by a broad claim of diagnostic decomposition: source-fitted linear heads average 27.55% on T4 versus 12.28-14.57% on the other target views. In a separately fitted and separately received return experiment, T4-fitted heads achieve 50.10-52.88% on the common SF3D validation receiver, whereas T1-T3-fitted heads return at approximately 12-16%. Yet T4-fitted decisions transport poorly to the other target cameras. These results suggest receiver-dependent compatibility that is not summarized by collection membership or a local recovery score. Because the forward and return populations differ, we do not call the two directions a symmetric effect or attribute the pattern to pose alone.

The reviewer's concern about subject- and context-related recovery is also central. We retain the original image-/driver-disjoint measurements and add fixed-Q12 four-draw support comparisons. At k=20, the seen-minus-unseen advantage averages +26.13 percentage points locally but +1.67 off-camera. Within local evaluation, the advantage is +38.17 points for support-overlapping query drivers versus -1.85 for nonoverlapping drivers. These findings make the dependence on the receiving population more concrete, without asserting that driver identity alone caused it; adjacent frames, sessions, vehicles and class evidence can contribute.

We thus claim bounded empirical knowledge, not a new mathematical relationship, new fitting method, or identification of an invariant representation mechanism. Results 5.1-5.4 and Discussion 6.1-6.5 now separate observed contrasts, possible explanations, practical evaluation consequences and proposed discriminating tests. The prior-work and appendix comparisons explicitly distinguish DFR/DARE, probing, representation geometry and transferability assessment by their questions and information contracts. We appreciate the request for clearer science beyond a toolbox and have reorganized the manuscript accordingly.

Reviewer_kNCeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work aims to investigates classification task recovery under viewpoint change and proposes several diagnostic tools. Diagnostic tools are applied and analysed in a setting based on the 100-Driver dataset across a set of vision backbones.

Critical strengths

The paper addresses an interesting and relevant topic in the field of domain generalisation that is specifically relevant for real-world applications.

Critical weaknesses

There are several critical weaknesses:

  • Limited datasets: The paper only assess a single pairing of datasets. Extending the experiments (Drive&Act, DMD, SynDD2, DAD) would strengthen the sometimes very broad claims. Also more cross source <-> target pairings would be desired.
  • While the paper claims to specifically analyse cross viewpoints generalisation, in the experiments also vehicle, drivers, sensor and label set all change together, so effects can't be attributed to viewpoint. The label mapping itself also shapes both results.
  • Limited models: no vision-language or fine-tuned ViT models, and no video models even though driver actions unfold over time.
  • Significance: the paper offers diagnostics only, with no method and no evidence that transfer prediction is better than existing transferability scores (LEEP and LogME are cited).
Questions for authors

What was the rational for limiting the analysis to the chosen datasets / model? What does the three-part decomposition reveal beyond Ponbagavathi et al. (https://arxiv.org/abs/2407.15605 and https://arxiv.org/abs/2508.16207) and DFR/DARE? Do the diagnostics predict accuracy on unseen views or unseen drivers better than LEEP or LogME?

1 comment
AuthorsOpenReview ↗

Data scope, confounding, prior art, and practical significance

We thank the reviewer for recognizing the real-world relevance of the question and for identifying important limitations of dataset scope and scientific attribution. We agree that source/target collection, camera viewpoint, drivers, vehicles and labels change jointly in the present SF3D-100-Driver pairing. Our revision does not claim a causal camera-position effect or independent replication over four datasets: four cameras belong to one target collection. It defines the shared image-level RGB ten-activity question, preserves native labels and visually checked action mapping, documents camera observation conditions, and separates source-to-target, within-target and target-to-source evaluation contracts. The eligibility and acquisition appendix states what further independent, matched or calibrated observations would be required to disentangle these factors.

The expanded empirical analysis provides results beyond one pair of cross-dataset accuracies. On fixed target development receivers, source-fitted heads show a marked T4 preference (27.55% versus 12.28-14.57% on T1-T3); in a distinct return evaluation T4-fitted heads obtain 50.10-52.88% on SF3D, whereas other target heads return at roughly 12-16%. Local recovery is much higher than unchanged cross-camera transport, and its support-population advantage is concentrated in same-camera/overlapping-driver queries (+26.13 versus +1.67 points overall local/off-camera; +38.17 versus -1.85 points in the two local driver strata). We interpret these as conditional associations, not pose or identity mechanisms.

We have strengthened comparisons with DFR/DARE and the cited Ponbagavathi fine-grained unseen-view/temporal work. Those studies investigate fitting, adaptation or video-related questions under different supervision and observation units. Our static image task remains relevant for camera-specific single-observation recognition and inspecting which fitted rules remain usable across conditions; temporal event and video/VLM recognition are clearly identified as next-step studies rather than experiments claimed here.

Regarding LEEP/LogME, we do not claim superiority in transfer-accuracy prediction: no matched predictive benchmark against those methods was completed. The present contribution is instead the observed directional and population-dependent behavior and practical evaluation guidance specifying source, local and transported accuracy on named receiving cohorts with declared fitting/normalization access. Discussion 6.4-6.5 now explicitly connects this to single-camera DMS evaluation and prospective independent-collection, driver-diverse, event-linked research. Thank you for motivating this tighter scope and contribution statement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript: empirical findings, scope, and reviewer concerns

We thank the reviewers and Area Chairs for their careful evaluation and constructive questions. We have revised the manuscript and extended the evidence to four 100-Driver camera observations, while keeping the independent SF3D-to-100-Driver pairing and its limits explicit. Substantive changes in the revised PDF are marked in blue.

Our scientific question is not whether established probes are mathematically equivalent or whether last-layer fitting is a new algorithm. We investigate what recognition and classifier reuse actually reveal when collection, camera observation, receiving drivers, and permitted target information change together. The strongest new empirical patterns are directional source/target compatibility, the difference between camera-local recovery and unchanged-rule transport, population-conditioned recovery, and sensitivity to RGB preprocessing.

ConcernRevised evidence and responseWhere to look
Scientific insight beyond combining existing methodsSource-fitted accuracy on T4 is 27.55% versus 12.28-14.57% on T1-T3; in a separate return experiment, T4-fitted heads reach 50.10-52.88% on SF3D, far above heads fitted at T1-T3. Yet T4 heads transfer poorly to the other target cameras. These are distinct fitting/receiving contracts, not a symmetric transfer coefficient.Results 5.1-5.2; Discussion 6.1-6.2; directed evidence appendix
Driver-population dependenceOn fixed Q12 development receivers, seen-minus-unseen support differences average +26.13 percentage points for local fits versus +1.67 for off-camera transport. Within local evaluation the corresponding driver-overlap and nonoverlap contrasts are +38.17 and -1.85 points. These are descriptive, not randomized identity effects.Results 5.3; Discussion 6.2
Camera versus dataset confoundingWe distinguish source-to-target, within-target camera, and target-to-source questions and disclose the native/canonical action mapping, observation conditions, and fitting/query populations. One independent source/target pair does not identify camera pose as a cause.Experimental Design; eligibility and data appendices
Practical value and predictionWe provide a lightweight evaluation sequence (unchanged source rule, camera-local fit, unchanged transported rule, unseen-driver receiver, declared normalization). We do not claim a new scalar transfer predictor or superiority over LEEP/LogME without that experiment.Discussion 6.4; Related Work
Model, temporal, and future-data scopeStatic image recognition isolates observation-level recoverability and decision reuse. Prospective multi-collection, matched-camera, driver-diverse and temporal/event-aligned extensions are now stated, not presented as completed results.Discussion 6.5; acquisition appendix

We have expanded the comparisons and model-level evidence, restored and contextualized the relevant prior literature (including DFR/DARE, probing, cross-view recognition and transferability assessment), and kept technical provenance and unmeasured comparisons visible in the appendix. We appreciate the opportunity to clarify what is established, which explanations remain open, and how these observations can guide subsequent research and practical DMS evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.