acceptodds
Under review as a conference paper at ICLR 2027

Where Models Diverge: Linear Predictability Trace the Direction of Representational Change

Abstract

We study what directional differences in cross-model predictability can reveal about how two models organize their representations on the same data. To do this, we fit linear maps between a pair of models' representations in both directions and use their predictive asymmetry – the difference in held-out linear predictability between models – as a diagnostic tool. This signal reveals which distinctions one model preserves relative to another, localizes which examples drive these differences, and can identify where relative performance gains between two models lie, without requiring any labels, concept/dictionary extraction, or clustering. Validating this diagnostic through controlled experiments, we show that it can recover differences in supervision granularity, training data coverage, and training progression between model pairs. Changing only the supervision objective can even reverse the direction of linear recoverability between a fine-tuned model and its parent. We apply predictive asymmetry to twelve domain-specialized foundation models and trace how their representations differ from their parent models after fine-tuning. Our experiments demonstrate that predictive asymmetry gives a label-free account of specialization: it tracks relative training exposure between specialist and parent and localizes where specialist–parent probe advantages hold. Predictive asymmetry therefore provides a simple, label-free view of where and in which direction model representations diverge.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, representation alignment, specialization, domain foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iAmJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a diagnostic metric, predictive asymmetry, that measures differences in linear recoverability between two models' representations using bidirectional affine mappings. By examining the difference in reconstruction quality in both directions, the method aims to identify which model preserves relatively less recoverable information and localize these representational differences.

Critical strengths
  1. The proposed method is conceptually simple, requiring only an evaluation dataset and the fitting of bidirectional affine mappings between the models' representations. Since the reconstruction targets are the models' own features, the approach does not require annotated data. This enables a label-free analysis of directional linear recoverability, allowing the identification of which model preserves representational distinctions that are less linearly accessible from the other.
Critical weaknesses
  1. Difficult to follow experimental narrative. A substantial portion of the paper is devoted to the experimental design and results. However, the organization and presentation of these sections make the paper unnecessarily difficult to follow. Section 4 provides approximately two pages of dense experimental descriptions, including detailed configurations and discussions of expected outcomes across several distinct experiments. The corresponding results are only presented afterward, requiring the reader to retain substantial amounts of information to understand the subsequent discussions.
    Furthermore, the results themselves are not always clearly connected to their supporting evidence. For instance, in lines 322–323, the authors present an empirical result and its interpretation without providing a direct reference to the corresponding figure or table. This makes it difficult to follow the argument on a first reading, often requiring the reader to navigate back and forth between the experimental setup and the results to reconstruct their relationship.
    Given the density of the experimental descriptions and the extensive discussion of expected outcomes, I frequently found it necessary to consult the experimental design while reading the corresponding results. This substantially disrupts the reading flow and makes the empirical findings more difficult to interpret than necessary.
    I strongly encourage the authors to reconsider the organization of Sections 4–6 to improve readability and establish a more coherent experimental narrative. One possible approach would be to organize the experiments into separate individual sections, each presenting the experimental motivation, setup, expected outcomes, and corresponding results together. This would allow readers to follow the reasoning behind each experiment and directly connect its findings to the underlying hypotheses. Additionally, I strongly recommend providing explicit cross-references to the relevant figures and tables when prior to introducing and discussing empirical findings, rather than requiring readers to identify the supporting evidence themselves.
  2. Limited demonstration of practical utility. While the authors provide evidence of the proposed method's diagnostic capabilities, the paper does not convincingly establish the actionable utility of the resulting insights. In particular, it remains unclear how predictive asymmetry could be used in practice to inform model development, evaluation, or adaptation beyond retrospective representation analysis. Although the method successfully identifies representational differences in controlled experiments, the paper does not demonstrate how these findings could translate into concrete practical benefits.
    Furthermore, while the authors compare predictive asymmetry against existing methods, including information imbalance [1], the practical advantages and limitations of predictive asymmetry relative to these baselines remain insufficiently discussed. I would encourage the authors to elaborate on the tradeoffs between the proposed method and existing approaches, and to clarify to what extent predictive asymmetry addresses an unmet need in the current landscape of representation comparison methods.

[1] Acevedo et al., "A quantitative analysis of semantic information in deep representations of text and images", TMLR, 2026.

Questions for authors
  1. What are the practical advantages of predictive asymmetry compared to existing methods, such as information imbalance [1], beyond the quantitative improvements reported in Table 1? In which scenarios would the authors recommend using predictive asymmetry rather than the alternatives?

  2. Can the authors provide a concrete use case in which predictive asymmetry could produce actionable insights? For instance, how could the insights obtained through predictive asymmetry inform decisions regarding model development, data selection, or fine-tuning?

  3. What are the tradeoffs associated with predictive asymmetry compared to the evaluated baselines? Are there particular settings or conditions under which the authors would recommend using existing methods instead of predictive asymmetry?

4 comments

Thank you for your detailed and constructive review. We appreciate that you found predictive asymmetry conceptually simple and label-free with useful directional signals. We address the experimental narrative, actionable utility, and tradeoffs below, and are happy to answer any remaining questions

1. Presentation and Experimental Narrative

Thank you for this feedback, which helped us substantially improve the paper's readability. Following your suggestion, we restructured the experimental sections so that each experiment's setup and results appear together.

  • Each experiment now presents its motivation, setup, expected outcome, and results together. In Sec. 4.1 (ImageNet controls), each control (supervision granularity, training-set coverage, training progression) presents its setup immediately followed by its result, followed by the baseline comparison and cross-architecture pairs in the same format. Sec. 4.2 (CLIP fine-tuning) and Sec. 5 (released specialists) each present a single experiment family as setup followed directly by results. All changes are highlighted in blue, and the Global Response ("Restructured Presentation") maps original to revised sections.
  • Every finding is introduced with an explicit reference. For example, the fine–coarse result at original L322–323 now points to App. B.1, which reports it across seeds and plots it per image (Fig. 6).

2. Practical Utility

We agree that demonstrating practical value is important. We added a new "Applications" section to the main paper (Sec. 6) describing potential uses, and validated one of them experimentally (App. B.3).

Because predictive asymmetry shows, without labels, which model holds distinctions the other lacks and on which inputs, it can guide decisions such as auditing fine-tuning, selecting unlabeled data, and targeting knowledge transfer. We tested this in label-free distillation, where a fine-tuned model is trained to match its parent's representations in order to recover distinctions it lost during fine-tuning:

  • Whether to distill. Distillation only helps if the parent holds structure the fine-tuned model lacks, and otherwise it can erase useful specialization. We therefore distill only models that asymmetry marks as parent-favored. Across our controlled CLIP fine-tuning models, this rule makes the right choice for all nine models: it flags the coarse-label models, where distillation recovers +4.6 points of species accuracy, and skips the others, where distillation would lower it by 2.2 points.
  • Which images to use. We also test whether a released domain specialist, BioTrove, benefits from selective distillation from its parent using only a small set of unlabeled images. Selecting the 20,000 most parent-favored images with predictive asymmetry (out of a 182,000-image pool) improves BioTrove's Rare Species accuracy by +1.14 points. This is roughly twice the gain from selecting images randomly, by coreset, or by Information Imbalance, and larger than the +0.45 gain from distilling on all 182,000 images.

Full results and additional details are in the Global Response ("Practical Applications"), in addition to the paper sections.

3. Advantages and Trade-offs Relative to Existing Methods

Thank you for these questions. We added a dedicated discussion of advantages, trade-offs, and when to use each method in App. B.6, referenced from the baseline comparison in Sec. 4.1. We also answer your specific questions below:

Unmet need. Symmetric metrics (symmetric predictivity, mutual-kNN, PNKA) detect where two models differ, but cannot say which model the difference comes from. Local dimensionality measures (e.g., local PR) score each model separately, so comparing them points to a model only if one assumes how training changes dimensionality. That assumption is not reliable: local PR correctly identified the later checkpoint in all 75 pairs, but fails for supervision and coverage changes (attribution .059 and .342; Table 1). Concept- and group-based methods (RSVC, RDX) require extracting concepts or clustering inputs first, and their results depend on that step. Predictive asymmetry identifies which model has organization the other cannot predict, for each image and any subset, directly from the embeddings, without labels or concept extraction.

Advantages over Information Imbalance. Both methods are directional and label-free. Information Imbalance asks whether each image's nearest neighbor is preserved across models, whereas predictive asymmetry measures how well each image's full embedding is predicted from the other model. This difference has practical consequences:

  • More reproducible localization: Class-level scores from two halves of the same data agreed closely for predictive asymmetry (.97) but not for Information Imbalance (.25). As Information Imbalance relies on neighborhood structure, it is very sensitive to data splits, even within the same distribution.

  • Better tracking of real specialist gains. On released specialists, predictive asymmetry is the only method whose class-level scores track per-class F1 gain in all eight core pairs (mean ρ = .54, versus .25 and 5/8 pairs for Information Imbalance; App. B.2.2, Table 12).

  • Robustness across architectures. Moving from same-architecture to cross-architecture pairs, predictive asymmetry's detection AUROC barely changes (1.000 to .995), while Information Imbalance's drops from .946 to .879 (Table 7).

  • More useful data selection. In the distillation experiment above, selecting by asymmetry gives a larger gain than Information Imbalance (+1.14 vs. +0.56 points), because it targets the images where the parent's structure is missing: 65% of its selections come from taxonomic classes BioTrove never saw, versus 17% for Information Imbalance (Table 15).

When to use alternatives. Symmetric metrics are the simpler choice when the question is only whether two models differ; local PR is cheap and accurate for ordering checkpoints of a single model; Information Imbalance needs no fitted map and may be preferable with very few images; and concept-based methods (RSVC, RDX) are needed to explain what differs semantically. These are complementary to asymmetry, which can first locate where and in which direction models diverge.

Limitations of predictive asymmetry. Predictive asymmetry captures only linearly accessible structure and needs enough images to fit a map in each direction. Its cost grows linearly with the number of images (versus quadratically for exact Information Imbalance; the two take similar time at 20,000 images, and asymmetry is about twice as fast at 40,000), but faster with embedding width. When two representations share little linear structure, so that predictivity is low in both directions, their asymmetry should be interpreted with caution. We also state these limitations in Sec. 7 and Appendix B.6.

AuthorsOpenReview ↗

A Kind Reminder

Dear Reviewer iAmJ,

Thank you again for your helpful review! Following your feedback, we have significantly improved the ordering/presentation of our paper to make the methods and experiments easier to follow by re-ordering Sections 4-6 as you suggested. We have also added more discussions and demonstrations of practical utility (in the rebuttal as well as in the paper).

As we approach the end of the discussion period, please let us know if you have further questions, and if the concerns are resolved, we would be grateful if you would consider raising your score. Thank you again for your time and effort!

Dear Reviewer iAmJ,

Thank you again for your time and insights! As we approach the last day of the discussion period, we would really appreciate it if you could let us know if the rebuttal has sufficiently addressed your concerns and if you would consider raising our score.

Reviewer_XQYBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to investigate how predictive asymmetry, the difference in the predictive power of linear maps between two models’ representations in both directions, can serve as a diagnostic tool to how two models differ. In particular, the authors claim that this signal reveals the directions that the models diverge, helps locate observations that cause the divergence, and predict which direction helps with downstream performance. Empirically, they apply predictive asymmetry in two main sets of controlled experiments: ImageNet with controlled supervision granularity, training data coverage, and training time; CLIP fine-tuning that covers multiple domain specialists. Results confirm that predictive asymmetry provides label-free signal of representation differences.

Critical strengths

The paper revisits the topic of representational similarity from a new perspective and has comprehensive experiments justifying the claims. The paper is easy to flow, the idea is simple, and experiments are well explained in detail, in which they artificially create controlled settings to ensure effective ablations.

Critical weaknesses

Incremental Novelty: The simple idea can be a novelty concern to me at the same time, as predictive asymmetry, bidirectional linear mapping, and representation mismatch are not new. One main extension is the analysis of per-example contributions, but I would view this as mostly empirical analysis instead of new algorithm/method. This largely comes from the nature of the selected problem but not the quality of the work. The comprehensive suite of experiments definitely bridges an important gap and serves as a step forward. Hence, I believe the work would benefit mainly from larger-scale experiments (see below).

Further Ablations: There are other experiments that can be added to strengthen the current argument:

  • Repeating the controlled experiments over training seeds to ensure universal conclusions
  • Verifying per-image contribution: the class-level results are convincing in the experiments, but there are no explicit reports on how Δqi\Delta q_i captures the influence of each individual image. Please consider adding some ablations on this (e.g. correlations obtained via held-out set or cross-fitting).
  • Analyzing predictive asymmetry between models in more contrast: the current experiments mostly compare models from the “same class” with somewhat related structures. It would be worth repeating similar experiments with model pairs that have very different architectures (model size, dimensionality, encoders, etc.)

Overall, I believe the paper provides some novel insights and has carefully designed experiments that can benefit from broader evaluation. Minor weaknesses arise mostly from the nature of this work, and I would recommend “weak acceptance” for now.

Questions for authors

Section 4.1 and Section 5 have identical names. Please consider fixing it. Otherwise, refer to the Weakness section.

5 comments

Thank you for your valuable feedback. We appreciate that you found our approach to representational similarity to be a new perspective, the paper easy to follow, and our experiments effective. We address your specific feedback below and are happy to answer any remaining questions.

1. Contribution and Novelty

Thank you for distinguishing the novelty of the underlying method from the contribution of the study. Our central finding is that a simple measurement carries a surprisingly rich diagnostic signal about how models differ. Bidirectional linear prediction and asymmetric recoverability has been observed before, as we acknowledge in Sec. 2. However, prior works leave an important empirical question unresolved: what can the direction and location of this asymmetry reliably tell us about training differences and relative downstream strengths?

Prior work treats asymmetry as a global property of a model pair (Muzellec & Kar, 2026) or computes it over extracted concepts (Kondapaneni et al., 2025). We decompose it to individual examples in the full representation space. Controlled interventions show that its sign and location recover known differences in supervision, training exposure, and training duration (Sec. 4), and on released specialists it tracks training exposure and localizes where a specialist outperforms its parent (Sec. 5). The simplicity of the diagnostic is a practical strength: these findings come directly from model embeddings through two fitted maps, without labels, concept extraction, clustering, or learned dictionaries.

Following your recommendation, we strengthened our work with larger-scale experiments across seeds, per-image reliability, and architectures (below). We also showed that asymmetry can guide practical decisions in label-free distillation (new "Applications" section in Sec. 6; Global Response, "Practical Applications").

2. Expanded Experiments

Repeating experiments over seeds. In addition to the existing seed repeated experiments (granularity experiment in Section 5), We have now added independent training runs for training coverage (five training seeds), checkpoint progression (five separate training runs), and CLIP fine-tuning (five training seeds per supervision), and each new backbone in the architectural evaluation uses three seeds. The direction of asymmetry is consistent across all seeds in every experiment, and revised Table 1, Figure 2, and Sections 4.1-4.2 report uncertainty across seeds. Full details are in App. B.1.

Verifying per-image contributions Thank you for this suggestion. Following your suggestion, we added cross-fitting and held-out analyses showing that per-image scores are reproducible and conducted within-class analyses to capture the influence of individual images, not only classes.

  • Cross-fitting and retraining stability. All scores are computed out of fold, on images held out from fitting and tuning the maps (App. A.2). Across ten random cross-fitting assignments, per-image scores correlate at r >= .98 for 11 of 12 released pairs (StreetCLIP, the smallest population: r=.86). Across five independently retrained CLIP models, per-image scores on the same images correlate at r=.90–.95.

  • Influence of individual images. To test whether individual images are informative beyond class-level signal, we measured per-image trends within each class: how the specialist's accuracy advantage over its parent changes with qi among images of the same class. This analysis requires enough images per class to form within-class bins, enough classes to aggregate over, and a clear performance gap between specialist and parent to localize. We therefore use RSDiX (RESISC45) and WildCLIP (Snapshot Serengeti), which meet all three criteria. In both pairs, the specialist's advantage grows toward the specialist-favored direction: from the least to the most specialist-favored 10% of images within each class, the accuracy gap increases by a median of 21.0 pp for RSDiX and 10.0 pp for WildCLIP (Fig. 14). Per-image scores therefore identify where the specialist's advantage concentrates within a class, beyond differences between classes.

Sec. 5.2.3 summarizes both results; details are in App. B.5.1 (stability) and App. B.2.4 and Fig. 14 (within-class analysis).

Model pairs with more contrast. We agree this was an important extension and ran it across architectures, model sizes, and embedding widths: predictive asymmetry remains accurate while baselines degrade (full results in the Global Response, "Expanded Scope"). In the paper, we’ve added Sec. 4.1 ("Cross-architecture pairs") and discussions to Secs. 5.1 and 5.2.3; full details in App. B.1.4 and B.2.3.

  • Controlled Experiments: We repeated the supervision-granularity control with four new backbones trained from scratch (ResNet-18, EfficientNet-B0, ConvNeXt-T, ViT-S/16; three seeds each), spanning convolutional and transformer architectures and 384–2,048 dimensions. Changed-class detection achieves AUROC 0.995 across architectures (1.000 same-architecture), with correct attribution in every pair and seed. Baselines degrade across architectures: Information Imbalance drops from 0.946 to 0.879, mutual-kNN from .813 to .582, and PNKA from .855 to .772 (Table 7).

  • Domain Specialists: Predictive asymmetry tracks specialist performance gains across architectures. We added specialist families released at several architectures (RemoteCLIP at ResNet-50, ViT-B/32, and ViT-L/14; BioCLIP at ViT-B/16; BioCLIP 2 at ViT-L/14) and paired them with CLIP references of different architectures, giving eight cross-architecture pairs (e.g., CLIP ResNet-50 vs. RemoteCLIP ViT-L/14). In all eight, the specialist's accuracy advantage grows from the least to the most specialist-favored images (by +6.8 pp on average; Table 13, Fig. 13).

Duplicate Section Titles

Thank you for noticing this. Following Reviewer iAmJ’s feedback, we have re-organized our sections so that each experiment's setup and results appear together, which also removes the duplicate titles (details in the Global Response).

Together, these results show that the conclusions hold across seeds, that per-image scores are reproducible and informative within classes, and that the diagnostic extends to model pairs with very different architectures and dimensionalities.

AuthorsOpenReview ↗

A Kind Reminder

Dear Reviewer XQYB,

Thank you again for your insightful review! Following your feedback, we clarified our paper’s novelty and have strengthened our results by expanding the scope of our experiments with further ablations that you have suggested: experiments over training seeds, verifying per-image contributions, and adding additional architectures.

As we approach the end of the discussion period, please let us know if you have further questions, or if there is anything else we can address for you to consider raising our score. Thank you again for your time and effort!

Reviewer_YQFwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to fit linear maps between a pair of models’ representations in both directions and use their predictive asymmetry – the difference in held-out linear predictability between models – as a diagnostic tool. This signal reveals which distinctions one model preserves relative to another, localizes which examples drive these differences, and can identify where relative performance gains between two models lie. The authors validate the diagnostic through controlled experiments.

Critical strengths
  • The proposed method is simple and interpretable. The signal reveals which distinctions one model preserves relative to another, localizes which examples drive these differences, and can identify where relative performance gains between two models lie, without requiring any labels, concept/dictionary extraction, or clustering.
  • The paper conducted extensive controlled experiments to showcase the diagnostic.
Critical weaknesses
  • Although the method is very general, the experiments are only conducted on models with the same architecture, either separately trained, fine-tuned, or at different training steps. This limits the scope of the method and experiments.
  • The linear mapping may not reflect the actual mapping between different representations. For example, even for the same architecture or the same model, the mapping between different layers should mostly be nonlinear.
Questions for authors
  • The EBmeanE^{mean}_B is not defined in line 159. This seems hard to understand:"pooled held-out error of predicting the target’s training fold mean".
  • How is the method useful in practice?
4 comments

Thank you for your valuable suggestions. We appreciate that you found predictive asymmetry simple, interpretable, and label-free, and our controlled experiments extensive. We address your comments below:

1. Limited Scope of Experiments

Thank you for your constructive feedback. We agree that broader scope is important, and we have substantially extended the evaluation. Predictive asymmetry holds up across architectures, model sizes, and embedding widths, while baselines get worse.

  • Controlled Experiments: We repeated the supervision-granularity control with four new backbones trained from scratch (ResNet-18, EfficientNet-B0, ConvNeXt-T, ViT-S/16; three seeds each), spanning convolutional and transformer architectures and 384–2,048 dimensions. Changed-class detection achieves AUROC 0.995 across architectures (1.000 same-architecture), with correct attribution in every pair and seed. Baselines degrade across architectures: Information Imbalance drops from 0.946 to 0.879, mutual-kNN from .813 to .582, and PNKA from .855 to .772 (Table 7).

  • Domain Specialists: Predictive asymmetry tracks specialist performance gains across architectures. We added specialist families released at several architectures (RemoteCLIP at ResNet-50, ViT-B/32, and ViT-L/14; BioCLIP at ViT-B/16; BioCLIP 2 at ViT-L/14) and paired them with CLIP references of different architectures, giving eight cross-architecture pairs (e.g., CLIP ResNet-50 vs. RemoteCLIP ViT-L/14). In all eight, the specialist's accuracy advantage grows from the least to the most specialist-favored images (by +6.8 pp on average; Table 13, Fig. 13).

We have added a section Sec. 4.1 ("Cross-architecture pairs"), and discussions to Secs. 5.1 and 5.2; A summary is also in the Global Response ("Expanded Scope").

2. Linear Mapping Concerns

Thank you for raising this. We agree that the true relationship between representations is generally nonlinear. We use linear maps deliberately, and our conclusions also hold with a nonlinear MLP.

  • Why linear maps. Our goal is to characterize how two models organize information differently. Two representations can retain the same information while making them differently accessible to a linear readout. A sufficiently expressive nonlinear map could recover the information regardless of how each model organizes it [1, 2], hiding the differences we aim to detect. Linear accessibility is also the constraint used by linear probes, linear model stitching [3], and our downstream validation. We also note that we compare the final pooled outputs of different models, not different layers within one network.
  • Nonlinear maps give the same conclusions. In our paper, we show that relaxing this constraint doesn’t change our key conclusions. As shown in Appendix Tables 19–20, nonlinear residual MLP preserves the controlled attributions, checkpoint ordering, and the global sign for all twelve released pairs. Because this result was easy to miss in the appendix, we now reference it in the main text and state the motivation for linear maps in Sec. 3. We also state this scope in our Limitations (Sec. 7): predictive asymmetry measures linear accessibility, not information content.

3. Practical Utility

Thank you for raising this. We agree that demonstrating practical value is important, so we have added a new "Applications" section (Sec. 6) describing potential uses, and validated one of them experimentally.

Since predictive asymmetry shows, without labels, which model holds distinctions the other lacks and on which inputs, it can guide decisions such as auditing fine-tuning, selecting unlabeled data, and targeting knowledge transfer. We tested this in label-free distillation, where a fine-tuned model is trained to match its parent's representations in order to recover distinctions it lost during fine-tuning:

  • Whether to distill. Distillation only helps if the parent holds structure the fine-tuned model lacks, and otherwise it can erase useful specialization. We therefore distill only models that asymmetry marks as parent-favored. Across our controlled CLIP fine-tuning models, this rule makes the right choice for all nine models: it flags the coarse-label models, where distillation recovers +4.6 points of species accuracy, and skips the others, where distillation would lower it by 2.2 points.
  • Which images to use. We also test whether a released domain specialist, BioTrove, benefits from selective distillation from its parent using only a small set of unlabeled images. Selecting the 20,000 most parent-favored images with predictive asymmetry (out of a 182,000-image pool) improves BioTrove's Rare Species accuracy by +1.14 points. This is roughly twice the gain from selecting images randomly, by coreset, or by Information Imbalance, and larger than the +0.45 gain from distilling on all 182,000 images.

Full results are in the Global Response ("Practical Applications") and App. B.3.

4. Definition of EBmeanE^{\text{mean}}_B

Thank you for catching this. For direction A→BA \to B, EBmeanE^{\text{mean}}_B is the held-out mean squared error of a baseline that always predicts the training-fold mean embedding. Raw residuals are not comparable across directions, because the two targets can differ in embedding size and scale. Dividing by EBmeanE^{\text{mean}}_B expresses each residual as a fraction of the target's own variation around its mean, so both directions are on the same scale even when embedding sizes differ, and qiq_i averages to the out-of-sample R2R^2. We have added this clarification in the main paper (Section 3)

References:

[1] Hewitt and Liang. Designing and Interpreting Probes with Control Tasks. EMNLP, 2019. https://arxiv.org/abs/1909.03368

[2] Xu et al. A Theory of Usable Information Under Computational Constraints. ICLR, 2020. https://arxiv.org/abs/2002.10689

[3] Bansal et al. Revisiting Model Stitching to Compare Neural Representations. NeurIPS, 2021. https://arxiv.org/abs/2106.07682

AuthorsOpenReview ↗

A Kind Reminder

Dear Reviewer YQFw,

Thank you again for your constructive review! Following your feedback, we have expanded the scope of experiments with additional architectures, addressed the linear mapping concerns, modified the paper to clarify E^mean_B, and added discussions (in the rebuttal and the paper) on how this approach could be useful in practice.

As we approach the end of the discussion period, please let us know if you have further questions, and if the concerns have been well-addressed, we would appreciate it if you could consider raising our score. Thank you again for your time and effort!

Dear Reviewer YQFw,

Thank you again for your time with the review! As we approach the last day of the discussion period, we would really appreciate it if you could let us know if the rebuttal has sufficiently addressed your concerns and if you could consider raising our score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response for Reviewers and AC

We thank all of the reviewers for their insightful and helpful feedback. We appreciate that reviewers noted the following strengths:

  1. The method is simple [YQFw, XQYB, iAmJ], interpretable [YQFw], and label-free [XQYB], requiring no concept extraction or clustering [YQFw, iAmJ]
  2. The work offers a new perspective on representation similarity and bridges an important gap [XQYB]
  3. The controlled experiments are extensive and carefully designed [YQFw, XQYB]

We want to emphasize again that our core contribution is establishing what predictive asymmetry reveals about training differences and relative downstream strengths. The reviewers helped identify several aspects of the paper that we were able to directly improve: 1) restructuring the paper for clarity, presenting each major experiment’s setup and results together [iAmJ], 2) expanding experimental scope with cross-architecture comparisons [YQFw, XQYB], and 3) proposing and validating practical uses of predictive asymmetry [YQFw, iAmJ]. Shared revisions are written below, and individual questions are addressed in each reviewer’s response. Updates are blue in our revised PDF.

Restructured Presentation and Section Ordering [iAmJ]

Reviewer concern: The experimental setups were presented in one long block, separate from their results, making the narrative hard to follow.

We agree with this concern and we saw restructuring as an opportunity to significantly improve the paper's flow and readability. Following iAmJ's suggestion, we reorganized the experimental sections so that each experiment's motivation, setup, expected outcome, and results appear together. The reorganization moves sections but does not change their content. The mapping is:

ExperimentOriginal designOriginal resultsNew section
Shared mapping and probing setupSec. 4 (preamble)—Sec. 3
ImageNet controlsSec. 4.1.1Sec. 5.1Sec. 4.1
CLIP fine-tuningSec. 4.1.2Sec. 5.2Sec. 4.2
Released specialistsSec. 4.2Sec. 6Sec. 5
All changes to text, content, and experiments are explicitly highlighted in blue, and moved sections are indicated by blue section titles. The pdfdiff may show these moves as large deletions and insertions, but the only content changes are those in blue; everything else is reorganization.

Expanded Scope: Additional Architectures and Comparisons [YQFw, XQYB]

Reviewer concern: Reviewers YQFw and XQYB asked whether the findings hold for model pairs that differ in architecture, size, and embedding dimension.

We agree that this expansion is important and we therefore extended our comparisons to a more diverse set of architectures and embedding dimensions, beyond the architecture-matched pairs in the submission. We find that predictive asymmetry results are consistent across architectures, model sizes, and embedding widths. We add two new cases:

1) Controlled experiments across architectures: We repeat the supervision-granularity control with four additional backbones trained from scratch (3 seeds each): ResNet-18 [1], EfficientNet-B0 [2], ConvNeXt-T [3], and ViT-S/16 [4,5]. Together with ResNet-50, these span convolutional and transformer architectures and embedding dimensions of 384–2,048. This enables the following comparisons:

  • Same-architecture pairs (as in the original paper)
  • Cross-architecture pairs, including pairs that are over 5x different in dimension: ViT-S (d=384) vs. ResNet-50 (d=2,048)

Results are in Table 7 (App. B.1.4) of our revision. We find:

  • Predictive asymmetry remains accurate across architectures: changed-class detection achieves AUROC 1.000 within and .995 across architectures, with correct attribution in every pair and seed.
  • Baselines degrade across architectures: for example, Information Imbalance drops from .946 to .879, mutual-kNN from .813 to .582, and PNKA from .855 to .772.

2) Released specialists across architectures: We add specialist families released at several architectures: RemoteCLIP (ResNet-50, ViT-B/32, ViT-L/14), BioCLIP (ViT-B/16), and BioCLIP 2 (ViT-L/14) [6]. Pairing each with CLIP references of matching and different architectures gives eight cross-architecture specialist–reference pairs (e.g., CLIP ResNet-50 vs. RemoteCLIP ViT-L/14).

Results are in Table 13 and Fig. 13 (App. B.2.3) of our revision. We observe:

  • In all eight pairs, the specialist's accuracy advantage is larger on specialist-favored images (mean +6.8), and class-level asymmetry tracks per-class F1 gain in 7 of 8 pairs.
  • These results show that predictive asymmetry tracks relative performance across architectures, even when the reference is not the specialist’s direct training parent.

In the main text, we added the controlled cross-architecture results to Section 4.1 and the released specialist families to Sections 5.1 and 5.2.3; full details are in App. A.2.2, B.1.4, and B.2.3.

Practical Applications [YQFw, iAmJ]

Reviewer concern: Reviewers YQFw and iAmJ asked for concrete, actionable use cases, e.g., for model development, data selection, or fine-tuning.

We have now both clarified practical uses in our paper and validated one of them experimentally, adding a new "Applications" Section (Section 6), with full experimental details in App. B.3. Predictive asymmetry gives a label-free signal of which model holds distinctions the other cannot linearly recover, and on which inputs. This supports several practical uses for model adaptation, development, or fine-tuning, including:

  • Auditing fine-tuning without target-domain labels: detect in which direction representations changed and on which inputs, including across successive model updates. Combined with concept-based methods (e.g., RDX), this could help explain semantically what changed.
  • Targeting knowledge transfer: identify where two models hold complementary distinctions, to decide whether and on which inputs to transfer knowledge (e.g., distillation or merging).
  • Investigating coverage gaps: relate class-level asymmetry to training exposure to guide future data collection.

We validated the second use with label-free parent-to-specialist distillation:

  • Whether to distill. Fine-tuning CLIP on coarse order-level labels makes it forget species distinctions its parent retains. Asymmetry flags exactly these models; distilling them back toward CLIP recovers +4.6 points of species accuracy, while distillation lowers accuracy (−2.2) for the models asymmetry does not flag. The rule picks the right action for all three objectives, across all nine models (App. B.3.1, Table 14).
  • Which images to use. Selecting 20,000 unlabeled images by asymmetry improves BioTrove's Rare Species accuracy by +1.14 points, roughly twice the gain of Information Imbalance (+0.56), and more than random or coreset selection or distilling on the full 182,000-image pool (+0.45) (App. B.3.2, Table 15).

We thank the reviewers again for their thoughtful feedback, which has substantially improved the clarity, scope, and practical relevance of the paper. We are happy to answer any remaining questions.

AuthorsOpenReview ↗

Message for the ACs

Dear Area Chairs,

Thank you so much for taking the time to oversee our discussion. As the discussion period is ending, we wanted to reiterate that we have addressed the reviewers’ concerns and feedback as follows: Expanding the analyses to additional architectures and addressing linear mapping concerns (Reviewer YQFw): We added experiments on more architectures. The results hold, and predictive asymmetry is most consistent relative to baselines in cross-architecture comparisons. We also clarified that linear mapping is not meant to estimate the true mapping between representations, and explained this design choice more clearly in the paper. Adding additional analyses across seeds, architectures, and clarifying per-image signals in predictive asymmetry (Reviewer XQYB): Our results are consistent across seeds and additional architectures. We have also added additional analyses that show more clearly that predictive asymmetry tracks relative performance at image-level, not just class-level. Paper presentation, and practical utility (Reviewer iAmJ): We have re-organized our paper section as suggested to make methods and experiments easier to follow. We have also added a discussion of concrete downstream applications, with an analysis demonstrating one of them.

We believe that the rebuttal and the additional updates to the paper address all the concerns in the three reviewers presented. We would really appreciate it if you could take these discussions into consideration.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.