- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces a diagnostic metric, predictive asymmetry, that measures differences in linear recoverability between two models' representations using bidirectional affine mappings. By examining the difference in reconstruction quality in both directions, the method aims to identify which model preserves relatively less recoverable information and localize these representational differences.
Critical strengths
- The proposed method is conceptually simple, requiring only an evaluation dataset and the fitting of bidirectional affine mappings between the models' representations. Since the reconstruction targets are the models' own features, the approach does not require annotated data. This enables a label-free analysis of directional linear recoverability, allowing the identification of which model preserves representational distinctions that are less linearly accessible from the other.
Critical weaknesses
- Difficult to follow experimental narrative. A substantial portion of the paper is devoted to the experimental design and results. However, the organization and presentation of these sections make the paper unnecessarily difficult to follow. Section 4 provides approximately two pages of dense experimental descriptions, including detailed configurations and discussions of expected outcomes across several distinct experiments. The corresponding results are only presented afterward, requiring the reader to retain substantial amounts of information to understand the subsequent discussions.
Furthermore, the results themselves are not always clearly connected to their supporting evidence. For instance, in lines 322–323, the authors present an empirical result and its interpretation without providing a direct reference to the corresponding figure or table. This makes it difficult to follow the argument on a first reading, often requiring the reader to navigate back and forth between the experimental setup and the results to reconstruct their relationship.
Given the density of the experimental descriptions and the extensive discussion of expected outcomes, I frequently found it necessary to consult the experimental design while reading the corresponding results. This substantially disrupts the reading flow and makes the empirical findings more difficult to interpret than necessary.
I strongly encourage the authors to reconsider the organization of Sections 4–6 to improve readability and establish a more coherent experimental narrative. One possible approach would be to organize the experiments into separate individual sections, each presenting the experimental motivation, setup, expected outcomes, and corresponding results together. This would allow readers to follow the reasoning behind each experiment and directly connect its findings to the underlying hypotheses. Additionally, I strongly recommend providing explicit cross-references to the relevant figures and tables when prior to introducing and discussing empirical findings, rather than requiring readers to identify the supporting evidence themselves. - Limited demonstration of practical utility. While the authors provide evidence of the proposed method's diagnostic capabilities, the paper does not convincingly establish the actionable utility of the resulting insights. In particular, it remains unclear how predictive asymmetry could be used in practice to inform model development, evaluation, or adaptation beyond retrospective representation analysis. Although the method successfully identifies representational differences in controlled experiments, the paper does not demonstrate how these findings could translate into concrete practical benefits.
Furthermore, while the authors compare predictive asymmetry against existing methods, including information imbalance [1], the practical advantages and limitations of predictive asymmetry relative to these baselines remain insufficiently discussed. I would encourage the authors to elaborate on the tradeoffs between the proposed method and existing approaches, and to clarify to what extent predictive asymmetry addresses an unmet need in the current landscape of representation comparison methods.
[1] Acevedo et al., "A quantitative analysis of semantic information in deep representations of text and images", TMLR, 2026.
Questions for authors
-
What are the practical advantages of predictive asymmetry compared to existing methods, such as information imbalance [1], beyond the quantitative improvements reported in Table 1? In which scenarios would the authors recommend using predictive asymmetry rather than the alternatives?
-
Can the authors provide a concrete use case in which predictive asymmetry could produce actionable insights? For instance, how could the insights obtained through predictive asymmetry inform decisions regarding model development, data selection, or fine-tuning?
-
What are the tradeoffs associated with predictive asymmetry compared to the evaluated baselines? Are there particular settings or conditions under which the authors would recommend using existing methods instead of predictive asymmetry?
4 comments
3. Advantages and Trade-offs Relative to Existing Methods
Thank you for these questions. We added a dedicated discussion of advantages, trade-offs, and when to use each method in App. B.6, referenced from the baseline comparison in Sec. 4.1. We also answer your specific questions below:
Unmet need. Symmetric metrics (symmetric predictivity, mutual-kNN, PNKA) detect where two models differ, but cannot say which model the difference comes from. Local dimensionality measures (e.g., local PR) score each model separately, so comparing them points to a model only if one assumes how training changes dimensionality. That assumption is not reliable: local PR correctly identified the later checkpoint in all 75 pairs, but fails for supervision and coverage changes (attribution .059 and .342; Table 1). Concept- and group-based methods (RSVC, RDX) require extracting concepts or clustering inputs first, and their results depend on that step. Predictive asymmetry identifies which model has organization the other cannot predict, for each image and any subset, directly from the embeddings, without labels or concept extraction.
Advantages over Information Imbalance. Both methods are directional and label-free. Information Imbalance asks whether each image's nearest neighbor is preserved across models, whereas predictive asymmetry measures how well each image's full embedding is predicted from the other model. This difference has practical consequences:
-
More reproducible localization: Class-level scores from two halves of the same data agreed closely for predictive asymmetry (.97) but not for Information Imbalance (.25). As Information Imbalance relies on neighborhood structure, it is very sensitive to data splits, even within the same distribution.
-
Better tracking of real specialist gains. On released specialists, predictive asymmetry is the only method whose class-level scores track per-class F1 gain in all eight core pairs (mean ρ = .54, versus .25 and 5/8 pairs for Information Imbalance; App. B.2.2, Table 12).
-
Robustness across architectures. Moving from same-architecture to cross-architecture pairs, predictive asymmetry's detection AUROC barely changes (1.000 to .995), while Information Imbalance's drops from .946 to .879 (Table 7).
-
More useful data selection. In the distillation experiment above, selecting by asymmetry gives a larger gain than Information Imbalance (+1.14 vs. +0.56 points), because it targets the images where the parent's structure is missing: 65% of its selections come from taxonomic classes BioTrove never saw, versus 17% for Information Imbalance (Table 15).
When to use alternatives. Symmetric metrics are the simpler choice when the question is only whether two models differ; local PR is cheap and accurate for ordering checkpoints of a single model; Information Imbalance needs no fitted map and may be preferable with very few images; and concept-based methods (RSVC, RDX) are needed to explain what differs semantically. These are complementary to asymmetry, which can first locate where and in which direction models diverge.
Limitations of predictive asymmetry. Predictive asymmetry captures only linearly accessible structure and needs enough images to fit a map in each direction. Its cost grows linearly with the number of images (versus quadratically for exact Information Imbalance; the two take similar time at 20,000 images, and asymmetry is about twice as fast at 40,000), but faster with embedding width. When two representations share little linear structure, so that predictivity is low in both directions, their asymmetry should be interpreted with caution. We also state these limitations in Sec. 7 and Appendix B.6.
A Kind Reminder
Dear Reviewer iAmJ,
Thank you again for your helpful review! Following your feedback, we have significantly improved the ordering/presentation of our paper to make the methods and experiments easier to follow by re-ordering Sections 4-6 as you suggested. We have also added more discussions and demonstrations of practical utility (in the rebuttal as well as in the paper).
As we approach the end of the discussion period, please let us know if you have further questions, and if the concerns are resolved, we would be grateful if you would consider raising your score. Thank you again for your time and effort!
Dear Reviewer iAmJ,
Thank you again for your time and insights! As we approach the last day of the discussion period, we would really appreciate it if you could let us know if the rebuttal has sufficiently addressed your concerns and if you would consider raising our score.
Thank you for your detailed and constructive review. We appreciate that you found predictive asymmetry conceptually simple and label-free with useful directional signals. We address the experimental narrative, actionable utility, and tradeoffs below, and are happy to answer any remaining questions
1. Presentation and Experimental Narrative
Thank you for this feedback, which helped us substantially improve the paper's readability. Following your suggestion, we restructured the experimental sections so that each experiment's setup and results appear together.
2. Practical Utility
We agree that demonstrating practical value is important. We added a new "Applications" section to the main paper (Sec. 6) describing potential uses, and validated one of them experimentally (App. B.3).
Because predictive asymmetry shows, without labels, which model holds distinctions the other lacks and on which inputs, it can guide decisions such as auditing fine-tuning, selecting unlabeled data, and targeting knowledge transfer. We tested this in label-free distillation, where a fine-tuned model is trained to match its parent's representations in order to recover distinctions it lost during fine-tuning:
Full results and additional details are in the Global Response ("Practical Applications"), in addition to the paper sections.