- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The submission investigates the "Platonic representation hypothesis" in the context of tabular foundation models: To what extent do different tabular foundation models with different architectures and different training data converge to similar internal representations. It trains sparse autoencoders (SAEs) on each model, matches SAE features across models and evaluates whether ablating matched and unmatched features explains differences in model performance.
Critical strengths
- I find the general research question interesting: To what extent does the Platonic representation hypothesis apply to tabular foundation models given their diversity of architectures and training data
Critical weaknesses
- The presentation of this submission is poor: It uses dense, jargon-heavy language that often omits necessary context, making the reader guess what the possible intended meaning of a sentence could be. This is particularly problematic in terms of understanding the experimental setup and results. There are many statements throughout the manuscript of which I cannot make sense unfortunately (see questions for examples).
- It is difficult to assess the technical merits of this submission with certainty given the severe presentation issues but I am not sure why it would be surprising that "infusing missing concepts" reduces the gap in performance between two models (see questions).
Questions for authors
- Why would it be surprising that "infusing missing concepts" reduces the gap in performance between two models? What else could explain the gap in performance?
Transfer requires a map between the two models’ embedding spaces. We estimate a global linear map M from matched concept pairs via ridge regression, filtering landmark correlations above the random-SAE noise floor (details in Appendix F.8).
- I am unfortunately not able to infer what was done here. Could you please explain in simple words how you use the matched concept pairs from two models' SAEs to establish a map between embedding spaces? On what is the ridge regression run? Why can't you use the SAEs directly for this?
We apply Leiden community detection Traag et al. (2019) to partition this graph into tightly-connected communities, each representing a single shared concept.
-
This statement seesm to apply that this procedure guarantees a single shared concept per partition? Why is this the case?
-
I am missing a comparison of how the pretraining data affects representational similarity across models (in addition to how architecture affects it). The introduction highlights the diverse data tabular models are trained on and I was expecting such an analysis as a consequence.
To find the layer where the optimal representations could be found for each model, we set output hooks at every intermediate layer to cache intermediate activations for every dataset.
- Optimal for what? The paragraph on "Embedding extraction" was hard to parse, could you re-explain to me based on what criterion you select the intermediate layer used for the ensuing model comparison?
The CKA analysis establishes that models share geometric structure, but does not reveal what that structure encodes or why models differ in performance.
- I am confused by this statement, which analysis results does this refer to at that point in the paper?
Matryoshka mitigates concept splitting, archetypal constraints improve cross-seed stability, and top-k sparsity reinforces both.
- This sentence seems odd, are you stating that top-k sparsity reinforces concept splitting?
Since models have different embedding dimensions (192 to 768), their decoder weight vectors are not directly comparable.
- CKA works in this setting, it is not clear to me why it cannot be used to calculate "Activation-based similarity" (the title of that paragraph)
The number of accepted concepts K is not preset: the per-row greedy admits concepts in importance order while each moves the recipient toward S, and stops when gap closed reaches a tolerance or no remaining concept improves it.
- The submission suffers from poor writing, this is an example sentence that illustrates this. It takes several readings to guess an intended meaning.
Tabular foundation models present a more challenging test: architectures span transformers, hypernetworks, and graph neural networks; training data ranges from synthetic priors to real tables; and there is no shared “Common Crawl” equivalent. Building on geometric tools—CKA and Procrustes alignment—we go beyond that comparison with concept-level causal analysis, examining what models learn and why they differ in performance.
-
What is "that comparision"? Is there a sentence missing?
-
What is a TPE sweep (L198)? This acronym has not been introduced.
-
The symbols in equation (2) are not defined. What is ?
-
How/where is "gap closed gc" used in Algorithm 1?
-
I am unable to decipher Figure 1a. Can you explain in simple words what is inteded to show?
The interventions are a causal diagnostic, not a deployment method: they take per-row knowledge of which model is stronger as an experimental control
- I don't understand what that means
Minor remarks
- There are several instances where citations are missing their parentheses (citet instead of citep), e.g. L167
Thank you for your response to my questions. I appreciate the additional hypotheses you added to attempt to explain what might might explain the closing of the gap to the stronger model.
As I highlighted in my initial review, a central concern for this submission is its poor presentation, specifically the dense, jargon-heavy language used throughout, that often omits the context that would be necessary to understand what is actually being done. In the same vein, it took me considerable effort to parse the responses to my questions. As it stands, I cannot confidently recommend this paper for acceptance and will maintain my score. That being said, I think the research direction is interesting and I would strongly encourage the authors to carefully rewrite the manuscript to make it (human) readable for re-submission at a future conference.
This made me wonder if there is some way to characterize this "weight edit" compared to fine-tuning the model for the particular error that the weight edit fixes. Are the respective updates aligned to each other? Is this effectively like training the model?