- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the geometric organization of knowledge accessibility in LLM representation spaces. The authors hypothesize that queries whose required knowledge is more accessible lie closer to a learned accessibility center. The authors compare isotropic and anisotropic center-based gemoetries against linear and nonlinear probing baselines. Across two LLMs and several QA datasets, distance from the learned center correlates with factual recall, transfers partially across datasets, and behaves differently from reasoning capability. Then, the authors further examine how query rewriting, chain-of-thought reasoning, and retrieval-augmented generation help in different regions of the learned knowledge boundary.
Critical strengths
S1. The question of focus in this paper is interesting and well-motivated. The attempt to separate accessibility from reasoning capability is a valuable distinction.
S2. The formulation is simple and interpretable. The comparison against linear probing, anisotropic distance, and an MLP provides initial evidence about this inductive bias.
S3. The downstream analyses are useful and thorough. Correctness stability near the boundary, cross-dataset transfer, and the differing effects of rewriting, reasoning, and retrieval make practical suggestions for adapting inference strategies to knowledge accessibility accordingly.
Critical weaknesses
W1. The experiment scope is quite limited. The main experiments use only two relatively small instruction-tuned LLMs, and the intervention experiments are restricted to TriviaQA. The support is thin for a claim about a general property of LLM knowledge accessibility. Larger models, additional families, and reasoning-capable models would help. It also remains unclear whether the geometry exists in the corresponding pre-trained models or emerges during instruction tuning, and whether tuning shifts the center or the shape of the boundary. In addition, testing on temporal updates, conflicting knowledge, and safety-related queries would be interesting.
W2. The single-center assumption is insufficiently justified. LLM knowledge spans diverse domains that may form multiple disconnected or non-convex regions, and the paper's own transfer experiments show that learned centers and dimension weights differ considerably across datasets. Comparisons against multi-center or mixture-based geometrics would establish whether this assumption is appropriate or overly resttrictive. It would also help to test whether paraphrases of the same question occupy similar regions.
W3. The empirical advantage over nonlinear baselines is marginal. In Figure 1, the anisotropic centered model beats the MLP only slightly on three of four factual QA settings and loses on the fourth. It is unclear whether these gaps are statistically significant. The setup also needs clearer description. The main paper, instead of Appendix A, should explain how the center and dimension weights are optimized, what objectives are used, how scores become correctness predictions, how per-layer AUROC is computed, and how hyperparameters are chosen.
W4. The writing is generally clear, but several paragraphs repeat the motivation and implications of the accessibility-center hypothesis over and over again. The related-work discussion could also be more comprehensive, especially regarding the organization of knowledge in representation space, knowledge-status characterization, and the effects of instruction tuning on knowledge accessibility.
Questions for authors
See weaknesses.