- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper examines why linear relational embeddings (LREs) are faithful for some relations but not others. It introduces the entity lens, a Jacobian-based readout from same-type entities under a neutral prompt, to rank relation-conditioned answers. Across 28 relations in OLMo-3-7B, lens scores strongly correlate with LRE faithfulness.
Critical strengths
It would be meaningful to explore why the linear approximation is accurate for some relations but completely inaccurate for others.
Critical weaknesses
The entity lens is defined as , whereas the original Jacobian computation is . Since , the two expressions are algebraically identical apart from the placement of parentheses and therefore do not represent different approximations. The actual differences should be clarified more carefully.
Questions for authors
- The paper conducts experiments on only 28 relations, so the conclusions drawn may not be reliable.
- Section 5 states that 8 (subject, object) examples will be provided. Does this not implicitly convey relation information, and thereby run counter to the original intent of using a neutral prompt?
- Table 6 shows that ρ is highly sensitive to neutral prompts; under such circumstances, drawing any conclusion about the correlation ρ is highly unreliable.
3 comments
Thanks for your response. I have raised the score.
We appreciate that the reviewer took the time to work through our response and reconsider their assessment of our submission.
Thank you!
Thank you for reading our paper and raising important questions.
For NEL (meaniJise), we do not claim an algebraic difference from Jˉs in Gurnee et al. (2026): both have the same functional form and differ only in where Ji is evaluated (at same-type reference entities under
"{} is known for"rather than arbitrary corpus tokens; lines 131-134 and 172-175).For CEL (meaniJi(se−si)=meaniJise−meaniJisi), it is algebraically different from Jˉs because the subtracted term meaniJisi is a vector, not a scalar constant. When projected onto the vocabulary, it subtracts a different value from each token (demoting attributes shared across reference entities) and changes the token ranking. Empirically on OLMo-3-7B, evaluating Ji on same-type entities in
"{} is known for"improves the mean log-rank from −3.48 to −2.59 and the correlation from ρ=0.630 (Gurnee et al.'s corpus Jˉs) to ρ=0.735 (CEL) and ρ=0.765 (NEL). We have revised Section 4 (lines 164-167) to make this distinction clearer: after the definition of NEL, the manuscript now states that NEL shares the form of the Jacobian lens and differs in where the Jacobians are evaluated, and that CEL subtracts a vector that changes the token ranking.To test a larger set, we added the BEAR benchmark (Wiland et al., 2024). Factual recall of OLMo-3-7B on BEAR is much lower than on the original LRE dataset (38.0% vs. 76.6% gold first-token accuracy), which leaves too few facts on which faithfulness to the model's own answer is informative. We therefore ran the evaluation on the larger Qwen3-32B, scoring, as in Hernandez et al. (2024), only facts whose gold first token the model predicts. Across all 49 entity relations, the correlation with LRE faithfulness is ρ=0.597 for CEL and 0.544 for NEL. On this larger and harder relation set, the correlation remains in the range of the seven models in Table 1. We have added this analysis to Appendix C and plan to extend the BEAR evaluation to the other models for the camera-ready version.
As stated in lines 196-199 and line 241, the 8
(subject, object)examples are only used as per definition for the LRE and to record the model's top-1 target token me; the entity lens itself only ever sees"{subject} is known for"with no relation template and no objects.In Table 6 (lines 768-781), the low values on Qwen3-8B come from position-0 activation norm spikes (discussed in lines 764-766) when
{s}is placed at token index 0 without a prefix, and from bare prompts like"{s}"or"{s} is". Across all prefix-guarded neutral prompts in Table 6, Qwen3-8B stays within ρ∈[0.665,0.733]. Separately, across 6 neutral prompts on OLMo-3-7B it stays within ρ∈[0.729,0.777] (all p≤1.1×10−5). While some variance across prompt phrasings is expected, it does not meaningfully change the results or correlation.