- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper presents an interesting perspective on quantitative extrapolation and includes careful experimental controls. However, the methodological contribution remains limited, as simpler regression alternatives achieve stronger performance and several components of the proposed correction are not convincingly justified. Furthermore, the main improvements are concentrated on a small set of numerical tasks and do not consistently transfer to more diverse quantitative representations.
Critical strengths
- The paper investigates whether endpoint-relative relationships can improve numerical predictions beyond the supervised range, providing an interesting perspective on quantitative information encoded in language-model representations.
- The distinction between information accessibility, distance alignment, and extrapolation provides useful insights into why recoverable numerical information does not necessarily translate into reliable out-of-range predictions.
Critical weaknesses
- The proposed approach primarily combines existing regression techniques with endpoint-relative distances and residual correction. More importantly, simpler alternatives such as direct radius regression and all-source fitting achieve substantially better performance, while removing residual detrending also improves the results, raising questions about the necessity of the proposed design.
- The proposed correction provides negligible improvements on entity attributes and even degrades performance on some unseen numerical notations, limiting the evidence that the approach generalizes beyond the specific experimental settings.
Questions for authors
1.Given that simpler direct regression methods outperform the proposed correction, what is the unique methodological advantage of the proposed framework? 2.What explains the substantial differences between the core numerical tasks and the entity-attribute and alternative-notation experiments? 3.How sensitive are the results to the global reader's extensive hyperparameter selection, particularly given the small number of source observations and pseudo-validation examples?
2 comments
Thanks for your constructive answers. I have increased my score correspondingly.
Response to LECb on baselines, task differences, and selection
Thank you for asking about the direct baselines and source selection. The revision measures what endpoint distance contributes to a particular reader and tests how that contribution changes across validation blocks.
Q1. What endpoint distance adds. Section 4 distinguishes scaling the reader’s existing gaps from adding a distinct pattern of gaps. In Table 1, the original correction improves pair nMAE relative to standalone boundary-radius ridge (0.5923 vs. 0.6048). The direct ridge gives point nMAE of 0.5349, compared with 0.6324 for the original correction. Fitting the radial component on all source items brings point and pair nMAE to 0.5289 and 0.5554. The variant without detrending gives 0.6216 and 0.5910. The point-error contrast between intact and permuted correspondence for the selected simple reader is +0.000747 (95% interval [−0.002565, +0.004731]). These comparisons show how the effect of an endpoint measurement depends on the starting reader and fitting support.
Q2. Differences across tasks. The Roman numeral and duration prompts state the target quantity explicitly. The entity prompts ask for a property of the named entity. In the notation panel, point nMAE is 0.5307 for the final-loss-selected reader and 0.5372 with the fixed correction. In the matched entity panel, replacing Euclidean distance with PLS raises the radial component’s mean gain from 0.000652 to 0.000749 for the ridge reader and changes it from 0.000288 to −0.000336 for the PLS reader (Table 2). In four upper birth- and death-year conditions, a probe trained within the held range recovers variation and beats the endpoint constant on both point and pair error. The source-trained probes have larger point errors than the constant in these four conditions. The matched comparison separates recoverability from transfer on the same representations.
Q3. Sensitivity to source selection. The global reader’s mean point nMAE improves under both final-loss selection (0.7141 to 0.6324) and upstream-loss selection (0.7189 to 0.6345). The two objectives select different nominal configurations in 376 of 384 conditions. In the re-extraction, changing which items enter the validation score changes the held forecast in 29 of 50 single-item deletions for the global reader and 32 of 50 for the simple reader, with candidate fits fixed (Appendix G). In upper percentages, the gain of joint ridge over pooled direct prediction is +0.2582 on a near-boundary block and −0.5963 farther inside, with four training labels and three validation items in each block (Section 5.5). The matched counts show that the choice of validation items matters in its own right.