- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The manuscript introduces SenseMath, a benchmark for evaluating whether LLMs (can) use number sense to solve mathematical problems. They vary the difficulty of the problem through a CLT inspired variation of intrinsic load to see whether the benefits change across a scale. The paper finds that LLMs can often execute shortcuts if prompted to use them, but its not clear to me from the paper whether it helps on the performance of math tasks. I think its a well done and written paper and the only concern that keeps me from an accept level rating is the decision to use multiple-choice options to evaluate math problems which may introduce confounds and is unintuitive.
Critical strengths
- I like the inspiration through CLT and I appreciate that the authors vary intrinsic load and look at how it impacts the performance/gain of models.
- The paper was a pleasure to read and I like its framing and the way the benchmark was inspired and designed.
- The paper seems well thought out and the benchmark tasks well designed.
Critical weaknesses
- These questions are mathematical problems but they are evaluated using multiple-choice options. This seems a bit unintuitive and suspicious to me. Its something different (use number sense) to solve a problem, or to check which of the solutions is the correct one, and since there is no trace level analysis to distinguish whether the LLMs are actually solving the problem. This is simple problems, and if LLMs can use, and benefit from number sense, then it should be trivial to evaluate their accuracy without a multiple choice option. The judge prompt also doesn't ask the judge to differentiate whether the model actually solves the problem or whether it brute force checks the multiple-choice answers. This is a fundamental concern that motivates my weak reject rating and I'm happy to move to an accept rating if the authors can explain this or provide results without multiple choice answering.
- I believe the extraneous load should also be varied by adding variable amounts of terms/noise in equations/problems that are unrelated to the shortcut. Its a bit more involved than the variation of intrinsic load through increasing the number of digits, but this would strengthen the diagnostic and move me to a clear/strong accept rating. I believe this is particularly relevant because in practical settings filtering out/recognizing the shortcuts is part of the problem, and this is what extraneous load would measure.
- I don't understand based on the paper whether the LLMs performance actually improves through number sense. The paper often compares CoT - NS but I don't find any results/discussion for how NS or CoT improves over strict prompting without number sense.
Questions for authors
- Could you explain why you opted for multiple-choice answering? Have you evaluated freeform answering, do the results change/differ?
- Do you think the framework could be expanded to also vary extraneous load, or do you think this think this would inherently be impossible?
- How does NS prompting improve the accuracy of LLMs over default prompting? Do LLMs benefit from using NS or is it mostly useful for a teaching/educational setting when explaining math to humans?
4 comments
Response to Reviewer tSkK (Part 2/2)
W2 / Q2 (continued): Implementation and evaluation details.
A: Matched settings. Within each model, all presentation conditions use matched prompt templates, provider configurations, and a 4,096-token output limit. The original-problem baseline was rerun under these settings. Qwen3-30B-A3B denotes Qwen3-30B-A3B-Instruct-2507 with fixed FP8 precision; Qwen3-8B uses non-thinking mode. Noise sets are nested across the 3-, 6-, and 9-item conditions.
Sample sizes. Each original-problem cell in the main table contains 400 strong items; each contextual cell contains 1,200 responses across three scenarios. One API failure in GPT-4.1-mini NS clean context is excluded. Across strong, weak, and control variants, all 155,999 valid responses were annotated, including incorrect, unparsed, and truncated answers.
Shortcut evaluation. We use gpt-5.6-luna with low reasoning effort and the manuscript’s human-cognitive-effort criterion. The judge receives the presented question, options, and full solution, without solver identity, condition labels, or gold answers. SU measures whether the visible method exploits numerical structure to simplify human calculation; correctness, brevity, and claims of using a shortcut are insufficient by themselves.
Judge checks and uncertainty. A stratified 100-response batch/single check yielded 90% agreement. Replacing the presented question with the canonical mathematical question yielded 73/80 agreement in a separate sensitivity check. These are automatic consistency checks, not independent human validation. Paired intervals use family-clustered resampling and condition on the judge labels; not every SU contrast excludes zero.
Interpretation. Added noise also increases input length. The experiment varies extraneous numerical information rather than directly measuring models’ internal cognitive load, and the observed effects are not necessarily monotonic across intermediate noise levels.
W3: Does NS prompting improve answer accuracy?
A: Thank you for pointing this out. Our results show that NS prompting can improve answer accuracy, beyond increasing shortcut usage. Figure 5(a) and Table 6 report these improvements on strong-shortcut problems, with the discussion in lines 353–359. For example, on eight-digit problems:
| Model | CoT accuracy | NS accuracy | Gain |
|---|---|---|---|
| GPT-4.1-mini | 71.2% | 90.0% | +18.8 pp |
| Qwen3-30B | 43.2% | 63.7% | +20.5 pp |
| Qwen3-8B | 63.7% | 72.5% | +8.8 pp |
The category-level results provide further examples. On eight-digit relative-distance problems, GPT-4.1-mini improves from 32% to 84%, and GPT-4o-mini from 46% to 66%, as discussed in lines 371–375 and shown in Figure 5(b). These gains depend on the model and problem structure.
Q3: Do models benefit from NS prompting, or is its main value educational?
A: The accuracy benefits of NS prompting depend on the model, problem structure, digit scale, and comparison prompt (Figure 5 and Tables 5–6).
Our primary motivation is educational. As stated in Section 3.1, lines 214–215, we examine whether models can recognize when a shortcut applies, explain it clearly, avoid misusing it, and generate new practice problems. SenseMath evaluates these capabilities; it does not assume that encouraging shortcuts will always improve answer accuracy.
Thank you for the detailed response and for running the new experiments. In my review I committed to move to an accept rating if the multiple-choice concern was addressed with results, and that the extraneous-load extension would be an interesting addition. You ran both these experiments, and I therefore raise my score from 2 to 3.
I am not going further than 3 because the new results show that the accuracy-gain claims, which are prominent in the abstract, depend on the multiple-choice format and the 512-token budget.
Thank you for your thoughtful follow-up and for acknowledging our additional experiments. We appreciate your willingness to update your rating.
We agree that the accuracy benefits of NS prompting should be interpreted carefully, as they depend on several factors, including the answer format, token budget, model capability, digit scale, and problem category. In particular, our category-level analysis (Figure 5(b) and Appendix D) shows that even under the same evaluation setting, NS prompting can yield substantial improvements for some shortcut categories while providing limited or even negative gains for others.
Our intention is not to claim that NS prompting universally improves accuracy, but rather to investigate when and how LLMs recognize and exploit numerical structure to simplify mathematical reasoning. We will revise the abstract and corresponding discussion to better qualify the accuracy-gain claims and clarify their dependence on evaluation conditions and problem characteristics. We will also incorporate the additional free-form evaluation and extraneous-load experiments into the revised manuscript.
Thank you again for your constructive feedback, which has helped us strengthen the paper.
Response to Reviewer tSkK (Part 1/2)
W1 / Q1: Why use multiple-choice questions, and how do results change without answer options?
A: Thank you for raising this concern. We chose a consistent multiple-choice format to keep the response format fixed across problem variants and digit scales. It also allows us to evaluate option-level number sense, such as using parity or the units digit to eliminate implausible answers without completing the full calculation.
We agree that answer options can influence performance. We additionally evaluated three models without answer options on structural shortcuts, cancellation, and equation reasoning problems. The evaluation covers strong-shortcut, weak-shortcut, and control problems with 2-, 4-, 8-, and 16-digit numbers.
As shown in the table, models retain substantial accuracy without options, although accuracy decreases in five of the six model–prompt conditions. These results show that answer options affect performance, and we will distinguish performance with and without options more explicitly in the manuscript. We will include this comparison in the revision.
W2 / Q2: Can SenseMath be extended to vary extraneous load?
A: Thank you for this suggestion. SenseMath can be extended to vary extraneous load, and we have conducted an additional experiment along this dimension.
Extraneous load concerns demands introduced by the presentation of a problem beyond those needed for the target calculation. In our setting, irrelevant numerical information requires models to identify the relevant quantities while recognizing and applying an available shortcut.
We constructed contextualized versions of the original problems at d = 4, using three scenarios per problem and four levels of numerical noise: 0, 3, 6, or 9 noise items. Across these levels, we preserved the target mathematical structure, answer options, and correct answer. We evaluated all five models under CoT and NS prompting, comparing the original problems, clean contextualized problems, and contextualized problems with increasing amounts of noise.
On strong-shortcut items, we observe two main findings:
The table reports accuracy / SU (%) on strong-shortcut items. We will include this extension in the revision.