acceptodds
Under review as a conference paper at ICLR 2027

SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation

Abstract

Large language models often default to step-by-step computation even when efficient numerical shortcuts are available. This raises a basic question: do they exhibit number sense in a human-like behavioral sense, i.e., the ability to recognize numerical structure, apply shortcuts when appropriate, and avoid them when they are not? We introduce SenseMath, a controlled benchmark for evaluating structure-sensitive numerical reasoning in LLMs. SenseMath contains 4,800 items spanning eight shortcut categories and four digit scales, with matched strong-shortcut, weak-shortcut, and control variants. Guided by Cognitive Load Theory, we manipulate intrinsic load through digit scale and shortcut availability while controlling extraneous load through a consistent item type and format. Following Bloom's Taxonomy, SenseMath supports three evaluation settings of increasing cognitive complexity: Shortcut Use, Applicability Judgment, and Problem Generation. Our evaluation across six model configurations reveals a consistent gap between executing shortcuts and understanding their applicability. Explicit number-sense prompting yields accuracy gains of up to 23.0 percentage points on shortcut-amenable items, yet under standard chain-of-thought prompting, shortcut usage on four-digit strong-shortcut items ranges from 26.3% to 58.8%. Models systematically overgeneralize shortcut applicability, with judgment accuracy ranging from 52.5% to 55.9%, and struggle to generate valid strong/control problem pairs, with only 0–21.9% passing all six automatic checks. Together, these results suggest that current LLMs exhibit procedural shortcut fluency without consistently demonstrating the structural understanding of when and why shortcuts work that underlies human number sense.

Primary area:
datasets and benchmarks
Keywords:
Number Sense, Reasoning, LLMs, Education

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tSkKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript introduces SenseMath, a benchmark for evaluating whether LLMs (can) use number sense to solve mathematical problems. They vary the difficulty of the problem through a CLT inspired variation of intrinsic load to see whether the benefits change across a scale. The paper finds that LLMs can often execute shortcuts if prompted to use them, but its not clear to me from the paper whether it helps on the performance of math tasks. I think its a well done and written paper and the only concern that keeps me from an accept level rating is the decision to use multiple-choice options to evaluate math problems which may introduce confounds and is unintuitive.

Critical strengths
  • I like the inspiration through CLT and I appreciate that the authors vary intrinsic load and look at how it impacts the performance/gain of models.
  • The paper was a pleasure to read and I like its framing and the way the benchmark was inspired and designed.
  • The paper seems well thought out and the benchmark tasks well designed.
Critical weaknesses
  • These questions are mathematical problems but they are evaluated using multiple-choice options. This seems a bit unintuitive and suspicious to me. Its something different (use number sense) to solve a problem, or to check which of the solutions is the correct one, and since there is no trace level analysis to distinguish whether the LLMs are actually solving the problem. This is simple problems, and if LLMs can use, and benefit from number sense, then it should be trivial to evaluate their accuracy without a multiple choice option. The judge prompt also doesn't ask the judge to differentiate whether the model actually solves the problem or whether it brute force checks the multiple-choice answers. This is a fundamental concern that motivates my weak reject rating and I'm happy to move to an accept rating if the authors can explain this or provide results without multiple choice answering.
  • I believe the extraneous load should also be varied by adding variable amounts of terms/noise in equations/problems that are unrelated to the shortcut. Its a bit more involved than the variation of intrinsic load through increasing the number of digits, but this would strengthen the diagnostic and move me to a clear/strong accept rating. I believe this is particularly relevant because in practical settings filtering out/recognizing the shortcuts is part of the problem, and this is what extraneous load would measure.
  • I don't understand based on the paper whether the LLMs performance actually improves through number sense. The paper often compares CoT - NS but I don't find any results/discussion for how NS or CoT improves over strict prompting without number sense.
Questions for authors
  • Could you explain why you opted for multiple-choice answering? Have you evaluated freeform answering, do the results change/differ?
  • Do you think the framework could be expanded to also vary extraneous load, or do you think this think this would inherently be impossible?
  • How does NS prompting improve the accuracy of LLMs over default prompting? Do LLMs benefit from using NS or is it mostly useful for a teaching/educational setting when explaining math to humans?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tSkK (Part 1/2)

W1 / Q1: Why use multiple-choice questions, and how do results change without answer options?

A: Thank you for raising this concern. We chose a consistent multiple-choice format to keep the response format fixed across problem variants and digit scales. It also allows us to evaluate option-level number sense, such as using parity or the units digit to eliminate implausible answers without completing the full calculation.

We agree that answer options can influence performance. We additionally evaluated three models without answer options on structural shortcuts, cancellation, and equation reasoning problems. The evaluation covers strong-shortcut, weak-shortcut, and control problems with 2-, 4-, 8-, and 16-digit numbers.

ModelPromptWith options (%)Without options (%)
GPT-4.1-miniCoT88.3981.22
GPT-4.1-miniNS88.1778.56
Qwen3-30B-A3B-Instruct-2507CoT58.1160.89
Qwen3-30B-A3B-Instruct-2507NS64.6158.72
Qwen3-8BCoT81.3973.72
Qwen3-8BNS67.7265.72

As shown in the table, models retain substantial accuracy without options, although accuracy decreases in five of the six model–prompt conditions. These results show that answer options affect performance, and we will distinguish performance with and without options more explicitly in the manuscript. We will include this comparison in the revision.

W2 / Q2: Can SenseMath be extended to vary extraneous load?

A: Thank you for this suggestion. SenseMath can be extended to vary extraneous load, and we have conducted an additional experiment along this dimension.

Extraneous load concerns demands introduced by the presentation of a problem beyond those needed for the target calculation. In our setting, irrelevant numerical information requires models to identify the relevant quantities while recognizing and applying an available shortcut.

We constructed contextualized versions of the original problems at d = 4, using three scenarios per problem and four levels of numerical noise: 0, 3, 6, or 9 noise items. Across these levels, we preserved the target mathematical structure, answer options, and correct answer. We evaluated all five models under CoT and NS prompting, comparing the original problems, clean contextualized problems, and contextualized problems with increasing amounts of noise.

On strong-shortcut items, we observe two main findings:

  1. Accuracy does not consistently decrease with added noise. Its changes vary across models and prompting conditions, with some showing decreases and others remaining stable or improving.
  2. Under NS prompting, shortcut usage is highest on the original problems without added textual context across all five models and lowest with nine pieces of numerical noise in four of the five models. This suggests that contextualization and irrelevant information can hinder the deployment of numerical shortcuts, even when answer accuracy does not decline.

The table reports accuracy / SU (%) on strong-shortcut items. We will include this extension in the revision.

ModelPromptOriginalClean context+3 noise items+6 noise items+9 noise items
GPT-4.1-miniCoT99.50 / 53.2599.58 / 45.3399.42 / 45.3399.58 / 45.1799.67 / 48.83
GPT-4.1-miniNS97.50 / 92.0097.91 / 83.9099.00 / 82.4299.08 / 82.2599.08 / 81.33
GPT-4o-miniCoT83.75 / 48.0082.42 / 34.5882.92 / 33.9282.50 / 37.0881.25 / 38.58
GPT-4o-miniNS86.00 / 83.2578.75 / 64.5075.75 / 62.1776.83 / 61.9276.08 / 61.67
Llama-3.1-8BCoT69.50 / 29.0074.67 / 15.2577.08 / 13.3377.25 / 10.9275.42 / 11.33
Llama-3.1-8BNS55.50 / 52.2564.00 / 40.5065.92 / 26.2569.67 / 29.9268.75 / 27.83
Qwen3-8BCoT93.50 / 52.5094.92 / 48.5895.33 / 47.5096.92 / 46.5896.92 / 49.83
Qwen3-8BNS93.25 / 87.5090.42 / 81.3392.33 / 77.3393.67 / 77.5894.42 / 75.58
Qwen3-30B-A3BCoT100.00 / 58.2599.75 / 54.5898.08 / 55.8395.08 / 56.7594.75 / 56.50
Qwen3-30B-A3BNS99.75 / 98.5099.83 / 93.3398.00 / 90.3395.33 / 90.0094.58 / 89.33
AuthorsOpenReview ↗

Response to Reviewer tSkK (Part 2/2)

W2 / Q2 (continued): Implementation and evaluation details.

A: Matched settings. Within each model, all presentation conditions use matched prompt templates, provider configurations, and a 4,096-token output limit. The original-problem baseline was rerun under these settings. Qwen3-30B-A3B denotes Qwen3-30B-A3B-Instruct-2507 with fixed FP8 precision; Qwen3-8B uses non-thinking mode. Noise sets are nested across the 3-, 6-, and 9-item conditions.

Sample sizes. Each original-problem cell in the main table contains 400 strong items; each contextual cell contains 1,200 responses across three scenarios. One API failure in GPT-4.1-mini NS clean context is excluded. Across strong, weak, and control variants, all 155,999 valid responses were annotated, including incorrect, unparsed, and truncated answers.

Shortcut evaluation. We use gpt-5.6-luna with low reasoning effort and the manuscript’s human-cognitive-effort criterion. The judge receives the presented question, options, and full solution, without solver identity, condition labels, or gold answers. SU measures whether the visible method exploits numerical structure to simplify human calculation; correctness, brevity, and claims of using a shortcut are insufficient by themselves.

Judge checks and uncertainty. A stratified 100-response batch/single check yielded 90% agreement. Replacing the presented question with the canonical mathematical question yielded 73/80 agreement in a separate sensitivity check. These are automatic consistency checks, not independent human validation. Paired intervals use family-clustered resampling and condition on the judge labels; not every SU contrast excludes zero.

Interpretation. Added noise also increases input length. The experiment varies extraneous numerical information rather than directly measuring models’ internal cognitive load, and the observed effects are not necessarily monotonic across intermediate noise levels.

W3: Does NS prompting improve answer accuracy?

A: Thank you for pointing this out. Our results show that NS prompting can improve answer accuracy, beyond increasing shortcut usage. Figure 5(a) and Table 6 report these improvements on strong-shortcut problems, with the discussion in lines 353–359. For example, on eight-digit problems:

ModelCoT accuracyNS accuracyGain
GPT-4.1-mini71.2%90.0%+18.8 pp
Qwen3-30B43.2%63.7%+20.5 pp
Qwen3-8B63.7%72.5%+8.8 pp

The category-level results provide further examples. On eight-digit relative-distance problems, GPT-4.1-mini improves from 32% to 84%, and GPT-4o-mini from 46% to 66%, as discussed in lines 371–375 and shown in Figure 5(b). These gains depend on the model and problem structure.

Q3: Do models benefit from NS prompting, or is its main value educational?

A: The accuracy benefits of NS prompting depend on the model, problem structure, digit scale, and comparison prompt (Figure 5 and Tables 5–6).

Our primary motivation is educational. As stated in Section 3.1, lines 214–215, we examine whether models can recognize when a shortcut applies, explain it clearly, avoid misusing it, and generate new practice problems. SenseMath evaluates these capabilities; it does not assume that encouraging shortcuts will always improve answer accuracy.

Thank you for the detailed response and for running the new experiments. In my review I committed to move to an accept rating if the multiple-choice concern was addressed with results, and that the extraneous-load extension would be an interesting addition. You ran both these experiments, and I therefore raise my score from 2 to 3.

I am not going further than 3 because the new results show that the accuracy-gain claims, which are prominent in the abstract, depend on the multiple-choice format and the 512-token budget.

Thank you for your thoughtful follow-up and for acknowledging our additional experiments. We appreciate your willingness to update your rating.

We agree that the accuracy benefits of NS prompting should be interpreted carefully, as they depend on several factors, including the answer format, token budget, model capability, digit scale, and problem category. In particular, our category-level analysis (Figure 5(b) and Appendix D) shows that even under the same evaluation setting, NS prompting can yield substantial improvements for some shortcut categories while providing limited or even negative gains for others.

Our intention is not to claim that NS prompting universally improves accuracy, but rather to investigate when and how LLMs recognize and exploit numerical structure to simplify mathematical reasoning. We will revise the abstract and corresponding discussion to better qualify the accuracy-gain claims and clarify their dependence on evaluation conditions and problem characteristics. We will also incorporate the additional free-form evaluation and extraneous-load experiments into the revised manuscript.

Thank you again for your constructive feedback, which has helped us strengthen the paper.

Reviewer_JaSWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SenseMath, a benchmark for evaluating "number sense" (the ability to use human-like computational shortcuts) in LLMs doing numerical reasoning tasks. Experiments show that LLMs can apply numerical shortcuts when prompted but struggle to use shortcuts without a proper prompt and to generate valid problems that require using shortcuts.

Critical strengths
  1. The paper explores an interesting problem beyond LLM accuracy, as well as a way to understand whether models behave or think in ways similar to humans.
  2. The paper is clearly written, and the benchmarks and experiments are carefully designed.
Critical weaknesses
  1. Number sense is a term that not only defines how an agent/a human solves problems but also their internal representation. The paper mainly explores it in the behavioral sense, with little understanding of how number sense is encoded internally. In fact, some research suggests that, for example, numbers are represented as Fourier features during addition (https://arxiv.org/pdf/2406.03445), suggesting numbers might not even be encoded in a human-like way. If so, it might be less meaningful to say a model has number sense, even if it behaves well in terms of SenseMath. The authors should think about a better way to define number sense, and perhaps to limit it to a behavioral-level evaluation while admitting limitations in internal structure.
  2. Experiments on number sense evaluation might have other confounding factors. For example, in CoT setting, " step-by-step " instructions could decrease the possibility of applying shortcuts (as it is supposed to do things step-by-step without shortcuts); and NS encourages using shortcuts more than CoT. In Applicability Judgment, the model's tendency to answer YES does not indicate poor number sense, as the model may find more creative ways to solve problems using shortcuts, where SenseMath does not think so.
  3. The authors map the three proposed tasks to Bloom’s Taxonomy in cognition, but this connection could be superficial. The current results may reflect differences in instruction-following ability rather than distinct levels of cognitive ability.
Questions for authors

I would appreciate it if the authors could address the points raised in the weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JaSW (Part 1/2)

W1: Evaluating number sense at the behavioral level rather than internal numerical representations.

A: Thank you for raising this point and for highlighting the relevant work on internal numerical representations. We would like to clarify that our focus is on whether models use numerical structure to simplify computation at the behavioral level from a human perspective, rather than how numbers are represented internally by LLMs. This aligns with our educational motivation: LLMs acting as tutors should demonstrate useful strategies and explain when they apply (as recognized by Reviewer kq1f and Reviewer tSkK). We will clarify this distinction in the introduction by revising the discussion of prior work as follows:

“Prior work has examined elementary numerical and arithmetic abilities (Testolin, 2024), as well as the integration of numerical skills into combinatorial problem solving (Rahman & Mishra, 2025). Our focus is whether models exploit numerical structure to simplify computation at the behavioral level from a human perspective. Recognizing a numerical property does not necessarily imply deploying it as a computational strategy. For example, a model may recognize that 98 = 100 − 2, yet still compute 98 × 34 through standard multiplication. Exploiting this relationship instead gives 100 × 34 − 2 × 34 = 3400 − 68 = 3332, replacing the original multiplication with an easy round-number calculation and a small correction. SenseMath examines whether models spontaneously make such strategic use of numerical structure, beyond whether they can obtain the correct answer.”

W2-a: Does step-by-step prompting discourage shortcuts?

A: Step-by-step explanations can also describe shortcut-based solutions. Nevertheless, we appreciate the reviewer’s concern that this wording may influence strategy selection. We therefore evaluated five models on the same 1,200 problems at d = 4 using a neutral prompt, replacing “Show your step-by-step reasoning” with “Explain your solution” while keeping the remaining settings fixed within each model.

Shortcut usage rate (%) across strong, weak, and control items:

ModelNeutral-explainCoTNS
Qwen3-8B33.92%34.58%61.92%
Qwen3-30B-A3B-Instruct-250747.58%41.83%88.58%
Llama-3.1-8B20.92%20.50%36.08%
GPT-4o-mini28.42%28.92%58.67%
GPT-4.1-mini34.00%31.50%76.75%

As shown in the table, the effect of step-by-step wording varies across models. However, NS prompting consistently elicits more shortcut use than neutral prompting across all five models. Thus, the difference between CoT and NS cannot be attributed solely to the step-by-step instruction. We will include this comparison in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer JaSW (Part 2/2)

W2-b: Can YES responses reveal valid alternative shortcuts?

A: We followed up on 249 YES responses from Qwen3-8B, Qwen3-30B-A3B-Instruct-2507, and GPT-4.1-mini using the existing J1 input set, asking each model to demonstrate its proposed method. In our review, none of these follow-up solutions was judged to substantially simplify computation under the paper’s human-computation criterion.

The examples below distinguish a transformation that does not simplify the calculation from one that adds computational burden. The first is an illustrative decomposition; the second is an abridged model response that introduces half-integers and repeatedly expands squares. The comments explain our assessment.

# Example 1: Rewriting without simplifying (illustrative example).
# Splitting both factors produces four partial products and a final sum.
# The round-number notation alone does not remove the computational work.

1642 * 5326
= (1600 + 42)(5300 + 26)
= 1600 * 5300 + 1600 * 26 + 42 * 5300 + 42 * 26
# Example 2: The transformation adds computational burden.
# The model introduces a half-integer midpoint, expands its square, and
# expands another large square before combining the intermediate results.
# We judged these extra calculations not to simplify human computation.

Problem: What is 5883 * 5894?
A. 34674412
B. 34674392
C. 34674422
D. 34674402

Model calculations (abridged):
(5883 + 5894) / 2 = 5888.5
5883 * 5894 = (5888.5 - 5.5)(5888.5 + 5.5)
            = 5888.5^2 - 5.5^2
5888.5^2 = (5888 + 0.5)^2
         = 5888^2 + 2 * 5888 * 0.5 + 0.5^2
5888^2 = (5900 - 12)^2
       = 34,810,000 - 141,600 + 144
       = 34,668,544
5888.5^2 = 34,668,544 + 5888 + 0.25 = 34,674,432.25
34,674,432.25 - 30.25 = 34,674,402
Model final answer: D

W3: Do the tasks assess different capabilities or merely instruction following?

A: Our three tasks assess different capabilities aligned with Bloom’s Taxonomy: applying a strategy, judging its applicability, and creating problems with the intended structure. This organization follows our educational motivation: models supporting number-sense learning need to solve problems, assess when a strategy is appropriate, and generate suitable practice problems.

Instruction following is a shared prerequisite for many LLM tasks, but it is too broad to fully characterize these capabilities. Following an instruction to generate a problem, for example, does not ensure that the problem contains the required numerical structure or admits a valid shortcut. Although instruction following can affect performance, describing the results solely in these terms overlooks the specific mathematical capabilities being evaluated.

Q1: Please address the points raised in the weaknesses section.

A: We have provided explanations and more analysis in response to the weak points and would like to continue the discussion if there are any other questions. Thank you.

Dear Reviewer JaSW,

Thank you again for your valuable feedback. We would like to kindly follow up on our rebuttal. We hope our responses have helped clarify the points you raised. We would greatly appreciate any further feedback you may have, and we would be happy to address any remaining questions or concerns.

Reviewer_kq1fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses the problem of reasoning in models specifically the type of reasoning that not necessarily go step by step through the whole solution in math problems but the type that would essentially use shortcuts that would make the solution shorter and the paper essentially checks whether the models understand such concepts. The authors use cognitive-science-grounded framework to also understand the properties of the models with respect to this problem and provide a benchmark that would have different systematic levels of difficulties for models’ evaluations. authors show that models actually do not coherently show number sense and even when they are told to use such reasoning they over generalize which further shows they do not quite flexibly understand the idea.

Critical strengths

This is Interesting subject and also pretty novel and also a good point of view overall because usually the literature treats shortcuts as things that are not supposed to happen but here the authors try to basically present it also as a medium of delivering math capabilities, which overall basically gives more depth into models either doing reasoning or not because simply using shortcuts might mean both a strength and weakness and this paper does this separation and provides interesting insights for the community about the reasoning capabilities of the models

I also really like how the authors are actually motivating their work in the educational setting where the use of llms as tutors must actually have the property that the authors of the current work are indeed discussing.

The results are also super interesting and insightful. For instance, Figure 3 and 5 suggest that the models are actually using heuristics when the computations might get more difficult, which is interesting and might actually explain why models may or may not be adopting such heuristics or number sense reasonings in different situations

The cognitive science frameworks of cognitive load theory and the Bloom’s taxonomy that authors use are actually nice, frameworks that is enabling the authors to check whether a strategy or a reasoning capability is genuine or not by checking the three methods or components of application, evaluation, and creation in a systematic way considering problem level shortcuts, reasoning level ones and option level ones.

Critical weaknesses

Although overall authors make a good argument for the difference between what the literature has already done and what they do in the lines 097-100, I suggest authors to provide more details and elaborate better what they mean exactly when they distinguish between perceiving numerical properties and reasoning with them using examples and specific use cases that would make their argument a bit more concrete.

The authors say in line 195 that the assumption of literature so far has been that the strategy of models for solving complicated and more simple problems should all be the same or is the same ?? That is a bit of an overgeneralization as the broader literature has already engaged in various methods to actually show the value of heuristics and also models switching between different modes of reasoning for instance system 1 and system 2 reasoning [1]. Can the authors please remove the overgeneralization here and better specify the contribution that they have that is differentiating them and rest of the current literature?

And this might actually relate to a more fundamental question that I have from the authors. Do they suggest we have to look at models reasoning as just two types of reasoning which is procedural and flexible as the authors suggest or it is more of a spectrum ? I’m not saying the authors should answer this as it will definitely be outside the scope of this paper but please note this in the manuscript as the current version is suggesting that you have already opted on the binary perspective.

Another question that I have from the authors is that right now you’re basically having the types of reasoning shortcuts that you’re discussing in this paper as ground truth right, or as something that humans do ? For instance for humans it’s easier to compute 100 * 14 than 98 * 14. But why this should be the case for models ? For instance for the models that are doing the computation, ignoring the reasoning tokens, the computation for both of these would just be a couple of tokens right ? “248×4,012≈250×4,000” I’m not questioning the whole premise of the paper but I think discussing this in the paper (perhaps in the limitations or other parts of the paper where it’s relevant) would be helpful for a reader that would have a similar question and when interpreting this paper.

Kinda relevant to this ? How are we actually explaining the definition of a shortcut to models because to the model it might be number of tokens or just something abstract that might come from the definition of heuristic or number sense in their training but maybe it’s not actually materialized in their reasoning and hence they wouldn’t be able to actually evaluate it either. Checking the prompts mentioned in the appendix I think it might as well be the case that the models are just not understanding what easy calculations are or what is a clever observation is. I think that would help a lot to also try some few-shot prompting counterparts to check the effect of demonstrating such heuristics and reasoning shortcuts than to simply verbally explain them.

[1] Ziabari, A. S., Ghazizadeh, N., Sourati, Z., Karimi-Malekabadi, F., Piray, P., & Dehghani, M. (2025). Reasoning on a spectrum: Aligning llms to system 1 and system 2 thinking. arXiv preprint arXiv:2502.12470.

Questions for authors

Also some questions regarding the claims and numbers for instance in the abstract (“fewer than 40” or “up to 15”) I couldn’t confirm these by looking at the tables in the paper could you please elaborate how you got these numbers?

What’s the effect of more or less reasoning on your results because from what I can see you have only ran the experiments with different models using specified fix reasoning settings.

In the 4.5 you’re showing that models are really terrible in generating such problems that would have these heuristics, but what is the problem by looking at the traces of the models or how do the generated questions actually look like ?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kq1f (Part 1/3)

W1: Distinguishing numerical perception from reasoning with numerical properties.

A: Thank you for pointing this out. We agree that the distinction between recognizing numerical properties and using them in reasoning would be less abstract when a concrete example is provided. We will include the following text in our Introduction Section:

“Prior work has examined elementary numerical and arithmetic abilities (Testolin, 2024), as well as the integration of numerical skills into combinatorial problem solving (Rahman & Mishra, 2025). Our focus is whether models exploit numerical structure to simplify computation. Recognizing a numerical property does not necessarily imply deploying it as a computational strategy. For example, a model may recognize that 98 = 100 − 2, yet still compute 98 × 34 through standard multiplication. Exploiting this relationship instead gives 100 × 34 − 2 × 34 = 3400 − 68 = 3332, replacing the original multiplication with an easy round-number calculation and a small correction. SenseMath examines whether models spontaneously make such strategic use of numerical structure, beyond whether they can obtain the correct answer.”

W2: The fixed-strategy claim overgeneralizes prior work.

A: Thank you for pointing out the overgeneralization and relevant work. We will remove the fixed-strategy claim, incorporate the suggested reference, and revise the passage to clarify our intended distinction:

“Token-efficiency-oriented methods aim to reduce the length or cost of generated reasoning through concise intermediate representations, adaptive reasoning paradigms, and problem-dependent token budgets (Xu et al., 2025; Aytes et al., 2025; Han et al., 2025). Related work also explores aligning models with System 1 and System 2 reasoning styles (Ziabari et al., 2025). Our focus is on a complementary notion of simplification, motivated by human numerical cognition: exploiting numerical structure to replace demanding calculations with easier ones. For example, computing 98 × 34 as 100 × 34 − 2 × 34 uses a round-number calculation and a small correction. Such strategies may introduce additional intermediate steps and require more tokens to explain, yet make the calculation easier and often faster for humans by replacing demanding operations with simpler ones. Our focus is therefore on whether models exploit numerical structure to achieve this form of computational simplification. ”

W3: Is procedural versus flexible reasoning a binary distinction or a spectrum?

A: Thank you for pointing this out. We agree that reasoning can exhibit a spectrum of flexibility. Our use of “procedural” and “flexible” was intended to describe how a problem is solved, rather than to classify a model’s reasoning into two mutually exclusive types. The same model may use different approaches across problems. We acknowledge that our wording may have suggested an overly binary distinction, and we will clarify this point in the manuscript to avoid misleading readers.

W4: Why should shortcuts that simplify human computation also simplify computation for models?

A: Thank you for raising this point. We agree that we made this clarification too late, in Section 4.6, where we state that a shortcut must “simplify computation from a human perspective.” We will make this explicit in the abstract and introduction: our shortcuts are defined from a human numerical cognition perspective and do not necessarily reduce models’ internal computational cost.

W5: Could few-shot demonstrations help models understand and use shortcuts?

A: Thank you for this suggestion. We compared the original NS prompt with NS plus two demonstrations: a strong-shortcut example and a matched control example for each category. We evaluated four models on 2,400 items each, covering all eight categories and strong, weak, and control variants at d = 4 and 8, with a matched 512-token output limit.

ModelAccuracy NS → NS + 2-shotShortcut usage NS → NS + 2-shot
GPT-4o-mini62.04% → 66.58%60.75% → 61.25%
GPT-4.1-mini79.08% → 81.63%73.54% → 77.75%
Qwen3-8B57.38% → 61.88%59.42% → 69.38%
Qwen3-30B-A3B66.46% → 71.29%60.75% → 78.33%

Overall accuracy increases by 2.54–4.83 percentage points, and shortcut usage increases by 0.50–17.58 percentage points across all four models. These results suggest that concrete examples help elicit number-sense strategies and improve answer accuracy. We will add these experiments to the revised manuscript to strengthen the empirical support for our conclusions.

[1] Ziabari, A. S., Ghazizadeh, N., Sourati, Z., Karimi-Malekabadi, F., Piray, P., & Dehghani, M. (2025). Reasoning on a spectrum: Aligning llms to system 1 and system 2 thinking. arXiv preprint arXiv:2502.12470.

AuthorsOpenReview ↗

Response to Reviewer kq1f (Part 2/3)

Q1: Where do “fewer than 40%” and “up to 15” in the abstract come from?

A: Thank you for pointing this out. “Fewer than 40%” refers to Llama-3.1-8B’s shortcut-use rate under CoT in Figure 2. The 15-percentage-point gain refers to GPT-4.1-mini on strong items at d = 16, from 63.0% to 78.0% accuracy (Table 6). We will clarify the scope and source of both numbers in the manuscript.

Q2: What is the effect of enabling or disabling thinking?

A: Thank you for this suggestion. We conducted a thinking-on/off comparison using Qwen3-8B and the original Qwen3-30B-A3B, keeping the model checkpoint and generation settings fixed within each model. We evaluated Apply on four- and eight-digit problems, together with the Judgment and Generation tasks.

The table reports accuracy for Apply, J1, and J2, and the percentage of generated pairs passing all six automatic checks for Generation.

ModelMetricThinking offThinking on
Qwen3-8BApply: CoT82.14%97.38%
Qwen3-8BApply: NS74.49%97.35%
Qwen3-30B-A3BApply: CoT86.29%99.12%
Qwen3-30B-A3BApply: NS79.06%99.42%
Qwen3-8BJ1 accuracy54.28%54.79%
Qwen3-30B-A3BJ1 accuracy58.24%55.68%
Qwen3-8BJ2 accuracy78.33%58.33%
Qwen3-30B-A3BJ2 accuracy67.08%62.08%
Qwen3-8BGeneration1.39%12.50%
Qwen3-30B-A3BGeneration2.08%12.50%

Enabling thinking increases Apply accuracy by 12.83–22.86 % and raises the proportion of generated pairs passing all six automatic checks to 12.50% for both models. However, applicability judgment accuracy changes little or decreases, and shortcut-identification accuracy decreases for both models. Thus, stronger problem-solving performance does not automatically translate into better judgment of when shortcuts apply or whether they have been used.

We also assess shortcut usage in the final answer and in the full visible trace, including thinking. Enabling thinking increases shortcut usage under both CoT and NS, and the full trace shows more shortcut use than the final answer alone. All SU values below are percentages; in thinking-off mode, the final-answer and full-trace values coincide.

ModelPromptOff SUOn final SUOn full SU
Qwen3-8BCoT41.29%46.93%69.82%
Qwen3-8BNS62.46%68.69%82.33%
Qwen3-30B-A3BCoT50.23%53.74%76.10%
Qwen3-30B-A3BNS69.02%70.21%84.60%
AuthorsOpenReview ↗

Response to Reviewer kq1f (Part 3/3)

Q3: What goes wrong when models generate shortcut problems? Can you show examples?

A: Thank you for this suggestion. The examples below illustrate three specific errors. o3 generates a control problem that can still be solved directly using the same distributive-property shortcut as the strong problem. GPT-4.1-mini gives a reasonable rounded estimate but selects an option that is not closest to that estimate or to the exact product. Llama-3.1-8B recognizes a difference of one in its explanation, yet gives an incorrect answer and omits the correct value from all answer options. These examples show failures to construct the intended strong/control contrast and to keep the explanation, answer, and options consistent. The examples are saved final outputs, with our analysis provided in the opening comments.

# o3: The intended strong/control contrast fails.
# Both problems are solved directly using the distributive property.
# Moving the blank does not remove the shortcut; both answers are correct.

Generated strong-shortcut problem
What value fills the blank?  9996 × (7408 + 2150) = 9996 × ___ + 9996 × 2150
(A) 7398
(B) 7408
(C) 7418
(D) 7428
Model answer: 7408
Model explanation:
By the distributive property, 9996*(7408+2150) expands to 9996*7408 +
9996*2150. Therefore the missing term is 7408.

Generated control problem
What value fills the blank?  6235 × (2817 + 5649) = ___ × 2817 + 6235 × 5649
(A) 6225
(B) 6235
(C) 6245
(D) 6255
Model answer: 6235
Model explanation:
Using the distributive property, the first product on the right must have
the same factor 6235 that appears outside the parentheses on the left. Hence
the blank is 6235.

# GPT-4.1-mini: The explanation does not support the selected answer.
# Strong: 23,998,000 (B) is closer to both 24,000,000 and the exact
# product 23,994,999 than the model's choice, 23,988,000 (A).
# Control: B is closest to the exact product, but the stated estimate
# 34,790,000 is closest to D, so the explanation does not justify B.

Generated strong-shortcut problem
Which of the following is the best estimate for 8001 * 2999?
(A) 23988000
(B) 23998000
(C) 23978000
(D) 23968000
Model answer: 23988000
Model explanation:
Estimate 8001 as 8000 and 2999 as 3000. Multiplying 8000 * 3000 =
24,000,000. Since 2999 is slightly less than 3000, the product will be
slightly less than 24,000,000. The closest option to this estimate is
23,988,000.

Generated control problem
Which of the following is the best estimate for 7123 * 4876?
(A) 34720000
(B) 34730000
(C) 34740000
(D) 34750000
Model answer: 34730000
Model explanation:
The numbers 7123 and 4876 do not round neatly to simple multiples that make
mental estimation straightforward. The product is approximately 7,100 *
4,900 = 34,790,000, so the best estimate among the options is 34,730,000.

# Llama-3.1-8B: The answers and options are inconsistent with the
# arithmetic.
# The correct values are 8232 and 8135; neither appears in the options.
# The control also retains the same difference-of-one structure.

Generated strong-shortcut problem
Evaluate: 8231 + 6192 - 6191
(A) 14232
(B) 14231
(C) 14230
(D) 14229
Model answer: 14232
Model explanation:
Notice that 6192 and 6191 are very close, and their difference is 1. This
suggests that the subtraction will result in a small difference, which can
be easily calculated by adding 8231 and 1, then subtracting 6191.

Generated control problem
Evaluate: 8134 + 2198 - 2197
(A) 10335
(B) 10336
(C) 10337
(D) 10338
Model answer: 10335
Model explanation:
This problem requires a straightforward calculation, and number sense does
not provide any shortcuts or simplifications.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We sincerely thank you for your time and effort in coordinating the review process, especially for kindly following up with the reviewers. We also appreciate all reviewers' constructive feedback. During the rebuttal period, we carefully addressed their concerns through detailed clarifications and additional experiments. We are pleased that Reviewer tSkK increased their score from 2 to 3, while Reviewer kq1f acknowledged that most of their concerns had been addressed and maintained a positive score of 3. Reviewer JaSW has not yet provided follow-up feedback.

Below, we summarize our main responses to each reviewer.

Summary of response to Reviewer kq1f (initial score 3, maintained 3 after follow-up)

  • We clarified the distinction between recognizing numerical properties and exploiting them for computational simplification, refined our positioning relative to prior work, and emphasized that SenseMath evaluates human-oriented numerical shortcuts at the behavioral level (See responses to W1–W4).
  • We added few-shot prompting experiments across four models, showing improvements in both accuracy and shortcut usage, demonstrating that concrete examples help elicit number-sense strategies (See response to W5).
  • We added thinking-on/off experiments, showing that stronger reasoning improves problem-solving accuracy but does not consistently improve applicability judgment or shortcut identification (See response to Q2).
  • We clarified the numerical claims and provided concrete examples of problem-generation failures. In our follow-up, we further explained why shortcut use alone is insufficient to assess number sense, motivating our three complementary evaluation tasks at the apply-, judge- and create-level (See responses to Q1, Q3, and follow-up to W4).
  • The reviewer explicitly acknowledged that our responses addressed six concerns (W1, W2, W3, W5, Q2, and Q3) and maintained their positive score.

Summary of response to Reviewer JaSW (initial score 2, no follow-up yet)

  • We clarified that SenseMath evaluates behavioral number sense rather than internal numerical representations, consistent with our educational motivation (See response to W1).
  • We added neutral-prompt experiments across five models, showing that increased shortcut usage under NS prompting cannot be attributed solely to step-by-step instructions (See response to W2-a).
  • We examined 249 YES responses in applicability judgment and found no demonstrated alternative methods that substantially simplify human computation (See response to W2-b).
  • We clarified that our three tasks evaluate complementary mathematical capabilities beyond general instruction following (See response to W3).

Summary of response to Reviewer tSkK (initial score 2, increased to 3 after follow-up)

  • We added free-form answering experiments across three models, showing that models retain substantial accuracy without multiple-choice options, while acknowledging the influence of answer format (See response to W1&Q1).
  • We extended SenseMath with extraneous numerical noise experiments across five models, showing that irrelevant information can hinder shortcut usage even when accuracy remains stable (See response to W2&Q2).
  • We clarified the accuracy benefits of NS prompting with quantitative evidence, while emphasizing their dependence on model, problem structure, and evaluation settings (See responses to W3&Q3).
  • We acknowledged the reviewer's follow-up concerns about evaluation settings and committed to qualifying the accuracy-gain claims and incorporating the new experiments into the revision (See follow-up response).
  • The reviewer explicitly acknowledged that our additional free-form and extraneous-load experiments addressed their major concerns and increased their score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.