acceptodds
Under review as a conference paper at ICLR 2027

Would You Walk to the Car Wash? Salience Bias in LLM Commonsense Reasoning

14 views

Abstract

Despite advances in complex reasoning, large language models (LLMs) can prioritize explicit input conditions over implicit task prerequisites. In everyday commonsense reasoning, this can lead to a failure we term Salience Bias: salient but task-irrelevant details (e.g., numerical values) draw models into computation while they overlook the physical or commonsense prerequisites of the task. A critical open question is whether this failure reflects a genuine gap in commonsense knowledge or merely its suppression under misleading task framing. To investigate this, we construct the SaliTrap Benchmark, comprising 1,145 items in four trap dimensions. Evaluating 12 LLMs, we find substantial vulnerability across the tested models, with higher numerical distractor counts associated with lower trap-avoidance rates and trap recognition not always leading to avoidance. Further probing of sycophantic-compliance cases shows that the relevant commonsense can often be elicited when the original task framing is removed, suggesting a gap between recognizing a constraint and applying it during task execution. Building on this diagnosis, we further show that lightweight, inference-time prompting alone substantially closes the gap without any retraining. Our findings highlight the importance of applying commonsense constraints during task execution, and we release SaliTrap as a testbed for studying this gap.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Salience Bias, Commonsense Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_448tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies salience bias in LLM reasoning: models can get distracted by explicit numerical or procedural details and overlook basic commonsense constraints that make the task infeasible in the first place. The authors introduce SaliTrap, a benchmark with 1,145 examples covering four types of traps, and evaluate 12 LLMs, finding that this failure is widespread and becomes more severe as the amount of distracting numerical information increases. They further distinguish different failure modes, such as completely missing the trap, noticing it only after starting the computation, or recognizing it but still complying.

Critical strengths
  1. The evaluation goes beyond reporting a single accuracy metric. By distinguishing different failure modes such as Hard Fail, CoT Hijacked, and Sycophantic Compliance, the paper provides a more fine-grained view of how models fail, which makes the analysis more comprehensive.

  2. The paper contributes a reasonably large benchmark. SaliTrap contains 1,145 examples across four categories of commonsense traps, providing a useful testbed for the community to further study and evaluate this type of reasoning failure.

Critical weaknesses
  1. Incomplete coverage of related work.
    The related work appears to miss several closely related studies. In particular, prior work has already investigated implicit feasibility constraints and surface-heuristic distraction, false-premise questions, and unreasonable or invalid problem formulations in mathematical reasoning. These works seem directly relevant to the positioning of SaliTrap and should be discussed to better clarify its novelty.

    • Li, Y., Zhang, L., Jiang, T., Krishnan, R., and Padman, R. The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning. 2026.
    • Tang, Y. BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models. 2026.
    • Hu, S., Luo, Y., Wang, H., Cheng, X., Liu, Z., and Sun, M. Won’t Get Fooled Again: Answering Questions with False Premises. ACL 2023.
    • Ma, J., Dai, D., Yuan, Z., et al. Large Language Models Struggle with Unreasonability in Math Problems. AAAI 2026.
    • Liang, H., Qiang, M., Li, Y., et al. MathDebugger: A Benchmark for Type-Aware Error Detection in Synthetic Math Data.
  2. The reliability of the LLM-as-a-Judge evaluation is insufficiently validated.
    The five-way behavioral categorization (e.g., Strict Pass, CoT Hijacked, and Sycophantic Compliance) involves fairly subjective judgments, yet the paper does not report any human validation of the Judge labels or human–LLM agreement. Since many of the paper’s main findings rely on these fine-grained categories, it would be important to manually annotate a representative subset of model responses and report agreement with the LLM Judge to verify that the labels are reliable.

  3. The mitigation experiments are limited to inference-time prompting and do not evaluate capability preservation on normal tasks.
    The paper shows that prompting can substantially improve performance on SaliTrap, but does not investigate whether the failure mode can be mitigated through training. More importantly, an intervention that encourages models to constantly question task premises may introduce over-skepticism or unnecessary refusals on otherwise valid tasks. It would be useful to evaluate training-based mitigation together with matched feasible control tasks, to test whether salience bias can be reduced without degrading normal task-solving ability.

Questions for authors

Could the authors provide several complete examples in the appendix, including the original prompt, the model response, and the corresponding Judge label for each of the five behavioral categories? This would help readers better understand how the scoring rubric is applied in practice and assess whether the distinctions between categories align with common human judgment or may be overly strict in some borderline cases.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 448t

Thank you for the careful review and for suggesting ways to strengthen the related work, validate the Judge, and evaluate mitigation.

W1. Related work and novelty

Thank you for pointing out these closely related studies. We have expanded Section 2.2 to clarify how SaliTrap relates to prior work. In The Model Says Walk, Li et al. use an insightful paired design to test whether models change their decisions when an implicit feasibility constraint is removed. SaliTrap studies salience bias in open-ended task execution. Across four types of commonsense traps, we examine how salient numerical or procedural details can draw models into calculation or planning when a necessary prerequisite is unmet. We distinguish failures to recognize the unmet prerequisite from cases in which the model recognizes it and still proceeds. We have also added citations to BrainBench, FalseQA, Unreasonable Math Problems (UMP), and MathDebugger in Section 2.2.

W2. Human validation of the LLM Judge

Appendix F.3 describes the five-way evaluation protocol. We report a human agreement check here. Three annotators with a computer science background each labeled 80 model responses. Across the 240 comparisons between an individual human label and the LLM Judge’s label, 212 matched, yielding 88.3% raw agreement.

W3. Mitigation and feasible-task performance

Thank you for raising the concern about over-skepticism. Appendix G.7 reports our evaluation of P1 on general QA. On tinyMMLU, P1 left GLM-5.1’s accuracy unchanged (93%) and reduced DeepSeek-V4-Pro’s accuracy by 3 percentage points (from 88% to 85%).

We also fine-tuned Qwen3-8B with LoRA SFT on 140 trap–feasible pairs and evaluated it before and after training on held-out seeds under Control.

MetricBefore LoRA SFTAfter LoRA SFT
Trap avoidance rate (TAR)1.25%70.00%
Correct completion on matched feasible tasks73.91%34.78%
False refusals on matched feasible tasks0.00%32.61%

We report the setup and results in Appendix J.

Question for authors

Appendix I includes one example for each of the five behavioral categories. Each example gives the task prompt, reference explanation, model’s final answer, and Judge label and rationale. The Sycophantic Compliance example also summarizes the relevant part of the model’s reasoning.

Thank you for the response, the human agreement check, and the examples in Appendix I.

Novelty. I appreciate the open-ended task format and the distinction between missing a constraint and recognizing it but proceeding anyway. Li et al. already showed that a minimal hint can help models apply an overlooked feasibility constraint. I still see substantial overlap in the central finding and view SaliTrap’s main contribution as extending the evaluation to open-ended tasks.

SFT. Table 9 reinforces my concern about over-skepticism. LoRA SFT raised trap avoidance from 1.25% to 70.00%, while correct completion on matched feasible tasks fell from 73.91% to 34.78% and false refusals rose from 0% to 32.61%. I find that trade-off too severe for the reported SFT setup to be a satisfactory mitigation.

For these reasons, I stand by my original rating of 2.

AuthorsOpenReview ↗

Clarifying SaliTrap’s Novelty and Mitigation Results

Thank you for the follow-up.

Novelty. Li et al.’s paper makes an important contribution through its paired evaluation of hidden feasibility constraints. SaliTrap differs in two respects.

  1. Research focus. Li et al. show that salient cues can steer models toward actions ruled out by hidden constraints. SaliTrap documents a failure that can occur even after a model explicitly identifies the constraint. The model states that a prerequisite is unmet and still proceeds with the requested calculation or plan.
  2. Task design. Li et al. primarily evaluate how models answer everyday questions about which action to take. SaliTrap focuses on reasoning tasks that require calculations or multistep planning. This lets us assess whether a constraint the model explicitly recognizes guides its subsequent reasoning.

Mitigation and over-skepticism. We have not claimed that SFT can reliably mitigate salience bias. We added the experiment in Table 9 at your suggestion to test whether LoRA SFT could improve trap avoidance while preserving performance on matched feasible tasks. We agree with your assessment of the trade-off.

Our positive mitigation result concerns inference-time prompting. P1 improved trap avoidance on GLM-5.1 and DeepSeek-V4-Pro. On tinyMMLU (Appendix G.7), GLM-5.1 scored 93% under both conditions, and DeepSeek-V4-Pro’s accuracy decreased by 3 percentage points, from 88% to 85%. P1 thus largely preserved general QA accuracy in the two models tested.

If you have any further questions, we would be happy to discuss them.

Reviewer_JVzdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that LLMs can fail to recognize basic commonsense prerequisites despite their ability to perform complex reasoning and mathematical calculations. They suggest that these failures arise not from a lack of commonsense knowledge, but from failing to recognize or apply existing knowledge when handling complex numerical or procedural conditions. To investigate this issue, they introduce SaliTrap, a benchmark consisting of 1,145 examples, and show that even frontier models struggle to recognize and avoid commonsense traps when numerical distractions are introduced. They also identify sycophantic compliance, where models continue to follow infeasible user requests despite recognizing the underlying problems. Finally, they show that simple inference-time prompting that encourages feasibility checks can help mitigate these failures.

Critical strengths
  1. Commonsense reasoning has long been an important research topic in NLP and language modeling. Although recent LLMs have achieved strong performance on many benchmarks and acquired extensive knowledge through large-scale pretraining, reliably applying commonsense knowledge in context remains challenging. As the authors also emphasize, there remains a substantial gap between possessing commonsense knowledge and being able to retrieve and apply it when needed. The authors further investigate this issue in everyday task scenarios, addressing a timely and important research problem with clear motivation and a systematically constructed benchmark.

  2. The paper is well-organized, allowing even readers unfamiliar with commonsense reasoning research to easily follow the main motivation and overall flow. The explanations are clear, and the examples effectively illustrate the key issues throughout the paper.

  3. The experimental setup and analysis cover several important aspects of the problem, and the results provide useful empirical evidence of the identified failure modes. Beyond analyzing these failures, the authors also conduct prompt-level intervention experiments, suggesting that some failures may arise from difficulties in retrieving or applying existing knowledge rather than a lack of knowledge itself. These findings highlight knowledge elicitation as a potentially important bottleneck and a promising direction for future improvements.

  4. The authors systematically benchmark and analyze LLM failures to recognize and apply implicit commonsense preconditions across four trap dimensions and 12 models, while distinguishing different types of failure behaviors. Overall, the paper presents interesting and valuable findings that could serve as a useful foundation for future research.

Critical weaknesses
  1. In Section 3.3.1 (Stage 1: Seed Generation and Scaling), the description of seed construction is somewhat unclear. Although the authors mention that experts create initial prototype seeds, the actual construction process and selection criteria are described rather abstractly. A more detailed explanation would improve the clarity and reproducibility of the benchmark.

  2. The paper lacks sufficient justification for several design choices, including the seed acceptance criteria, filtering methods, use of Jaccard similarity, and selection of five behavioral labels. While the function of each component is explained, it remains unclear why these choices are appropriate and what advantages they offer over alternatives.

  3. I would like to see more quantitative analysis and error analysis of the dataset construction process across Stages 1, 2, and 3. For example, acceptance and rejection rates, checker reliability, quality changes after rewriting, and major error types could be reported. I believe that analyzing findings from the construction process itself, beyond the final benchmark results, would further strengthen the paper.

  4. Adversarial benchmark construction is a reasonable and widely adopted approach. However, it remains unclear whether the constructed questions are sufficiently natural and representative of real-world user queries. Although an LLM-based Naturalness Checker is used, additional validation through independent human evaluation or comparison with actual user queries would strengthen the benchmark's real-world validity.

Questions for authors

Could the authors further clarify the independence of the evaluation? All four Solvers used during dataset construction are also included in the final evaluation in Section 4, which may introduce selection bias into the results. How do the authors address this potential bias? For example, could the authors compare the performance of models used during dataset construction with those not involved in the process?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JVzd

Thank you for the careful review and for suggesting ways to make the benchmark construction and evaluation clearer.

W1. Expert-authored prototype seeds

Section 3.3.1 and Appendix E.1 describe how the expert-authored prototypes are used for seed expansion. To clarify how we chose them, our experts first wrote candidate scenarios by hand and tried turning each scenario into a concrete benchmark question. Other experts reviewed the naturalness of the proposed scenarios. We also tested whether questions derived from each prototype could elicit a failure from an LLM. We retained six hand-written prototypes that met both criteria and used them as examples for seed expansion.

W2. Rationale for the construction and labeling choices

Sections 3.1 and 3.3.1–3.3.2 describe the task criteria, deduplication filters, and behavioral labels. The seed checks ensure that an item contains a clear commonsense contradiction, that the requested action depends on it, and that a solver cannot sidestep it by answering an ordinary subtask. Naturalness screening helps preserve the framing of an everyday request. The deduplication filters catch overlap in scenario names, prompt wording, trap-core wording, and tool–object–action triples. Character-level Jaccard is a simple, inspectable lexical check. Normalizing whitespace and numerals prevents minor formatting or value changes from masking repeated wording.

The five labels distinguish whether and when the model notices the trap, whether it proceeds after noticing it, and whether a refusal demonstrates understanding. These distinctions let us separate missing the trap, recognizing it late, and recognizing it early but proceeding anyway. Appendix I gives an example of each category with the Judge's rationale.

W3. Construction statistics and error analysis

Section 3.3 and Appendix E describe the construction pipeline. We report the corresponding counts here. Stage 1 generated 1,000 candidate seeds and retained 956 after deduplication (95.6%). For each retained seed, the original question and three additional questions entered Stage 2, for 3,824 initial candidates. The two main reasons for screening out candidates were insufficient naturalness (15.64%) and the absence of a target failure in the Solver's response (30.49%). Both percentages use all 3,824 initial Stage 2 candidates as the denominator. The released benchmark contains 1,145 items from 854 distinct seeds.

For the five-way response Judge described in Appendix F.3, three annotators with a computer science background each independently labeled 80 model responses. The Judge matched 212 of the 240 individual human labels (88.3% raw agreement). This audit covers the response labels.

W4. Independent assessment of question naturalness

We report a separate human assessment of question naturalness here. We randomly sampled 20 released items from each of the four trap dimensions, for 80 items in total. The same three annotators independently rated the naturalness of every question on a 1–5 scale. The mean across all 240 ratings was 4.325/5. This audit evaluates how naturally the questions read. Representativeness would require a comparison with actual user queries.

Question for authors

We agree that selecting items based on construction-Solver responses can affect the evaluation. Appendix E lists the four construction Solvers (Claude-Opus-4.7, GPT-5.5, DeepSeek-R1, and Gemini-2.5-Pro). We report the group comparison here using the original cohort listed in Appendix F.1 and the revised cohort in Section 4.1.2. In the original cohort, the unweighted mean Trap Avoidance Rate was 38.0% for the four construction Solvers and 23.8% for the eight other models. In the revised cohort, the corresponding means are 41.4% for the three remaining construction Solvers and 28.2% for the nine other models. These comparisons show that the difficulty extends beyond the construction Solver pool.

Thank you for the response. The clarifications on expert-written seeds and the additional error analysis have addressed many of my concerns. However, I still find the justification for using Jaccard similarity insufficient, and my concern about potential selection bias remains. I really appreciate the paper's contribution to commonsense reasoning research. With the rise of increasingly powerful LLMs, it has become harder to find new research directions in this field. I think this paper raises an interesting problem and shows promising directions for future work. I would like to give the authors credit for this contribution.

Considering the remaining concerns and the comments from other reviewers, I have decided to keep my original score.

Thank you for the thoughtful follow-up and for recognizing the contribution of our work. We appreciate that the clarifications addressed many of your concerns.

If you have any further questions, we would be happy to discuss them.

Best wishes!

Reviewer_5gAVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

the paper investigates a failure mode in LLM commonsense reasoning in which models focus on explicit numerical or procedural details while overlooking implicit physical or real-world prerequisites. They refer to this behaviour as salience bias. The papers introduces SaliTrap, a benchmark of 1,145 synthetically generated questions covering four categories: missing prerequisites, environmental mismatches, temporal or physiological violations, and rule mismatches. The dataset is constructed through expert-authored seed examples, LLM-based generation, automated validation, and iterative adversarial refinement.

Critical strengths

The paper introduces a datasets of ~1k synthetic samples, covering different scenarios, for measuring commonsense reasoning failures in LLMs. The evaluation includes 12 models.

Critical weaknesses
  1. the paper does not cite the origins of its central car-wash example (Mastodon post by Kévin andanalysis by Opper AI). Furthermore, the authors do not cite Li et al. (2026), The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning, which investigates a closely related problem, introduces a 500-item benchmark and studies prompting-based mitigation. This earlier work raises questions about novelty of proposed approach and findings.

  2. the paper attributes the observed failures to the suppression of existing commonsense knowledge by salient information. But, the experiments primarily demonstrate correlations and changes in model behavior under different prompting conditions rather than establishing this causal explanation. demonstrating that models can recognize the same premise when explicitly asked does not explain why models initially fail to recognize it

  3. Although the paper employ multiple LLM-based checkers and repeated evaluations, there is no clearly documented independent human evaluation of the final dataset or assessment of agreement between human annotators and automated judges. The number of expert-authored initial seeds is also not reported. These omissions make it difficult to assess the reliability, diversity, and correctness of the 1,145 generated questions and their ground-truth explanations.

  4. The paper describes the evaluated models as state-of-the-art, yet excludes newer models available well before submission. For example, Gemini 3.1 Pro was released in Feb 2026, while the paper evaluates Gemini 2.5 Pro. Consequently, the results do not adequately establish how prevalent these failures are among contemporary frontier models that were available several months before submission deadline.

  5. The paper labels responses that recognize an infeasible premise but continue executing the task as "Sycophantic Compliance." However, this behavior does not necessarily demonstrate sycophancy and could instead reflect over-compliance or inconsistent application of constraints. Further, the proposed Sycophancy Index excludes successful responses that correctly recognize the trap, despite being interpreted as a compliance rate conditional on trap awareness. This makes the terminology and interpretation of the metric questionable.

  6. The paper reports that trap avoidance decreases as the number of numerical distractors increases, but does not establish this relationship through controlled, matched comparisons that vary distractor counts while keeping the underlying task fixed. Differences in question complexity, length, or difficulty could contribute to the observed trends

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5gAV (Part 1)

Thank you for the careful review and for identifying ways to clarify the paper's contribution, evaluation, and interpretation.

W1. Attribution and novelty

Thank you for pointing us to these sources. We cite Kévin’s original car-wash question and Opper AI’s follow-up evaluation in Section 1. We have also added a discussion of The Model Says Walk in Section 2.2. Li et al. study heuristic override, in which a surface cue favors an otherwise reasonable choice that conflicts with an implicit feasibility constraint. Their controlled pairs offer a particularly insightful test of whether the model’s decision changes when that constraint is removed.

SaliTrap studies salience bias in open-ended task execution. Our tasks have an unmet prerequisite and contain salient but task-irrelevant numerical or procedural details that invite calculation or planning. This setup lets us examine whether models check feasibility before proceeding and whether they act on constraints they have already recognized.

W2. Interpreting the re-elicitation results

In the re-elicitation experiment (Section 5.3 and Appendix G.5), we selected cases in which the model had already identified the key constraint and still proceeded with the task. When we removed the original task framing and asked directly about the relevant commonsense knowledge, each of the four models gave a correct explanation in over 90% of its selected cases. This shows that the relevant knowledge can often be elicited by changing the question.

We report an exploratory single-item analysis with Qwen3-8B here. The item asks how to organize communal television viewing during a power outage when neither mains nor backup power is available. With the feasibility prompt, the model explicitly concluded that the plan could not be carried out under the stated conditions. Compared with a neutral prefix, the feasibility prompt increased the mean attention per token to the power-supply cue by 9.8% and decreased it for the other numeric tokens by 15.6%, measured just before the first answer token was generated.

We interpret these observations as evidence that task framing can affect whether models apply relevant commonsense knowledge, while the cause of the original failures remains open.

W3. Prototype seeds and human validation

Section 3.3.1 describes how we expand expert-authored prototypes. The starting set comprised six manually authored seeds. To assess the reliability of the five-way behavioral categorization described in Appendix F.3, three annotators with a computer science background each labeled 80 model responses. Across the 240 comparisons between an individual human label and the LLM Judge’s label, 212 matched, yielding 88.3% raw agreement.

We also randomly sampled 20 released questions from each of the four trap dimensions, for 80 questions in total. The same three annotators independently rated the wording of every question for naturalness on a 1–5 scale. The mean across all 240 ratings was 4.325/5. These human checks cover response-label agreement and question wording. The construction checks for whether each question preserves the intended trap and aligns with its reference explanation are described in Section 3.3.2 and Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer 5gAV (Part 2)

W4. Model coverage

Thank you for pointing out this gap in model coverage. We evaluated Gemini 3.1 Pro and Kimi K3 on SaliTrap and included them in the model roster in Section 4.1.2 and the main results in Section 4.2 (Tables 1 and 2). Their overall results are shown below.

ModelTAR (%)HFR (%)SCR (%)SI (%)
Gemini 3.1 Pro43.244.512.356.4
Kimi K344.528.526.676.4

W5. Sycophantic Compliance terminology and interpretation of SI

Thank you for raising this distinction. We agree that over-compliance describes the behavior we observe. We use Sycophantic Compliance as an operational label for a specific form of over-compliance: the response explicitly recognizes that a prerequisite is unmet, yet continues to fulfill the request as though that prerequisite held. We use sycophantic in this behavioral sense, to describe how the response continues to follow the user’s framing despite having identified the conflict. This distinction matters for SaliTrap because a response can recognize the relevant constraint and still fail to apply it.

We agree that Strict Pass also demonstrates trap awareness and that our original description of SI as the compliance rate given trap awareness was too broad. SI was designed to compare two failure modes that both show trap awareness. Sycophantic Compliance captures responses that recognize the trap early and still proceed, while CoT Hijacked captures responses that engage with the distractors before recognizing the trap. SI measures the share of these failures assigned to Sycophantic Compliance. Strict Pass is a successful outcome and contributes to TAR. Section 4.1.1 and Appendix A now define this scope explicitly. SI is computed as SC / (SC + CoT).

W6. Within-item comparison of numerical distractor counts

Thank you for suggesting a within-item comparison. We used all 19 benchmark items whose released prompts contain exactly two Arabic-numeral expressions. Four of these prompts are visibly truncated in the public release. For each item, we used Claude-Opus-4.6 to add one numerical detail at a time, producing variants with three to six numerical expressions.

Across the five conditions, DeepSeek-V4-Pro successfully avoided the trap on 4, 4, 4, 3, and 2 of the 19 items, respectively. GPT-5.5 succeeded on 11, 11, 9, 9, and 8 items. Both models show the same overall direction as the trend in the main paper. The comparison uses the same base items at each count, while the added details can also change prompt length and content. We report the experiment in Appendix G.4 (Figure 11).

Than you for the response. I have read the revised paper and the rebuttal and my concerns still remain especially around novelty.

  • It's unclear what the paper adds in relation to prior work, including the ones raised by Reviewer 448t. It appears to study the same problem under the new label "salience bias,". The paper doesn't seem to provide new information about the problem beyond what previous studies have already established.
  • The proposed benchmark incrementally extends prior work, but the paper does not clearly explain/demonstrate what it measures that previous benchmarks do not.
  • There is inconsistency between the response and revised paper. The response acknowledges that the cause of the original failures remains open, but the introduction, contribution (iv), and conclusion still assert that the failures constitute "a failure of elicitation, not competence," are "fundamentally a failure of knowledge suppression," and "stem from suppression rather than absence of commonsense knowledge."
  • I could not find the six-seed count or the human-validation results reported in the rebuttal in the revised paper.

Thank you for carefully reading the revision and for identifying these remaining issues. We would like to clarify the additional information provided by SaliTrap and acknowledge two shortcomings in our revision.

  1. We agree that prior work, including Li et al.’s insightful study, has already established that surface cues can override implicit feasibility constraints. We do not claim priority for this broad observation. SaliTrap’s contribution is a more detailed evaluation of how these failures unfold during open-ended calculation and planning. In particular, it distinguishes a model that never identifies the constraint, one that identifies it only after engaging with the distracting computation, and one that identifies it early but nevertheless proceeds. These distinctions reveal whether recognizing an unmet prerequisite actually governs subsequent task execution—information that a final correctness score alone does not provide. Our additional analyses examine how these behaviors vary with numerical distractor counts and prompting. The within-item experiment supplements the observational density analysis, although its small sample and changes in prompt length and content limit its interpretation. We intend salience bias as a description of the studied behavior, rather than treating the terminology itself as a novelty claim.

  2. You are correct that the categorical statements remaining in the introduction, contribution (iv), and conclusion are stronger than the interpretation given in our rebuttal. We acknowledge this inconsistency. Our supported conclusion is narrower: among the selected cases where models had already recognized the constraint but continued, the relevant commonsense explanation could often be elicited under a different framing. This supports a gap between recognizing and applying a constraint, but does not establish knowledge suppression as the general causal explanation for all benchmark failures. The single-item attention analysis is exploratory and does not resolve this causal question.

  3. You are also correct that the six-seed count and human-validation results were reported in our rebuttal but omitted from the revised manuscript. We apologize for this omission. To make the seed description concrete, the six human-developed prototype scenarios were:

Trap dimensionPrototype scenarios
Missing prerequisiteTransporting water with a perforated spoon
Environmental mismatchRefueling a battery-electric vehicle with gasoline; lighting an ordinary lighter underwater
Temporal/physiological violationBuying an umbrella today to prevent yesterday’s rain; attempting to cook food in the stomach with boiling water
Rule mismatchSending a photograph of cash as a monetary deposit
  1. For human validation, three annotators with a computer science background independently labeled the same 80 model responses. Their labels agreed with the LLM Judge in 212 of 240 comparisons (88.3% raw agreement). Separately, they each rated all 80 sampled questions—20 randomly selected from each dimension—for naturalness on a 1–5 scale, yielding a mean of 4.325/5 across 240 ratings. These checks assess label agreement and perceived naturalness; they do not establish representativeness of real-world query frequencies.

We have incorporated all the above clarifications and additional information into the revised manuscript.

If you have any further questions, we would be happy to discuss them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.