acceptodds
Under review as a conference paper at ICLR 2027

Measuring and Reducing Over-Reliance on Convenient Evidence in LLM Agents

Abstract

LLM agents often over-rely on easily inferable information, committing to convenient sources or hints without verifying them. This gullibility is often unrepresented in benchmarks where logic steps are well defined, but abundant in everyday life. We attempt to measure this phenomenon by introducing GulliBench: a set of 50 simple data tasks, each with a planted wrong shortcut and a correct answer that must be derived from the underlying records. The tasks are designed to be well within the coding and reasoning capabilities of frontier models, save for the over-reliance traps. Of the 23 models we tested, even the best one fails on 39% of its attempts, and the other 22 fail on more than half of theirs. We conjecture that this gullibility reflects a behaviour that general capability does not reliably produce, and that it can be learned. By post-training a 35B-parameter LLM with RL on a different set of tasks that expose these contradictions, we improve its accuracy on the benchmark from 3% to 31%. On third-party data-analysis and terminal benchmarks the post-trained model checks its work more often, but this is barely reflected in their scores, and we show through case studies why such benchmarks rarely reward it.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, over-reliance, benchmark, verification, reinforcement learning, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_kktAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study the tendency of LLM agents to over-rely on convenient but potentially incorrect evidence instead of verifying them. To systematically investigate this phenomenon, they introduce GulliBench, where each task contains designed shortcuts that produce incorrect answers, while the correct answer can be obtained by examining the underlying data. Through the evaluations on 23 frontier models, the paper finds that even the strongest model has the over-reliance problem. Furthermore, the authors post-train Qwen3.5-35B-A3B on synthetic tasks with shortcuts and find that the over-reliance problem can be mitigated through a simple training process. Overall, this paper contributes a targeted evaluation benchmark, an in-depth investigation of the over-reliance phenomenon, and preliminary evidence that verification-related behaviors can be improved through targeted reinforcement learning.

Critical strengths
  1. Clear motivation and important significance. The paper studies a well-motivated and practically important problem. As LLM agents become increasingly capable, their tendency to over-rely on convenient but unverified evidence can significantly undermine their reliability. Despite its importance, this phenomenon has received relatively limited attention through systematic and controlled studies. In this regard, both the proposed GulliBench benchmark and the findings on mitigating over-reliance offer meaningful contributions to improving the reliability of LLM agents.
  2. A coherent and complete research framework. The paper presents a well-structured and logically coherent investigation of the problem. It first identifies and categorizes representative shortcut types associated with over-reliance, then constructs a benchmark to systematically evaluate and analyze model behaviors. Finally, it gives a mitigation approach to reduce the identified failures. This progression from problem characterization and measurement to behavioral analysis and mitigation forms a complete research narrative.
Critical weaknesses
  1. Poor academic writing and lack of a coherent narrative. This paper's main weakness lies in its poor writing. Several passages contain unnecessarily long sentences with multiple nested clauses, making the arguments difficult to follow (e.g. the sentence around Line 440 spans three lines and introduces multiple conditions and qualifications, making its logical structure unnecessarily complicated). More broadly, the paper often reads like a chronological research log rather than a well-structured academic manuscript. For instance, around Line 184, the authors explicitly document the date of an intermediate stage in their experimental process. In my view, an academic paper should prioritize a clear and systematic presentation of the research problem and limitations of existing approaches, the core principles underlying the proposed methodology and experimental design, and the evidence supporting the conclusions, rather than extensively documenting every detail of the research process. I believe the paper would benefit significantly from restructuring the main text around its central arguments and moving secondary experimental details to the appendix.
  2. Benchmark selection is unfair and unreasonable. Specifically, the authors select the most challenging tasks based on the performance of Claude Fable 5. Although such adversarial selection can be reasonable for constructing a challenging diagnostic benchmark, it raises concerns about the fairness and representativeness of the evaluation. In particular, the paper's finding that "almost all frontier models fail on more than half of their attempts" may largely depend on the difficulty distribution of the selected tasks, rather than reflecting a general and widespread tendency of LLM agents to fall for these shortcuts.
  3. Benchmark design does not adequately support the claimed lack of verification behavior. A more fundamental concern lies in the benchmark design itself, which weakens the paper's conclusions about LLM agents' lack of verification behavior. Specifically, in GulliBench, the underlying assumption is that values recomputed from the original records are always more trustworthy than stored values. Consequently, an agent that simply adopts a fixed strategy of recomputing stored aggregates could achieve a high score without genuinely learning when verification is necessary or how to evaluate and reconcile conflicting evidence in realistic scenarios. Therefore, strong performance on GulliBench does not necessarily demonstrate a general ability to verify evidence, and failures on this benchmark may not be sufficient to support the broader claim that LLM agents lack such an ability.
Questions for authors
  1. Could the authors provide model performance on all 93 eligible task pairs, or at least a more systematic comparison between the selected and excluded tasks?
  2. Could the authors provide additional experiments to distinguish between the following two behaviors for both the evaluated frontier models and the post-trained model? (1) systematically distrusting stored values and recomputing them from underlying records, and (2) selectively verifying information based on the reliability of the available evidence?
  3. Could the authors clarify why the transaction records should be considered more authoritative than the stored account balances in Figure 1? Neither the question nor the provided data explicitly establishes which source is more up-to-date or complete. For instance, it is also possible that the account balances have been updated while the transaction records have not yet been synchronized. Without additional assumptions about data completeness or source reliability, how can the intended answer be considered unambiguously correct?
  4. Could the authors provide stronger justification for the specific structural choices in GulliBench, particularly the use of two tables and two-step tasks? Although Section 2.4 provides some intuition for the two-step design, it remains unclear why these design choices are necessary for measuring over-reliance.
  5. In Section 2.3, the authors state that, in the table format, the agent "must do any filtering or arithmetic itself." However, it appears that agents in the document format also need to perform similar computations, whereas the SQL format allows these operations to be executed directly by the database. Could the authors clarify the computational capabilities available in each format?
  6. Section 4.1 states that the training and evaluation sets use disjoint shortcut types, while also acknowledging that some training shortcuts share underlying patterns with evaluation shortcuts, such as stale aggregates. I think these two things are somewhat contradictory. Could the authors clarify it more clearly?
  7. Could the authors provide an analysis of the inference-time efficiency of the post-trained model compared with the base model? Since increased verification may require additional computation and tool interactions, it would be valuable to understand whether the improvements in reliability come at a substantial efficiency cost.
3 comments

We are very grateful for such a thorough and thoughtful review. Each question pointed to a real gap in how we presented our work, and we have revised the paper substantially in response.

W1 (writing). We fully agree, and thank the reviewer for the concrete examples. We documented the process in detail, including the date at line 184, because a benchmark selected against a model invites the worry that it was tuned after the fact, and we wanted every step to be checkable. The reviewer is right that this belongs in an appendix. We have split the sentence at line 440, moved the process detail of §2.4 and §4.2 to the appendix, and added a diagram of the task structure (Fig. 3). We keep the date only to show that ten of the 24 entries were released after the selection was fixed, and now say so.

W2 and Q1 (selection). We agree the finding should not depend on the selection. As requested, we ran 22 of the 24 entries on all 93 eligible pairs (App. B gives every entry).

Table 1. Pass rates of 22 entries on the 50 kept pairs, the 43 excluded pairs and all 93, how many entries fail more than half their attempts, and how closely each set ranks the models like the kept pairs (Spearman).

MeasureKept 50Excluded 43All 93
Mean pass rate19.8%43.4%30.7%
Entries below 50%21 of 2214 of 2217 of 22
Rank correlation with kept10.850.94

The selection made the benchmark harder, as intended, but the finding holds without it. On all 93 pairs, 17 of the 22 entries fail more than half of their attempts, and the order is nearly the same, with the same top three (Fable 5.1, Fable 5, Opus 5). The selection does not favour Fable 5, since the excluded pairs are easier mostly for its own family (+43 points against +18). We also generated the 100 candidates before running any model, so the selection did not shape the tasks.

W3 and Q2 (recomputing versus selective verification). We agree, and we have tightened our claims (§6). An agent that always recomputes would score well, so a high score cannot show selective verification. Two results separate the behaviours. For frontier models, 68% of failures never computed the contradicting value (Q3), so they neither distrusted the stored value systematically nor verified it selectively. For the post-trained model, a recompute rule has little to act on in §5's third-party benchmarks, yet it tests its premises more often, even on Terminal-Bench 2.1, which has no stored values (8.9% to 22.2%). When a check confirms a value it keeps it, and when the check finds a problem it changes course in 68% to 100% of cases.

Q3 (why the records are authoritative). We thank the reviewer for this important question. The reviewer is right that, in general, records need not be more reliable than a stored value. We address it in three ways. First, each task shows which source is authoritative. For every other entity the stored value equals the value recomputed from its records (200 of 200), and only the shortcut's entity differs. Second, most failures never reach this choice. In 68% of the 14,019 failed attempts that reported the stored value, the agent never computed the contradicting value, and in 28% it never read the records. Third, when the convention is stated, models follow it. Told to reconcile from the records, with the tables in its prompt, DeepSeek-V4 Flash passes 87% of runs, against 16% without (App. D), and Fable 5.1 and GPT-6 Astra report the stored value in under 20% of attempts in which they had obtained the correct one. GulliBench thus measures whether an agent checks, not which source it prefers. We have added the first point to the caption of Fig. 1.

Q4 (two tables, two steps). Two tables are the smallest setting in which a derived value and its source coexist. Two steps make the first result an input the agent can check or trust, which one step cannot. Fig. 3 now shows this.

Q5 (formats). Thank you, the reviewer is right, and §2.3 is corrected. The table format offers schema, count_rows, page_rows and read_row, documents offers list_files, grep and read_file, and SQL offers schema and a read-only query. No format executes code, so only SQL computes for the agent (App. A.2).

Q6 (disjoint types, shared patterns). Thank you for pointing out this unclear wording. No evaluation shortcut type appears in training, but both sets draw on the same five families (e.g. stale stored values, 7 training and 8 evaluation types). The first rules out memorising GulliBench's shortcuts, and the second is why we test on third-party benchmarks (§5). §4.1 now says this.

Q7 (efficiency). Median steps per attempt go from 6 to 9 on GulliBench, 14 to 15 on DABstep and 5 to 6 on BIRD, and do not rise on coding (Terminal-Bench 2.1 17.5 to 17.5, SWE-bench Pro 76 to 66). The cost is a few steps on data tasks (App. K).

Thank you for the authors' feedback. I have reread the your revised manuscript, and it is now much clearer. However, this suggests that the authors are capable of write a good paper, but their initial submission did not reflect the same level of care and effort. I therefore do not think the authors demonstrated an appropriate level of commitment to the quality of their work in the original submission, and I will maintain my original score.

We thank the reviewer for rereading the revised manuscript, and we are glad it is now much clearer. We regret that the original submission did not present the work as clearly as it should have, and the reviewer's detailed comments are a large part of why the revision improved. If any concern about the method, the evidence or the claims remains, we would welcome the chance to address it before the discussion closes. Otherwise, we would be grateful if the score could reflect the paper as revised.

Reviewer_VwXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies over-reliance in LLM agents: committing to a convenient but wrong source, such as a stale stored value, instead of checking it against the underlying records. To measure this, the authors introduce GulliBench, a benchmark of 50 deliberately easy two-step data tasks in which the data offers a convenient but wrong route alongside the correct one. Across 23 frontier models, the best passes 61% of attempts and most pass far fewer, with pass rates only loosely tracking general capability. The authors conjecture that checking is a learnable behaviour and post-train a 35B open model with reinforcement learning on tasks with different shortcut types, raising its pass rate from 3.2% to 31.1%. The post-trained model also checks its work more often on three of four third-party benchmarks, though their scores barely change, which the authors explain through case studies.

Critical strengths

The problem setup is clean. Over-reliance is widespread even among frontier models so the model bias is prominent and worth studying. Code and data are released.

Critical weaknesses

Four-benchmark evaluation feels disconnected from the paper’s general goal. They contain no planted shortcuts, so they can’t measure over-reliance.

Experiment setup issues: 1) Selecting tasks adversarially against one model (Claude Fable 5) is problematic. 2) The evaluation set is too small. 3) Also three shortcuts (Stale rollup total, Stored compound total, Stored converted amount) make up most of the pairs, and they are among the hardest. This is likely a side effect of the adversarial selection, which kept the pairs Fable 5 failed most. The authors should explain here. 4) The training tasks are close to the evaluation tasks, which makes the performance gain expected.

Questions for authors

Do we have a failure analysis to tell us whether an agent never read the underlying records, read them without computing the total, or noticed the mismatch and deliberately trusted the stored value. I wanted to see a better failure classification , e.g., whether the records were accessed.

1 comment

We sincerely thank the reviewer for a careful review. The question on failure classification was especially helpful and led to a new analysis that strengthens the paper.

W (the four benchmarks). The reviewer is right that these benchmarks contain no planted shortcuts and so cannot measure over-reliance directly. To our knowledge no existing benchmark does. We use them to test whether the learned behaviour appears on tasks we did not build, in the same kinds of work. This also acts as a control. The post-trained model tests its premises more often on BIRD, yet its BIRD score does not rise (−2.1 points), so its GulliBench gain comes from checking more, not from better SQL skills. Where third-party tasks do contain GulliBench's pattern, the gain shows in the score. On 15 such tasks found in public benchmarks, the post-trained model gains 11.3 points (CI [+3.3, +20.0], new App. J).

Setup 1 (selection against Fable 5). Thank you for raising this. We generated 100 candidate pairs before running any model on them, and only then kept the 50 that Fable 5 failed most, so the selection chose among existing tasks and did not shape them. Selecting against strong models is also common practice. Humanity's Last Exam rejects questions frontier models answer correctly, and Terminal-Bench 3 and Terminal-Bench Science set aside candidate tasks that frontier models solve easily. To measure the effect directly, we ran 22 of the 24 entries on all 93 screened pairs (App. B).

Table 1. Pass rates of 22 entries on the 50 kept pairs, the 43 excluded pairs and all 93, how many entries fail more than half their attempts, and how closely each set ranks the models like the kept pairs (Spearman).

MeasureKept 50Excluded 43All 93
Mean pass rate19.8%43.4%30.7%
Entries below 50%21 of 2214 of 2217 of 22
Rank correlation with kept10.850.94

The selection made the benchmark harder, as intended, but most models still fail without it, with the same top three. It does not favour Fable 5's family, since the excluded pairs are easier mostly for the Claude models (+43 points against +18 for the others). Ten of the 24 entries, including Fable 5.1, the best, were released after the selection was fixed, and 93.6% of failed attempts give exactly a planted shortcut answer, which a selection on pass rate cannot produce.

Setup 2 (size). We agree the set is small by training standards. It is comparable to recent agent benchmarks such as Terminal-Bench 4.0 (66 tasks), FrontierSWE v2 (34) and DeepSWE (113), and our confidence intervals resample the 50 pairs.

Setup 3 (three shortcuts). This observation is correct, and we now explain it in §2.4. The concentration comes from selecting against Fable 5. It does not drive the finding. On the other 21 pairs the 24 frontier entries pass 32% of attempts on average. With every aggregation shortcut weighted equally (App. F.6), scores rise but most models still fail.

Table 2. Pass rates on the released set and with each of the ten aggregation shortcuts weighted equally.

MeasureAs releasedShortcuts weighted equally
Mean pass rate, 24 frontier entries19.3%25.7%
Entries below 50%23 of 2419 of 24
Claude Fable 5.1 (best)61.3%68.5%
Qwen base → post-trained3.2% → 31.1%5.0% → 45.0%

The model order is nearly the same (rank correlation 0.90).

Setup 4 (training close to evaluation). We agree a GulliBench gain is expected to some degree. No shortcut type or data used in evaluation appears in training, but the sets share families of types, which §4.1 now states. This is why we test on third-party benchmarks (W above).

Q (failure classes). Thank you for this suggestion. We classified all 14,019 failed attempts that reported the stored value.

Table 3. Failed attempts that reported the stored value, split by whether the agent read the records behind it and whether it computed the correct value.

AttemptsnRecords not readRead, value not computedCorrect value in view
All entries14,01927.7%40.3%32.0%
Base Qwen55855.0%20.3%24.7%
Post-trained Qwen31124.4%25.1%50.5%

In 68% of failures the agent never computed the value that contradicts the stored one, so most failures are not a deliberate choice between sources (new §3 text and App. F.7).

Reviewer_vLpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the tendency of LLM agents to over-rely on convenient but potentially incorrect information without independently verifying it. The authors introduce GulliBench, a benchmark comprising 50 simple two-step data tasks presented in three formats (tables, documents, and SQL), where misleading shortcuts compete with correct answers derivable from underlying records. Evaluating 23 frontier models reveals substantial susceptibility to these shortcuts, with the best model achieving only 61.3% accuracy and 93.6% of failed attempts producing shortcut answers. To mitigate this behavior, the authors post-train Qwen3.5-35B-A3B using reinforcement learning on tasks with different shortcut types, improving its GulliBench accuracy from 3.2% to 31.1%. Additional evaluations on four third-party data-analysis and coding benchmarks show increased verification behavior on three of them, although task performance improves only modestly. These findings highlight over-reliance as an important agent reliability issue and suggest that verification behavior can be improved through targeted training, while its broader practical benefits remain less conclusive.

Critical strengths
  1. GulliBench uses deliberately simple tasks with planted incorrect shortcuts and verifiable ground-truth answers, allowing the authors to isolate over-reliance from general task difficulty. The evaluation of 23 frontier models reveals widespread failures, with 93.6% of unsuccessful attempts producing shortcut answers. This provides compelling evidence of a meaningful reliability problem that conventional benchmarks may overlook.

  2. The paper goes beyond identifying the problem by demonstrating that targeted reinforcement learning improves performance from 3.2% to 31.1% on held-out shortcut types. Additional evaluations on four external benchmarks show increased verification behavior, even when task accuracy changes little. These findings provide useful insights into both the potential for improving agent reliability and the limitations of existing evaluation metrics.

Critical weaknesses
  1. GulliBench exclusively contains tasks where convenient shortcuts are incorrect and recomputing from underlying records is the correct strategy. Consequently, the benchmark and training results cannot distinguish whether the model learns to verify evidence appropriately or simply develops a preference for bypassing stored values. This limits the strength of the paper's broader claims about reducing over-reliance.

  2. Although the post-trained model exhibits more verification behavior on external benchmarks, meaningful accuracy improvements are largely absent. Moreover, verification behavior is assessed using LLM judges calibrated on a small set of 52 trajectories, without an independent human validation set. This leaves uncertainty about how reliably the reported behavioral improvements generalize to more diverse agent tasks.

Questions for authors
  1. Since GulliBench always makes the convenient shortcut incorrect, how can we distinguish a model that has learned to verify evidence when necessary from one that simply learns to avoid stored values and recompute from underlying records? What evidence supports interpreting the training improvement as a more general verification behavior rather than adaptation to this particular data pattern?

  2. The verification behavior on external benchmarks is assessed using LLM judges calibrated on 52 human-graded trajectories, which were also used to refine the grading criteria. Could the authors validate judge agreement on a separate, previously unseen set of trajectories, particularly from coding tasks? This would strengthen confidence in the reported behavioral transfer.

  3. On external benchmarks, the post-trained model performs substantially more checks, but task accuracy changes only modestly. Do the authors have evidence showing how often these additional checks actually identify incorrect assumptions or change the final answer? This would help clarify whether the observed improvement reflects meaningful verification rather than simply more frequent checking.

1 comment

We sincerely thank the reviewer for a clear and precise review. The three questions go to the heart of what the paper can claim, and answering them has sharpened both our claims and our evidence.

W1 and Q1 (verification or a preference for recomputing). The reviewer is right that on GulliBench alone a rule of always recomputing and real verification look the same. Passing is therefore necessary but not sufficient evidence of verification, and we have tightened the paper's wording accordingly (§6). GulliBench measures a predisposition not to check, and three results support this reading.

  • When told to reconcile, a weak model succeeds. Told to check stored values against the records, with the tables in its prompt, a low-cost model passes 87% of runs, against 16% in the benchmark without the instruction (App. D).
  • Most failures never compute the correct value. In 68% of failed attempts that submitted the stored value, the model never computed the value that contradicts it, so most failures are not a choice between two sources.
  • The post-trained model tests its premises more, also outside GulliBench. On four third-party benchmarks never used in training, it tests its premises more often, even on Terminal-Bench 2.1, which has no stored values to recompute (8.9% to 22.2% of attempts). When a check confirms a value, it keeps it, and when the check finds a problem, it usually changes course (Q3).

W2 and Q2 (accuracy gains and judge validation). We agree that the accuracy gains outside GulliBench are small, and we believe the main reason is dilution. The pattern GulliBench targets is rare. Searching about 1,600 tasks in public benchmarks, we found only 17 that share it, 13 of them in DABstep. On the 15 we could run (GulliBench-In-The-Wild, new App. J), the post-trained model passes 33.3% of attempts against 22.0% for the base model (+11.3 points, CI [+3.3, +20.0]) and tests a premise in 29.3% against 14.7%. Frontier agents also fall for these tasks, missing the answer in 42% of their public DABstep submissions.

On the judges, we also read the trajectories ourselves, the 52 used to build the definition and every case in §5.3, which we verified on the benchmark's own data. To test the judges beyond that set, one author graded 24 new trajectories from the four benchmarks, coding over-represented, blind to model, outcome and judge grade (new in App. H). The author and the judges agree on whether an attempt tests a premise in 21 of 24 (κ 0.60, Table 1). The three disagreements are all on the two coding benchmarks, which our calibration set covered least, in line with the caution in §5 that our grading may miss checking on coding tasks.

Table 1. Agreement between the author and the judges on whether an attempt tests a premise, on 24 new trajectories.

BenchmarkTrajectoriesAuthor and judges agree
BIRD55
DABstep55
Terminal-Bench 2.165
SWE-bench Pro86
All2421

Q3 (do the checks find something and change the answer?). Thank you for this question, which led to a new analysis. For each of the 2,689 premise-testing attempts, two judges recorded what the check found and what the agent did next.

Table 2. The post-trained model's premise-testing attempts on each benchmark, how often the check found a problem, how often the model then changed course, and the pass-rate gain from changing course over other attempts on the same task.

BenchmarkPremise-testing attemptsCheck finds a problemChanges course when it doesPass gain from changing course, same task
GulliBench60685%57%+19.7 [+13.0, +26.5]
DABstep89176%93%+4.8 [−0.8, +10.5]
BIRD11335%68%−2.8 [−36.1, +30.6]
Terminal-Bench 2.12065%100%too few tasks
SWE-bench Pro4787%71%too few tasks

The change is therefore more than more frequent checking. The checks often find a problem, and the model usually acts on it. Whether acting raises the score depends on whether the task contains a discrepancy to find. On GulliBench, where it always does, changing course adds almost 20 points. On the third-party benchmarks it rarely changes correctness, because the answer was already right, the reference skips the same check, or the task stays out of reach (§5.3). We have added this analysis to §5 and the appendix.

Reviewer_pTnHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "over-reliance" phenomenon in LLM agents. The authors propose GulliBench, covering 61 shortcut types of tasks, which misguides the LLM agents using incorrect numbers. The evaluation set consists of 50 "two-step" tasks for 150 instances. Across several frontier models, the best model passes only 61.3% of attempts, with most errors occurring at the aggregation step. The results demonstrate limited correlation between pass rate and model capabilities like model size and reasoning effort. Then, the authors improve the models' abilities through post-training on these task suites. The authors also point out the limitations of several benchmarks (BIRD, DABstep, Terminal-Bench 2.1, and SWE-bench Pro), which barely encourage such checking behavior and sometimes even penalize it when the reference answer is wrong.

Critical strengths
  1. The paper points out an important risk in real agent workflows. The authors explore the inabilities of agents to give answers under wrong numerical summarizations, which are practical problems in data analysis, code development, and document processing.

  2. The authors study the wrong shortcuts analysis through their proposed GulliBench. By constructing both a convenient wrong path and a verifiable correct path, the authors count final errors as well as analyze whether a model chose a specific shortcut and whether it tried to verify the underlying evidence.

  3. The paper further explores models' behavior from final task outcomes. Beyond reporting post-training benchmark scores, the authors examine whether trajectories recompute results or test key premises.

  4. The cross-benchmark experiments validate the authors' claim. The paper examines behavioral changes on data-analysis, database-query, and coding tasks, and reports the score gain on data analysis, offering a basis for analyzing the limitations.

Critical weaknesses
  1. A distribution gap may be between the synthetic benchmark and real problems. GulliBench's wrong shortcuts are constructed by the authors, and the correct answer is always determinable from the underlying data. In real work, extra checking does not necessarily have a clear outcome.

  2. The training and test tasks share the identical generation paradigm, which may lead the model to learn the structure of the synthetic data or a specific checking routine. The paper also notes that the gains do not cover all held-out recomputation or selection errors.

  3. The cross-benchmark results may prove the effectiveness of frequent checking rather than generally more reliable final answers. The DABstep score improves by about 4.4 percentage points, but the improvements for BIRD and the two coding benchmarks reach nearly zero. Changes in checking behavior and improvements in correctness should be interpreted separately.

  4. The behavior annotation carries measurement uncertainty. The paper uses LLM judges to grade a large number of trajectories with a limited human calibration set, and the authors report high judge disagreement on coding tasks.

  5. The independent contribution of the training mechanism is not fully clear. The current results show that a specific RL training recipe is effective, but comparisons with supervised fine-tuning, RL on non-conflict data, optimized prompting, and increased test-time compute are needed to determine where the advantage comes from.

Questions for authors
  1. Is there any evidence from real user workflows or in-the-wild data, even a small annotated set, that the learned checking behavior transfers beyond the constructed conflicts?
  2. The training and test tasks come from the same generation pipeline. Can the authors evaluate on held-out tasks built with a different template, domain or conflict-injection method, to check whether the model learning artifacts of synthetic data templates or a fixed checking routine?
  3. Across BIRD, DABstep, Terminal-Bench 2.1, or SWE-bench Pro, can the authors report checking frequency and correctness jointly at the instance level? In what cases does extra checking turn a wrong answer into a right one?
  4. The training setup can be further ablated. The comparisons over reward shaping and the training recipe can show where the advantage comes from.
1 comment

We sincerely thank the reviewer for a careful and generous review. The questions on in-the-wild evidence and on reading checking and correctness together led to most of our new analyses, and we believe they have made the paper considerably stronger.

W1 and Q1 (in-the-wild evidence). The reviewer is right that in real work a check need not have a clear outcome. GulliBench isolates the case where it does, so that the measurement is unambiguous, and §5 tests the behaviour on third-party tasks where it need not. To answer directly, we searched about 1,600 tasks in nine public benchmarks for GulliBench's pattern, a convenient value that the data contradict, and found 17 (13 in DABstep, 3 in Terminal-Bench, 1 in Terminal-Bench Science). We call them GulliBench-In-The-Wild (new App. J) and ran the 15 we could, ten attempts per model.

Table 1. Base and post-trained Qwen on the 15 GulliBench-In-The-Wild tasks, ten attempts per task. Share of attempts that pass, re-check their work, and test a premise.

Share of attempts (%)Base QwenPost-trained QwenΔ (95% CI)
Passes22.033.3+11.3 [+3.3, +20.0]
Re-checks its work45.384.0+38.7 [+26.0, +52.0]
Tests a premise14.729.3+14.7 [+0.7, +30.7]

Table 2. The same 15 tasks grouped by outcome, with each model's mean pass rate and that of public agent runs on the same tasks.

Tasks (mean pass rate, %)NumberBase QwenPost-trained QwenPublic agent runs
Post-trained model gains621.751.766.2
Neither model passes70.00.050.9
Both models pass2100.095.083.0
All1522.033.361.3

The post-trained model gains on six tasks and loses one attempt on one. Frontier agents also fall for these tasks, missing the answer in 42% of their public DABstep submissions. Such tasks are rare in existing benchmarks, which is the gap GulliBench fills.

W2 and Q2 (same generator). We agree that a gain on GulliBench alone cannot rule out that the model learned something specific to our generator. As §4.1 states, "the evaluation set shares neither its shortcut types nor its data with the training set", so the gain cannot come from memorising the benchmark's shortcuts. The two sets do share families of types, such as stale stored values, and the revision now states this limitation in §4.1. This is why §5 serves as the held-out evaluation the reviewer asks for, on four third-party benchmarks never used in training.

W3 and Q3 (checking and correctness, jointly). Thank you for this request. We now read both for every attempt. For each of the 2,689 premise-testing attempts, two judges also recorded what the check found and what the agent did next.

Table 3. Score and premise testing on each benchmark for both models, then, for the post-trained model's premise tests, how often the check found a problem and how often the model then changed course.

BenchmarkScore, base → postTests a premise, base → postCheck finds a problemModel changes course when it does
GulliBench3.2 → 31.145.1 → 80.885%57%
DABstep25.8 → 30.226.1 → 39.676%93%
BIRD68.8 → 66.74.8 → 15.135%68%
Terminal-Bench 2.148.9 → 52.28.9 → 22.265%100%
SWE-bench Pro37.3 → 39.014.3 → 17.387%71%

The checks often find a problem, and the model usually acts on it. Where the task contains a discrepancy, as on GulliBench, acting on it turns wrong answers into right ones, adding 19.7 points (CI [+13.0, +26.5]) over other attempts on the same task. On the third-party benchmarks it rarely changes correctness, because the answer was already right, the reference skips the same check, or the task stays out of reach (§5.3 gives a case of each). Where tasks do turn on such a check, as in GulliBench-In-The-Wild, the score rises by 11.3 points.

W4 (judge uncertainty). We agree, and have added a fresh validation set (App. H). One author graded 24 new trajectories from the four benchmarks, coding over-represented, blind to model, outcome and judge grade. The author and the judges agree on whether an attempt tests a premise in 21 of 24 (κ 0.60). The three disagreements are all on the two coding benchmarks, which our calibration set covered least. The paper also reports the median of three judges and a replicate check (App. H).

W5 and Q4 (training ablations). We agree. Our training claim is an existence claim, that the behaviour can be trained with a plain recipe (GRPO, binary reward, one run). Two requested comparisons are in the paper. A skeptical prompt has no measurable effect (+0.7 points, CI [−0.9, +2.5]), and more reasoning effort does not help (GPT-6 Astra 19.9% at low and 20.9% at high effort). RL on conflict-free data would be a valuable control, and we leave it, as we now note in §6, and a comparison with supervised fine-tuning for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.