- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study the tendency of LLM agents to over-rely on convenient but potentially incorrect evidence instead of verifying them. To systematically investigate this phenomenon, they introduce GulliBench, where each task contains designed shortcuts that produce incorrect answers, while the correct answer can be obtained by examining the underlying data. Through the evaluations on 23 frontier models, the paper finds that even the strongest model has the over-reliance problem. Furthermore, the authors post-train Qwen3.5-35B-A3B on synthetic tasks with shortcuts and find that the over-reliance problem can be mitigated through a simple training process. Overall, this paper contributes a targeted evaluation benchmark, an in-depth investigation of the over-reliance phenomenon, and preliminary evidence that verification-related behaviors can be improved through targeted reinforcement learning.
Critical strengths
- Clear motivation and important significance. The paper studies a well-motivated and practically important problem. As LLM agents become increasingly capable, their tendency to over-rely on convenient but unverified evidence can significantly undermine their reliability. Despite its importance, this phenomenon has received relatively limited attention through systematic and controlled studies. In this regard, both the proposed GulliBench benchmark and the findings on mitigating over-reliance offer meaningful contributions to improving the reliability of LLM agents.
- A coherent and complete research framework. The paper presents a well-structured and logically coherent investigation of the problem. It first identifies and categorizes representative shortcut types associated with over-reliance, then constructs a benchmark to systematically evaluate and analyze model behaviors. Finally, it gives a mitigation approach to reduce the identified failures. This progression from problem characterization and measurement to behavioral analysis and mitigation forms a complete research narrative.
Critical weaknesses
- Poor academic writing and lack of a coherent narrative. This paper's main weakness lies in its poor writing. Several passages contain unnecessarily long sentences with multiple nested clauses, making the arguments difficult to follow (e.g. the sentence around Line 440 spans three lines and introduces multiple conditions and qualifications, making its logical structure unnecessarily complicated). More broadly, the paper often reads like a chronological research log rather than a well-structured academic manuscript. For instance, around Line 184, the authors explicitly document the date of an intermediate stage in their experimental process. In my view, an academic paper should prioritize a clear and systematic presentation of the research problem and limitations of existing approaches, the core principles underlying the proposed methodology and experimental design, and the evidence supporting the conclusions, rather than extensively documenting every detail of the research process. I believe the paper would benefit significantly from restructuring the main text around its central arguments and moving secondary experimental details to the appendix.
- Benchmark selection is unfair and unreasonable. Specifically, the authors select the most challenging tasks based on the performance of Claude Fable 5. Although such adversarial selection can be reasonable for constructing a challenging diagnostic benchmark, it raises concerns about the fairness and representativeness of the evaluation. In particular, the paper's finding that "almost all frontier models fail on more than half of their attempts" may largely depend on the difficulty distribution of the selected tasks, rather than reflecting a general and widespread tendency of LLM agents to fall for these shortcuts.
- Benchmark design does not adequately support the claimed lack of verification behavior. A more fundamental concern lies in the benchmark design itself, which weakens the paper's conclusions about LLM agents' lack of verification behavior. Specifically, in GulliBench, the underlying assumption is that values recomputed from the original records are always more trustworthy than stored values. Consequently, an agent that simply adopts a fixed strategy of recomputing stored aggregates could achieve a high score without genuinely learning when verification is necessary or how to evaluate and reconcile conflicting evidence in realistic scenarios. Therefore, strong performance on GulliBench does not necessarily demonstrate a general ability to verify evidence, and failures on this benchmark may not be sufficient to support the broader claim that LLM agents lack such an ability.
Questions for authors
- Could the authors provide model performance on all 93 eligible task pairs, or at least a more systematic comparison between the selected and excluded tasks?
- Could the authors provide additional experiments to distinguish between the following two behaviors for both the evaluated frontier models and the post-trained model? (1) systematically distrusting stored values and recomputing them from underlying records, and (2) selectively verifying information based on the reliability of the available evidence?
- Could the authors clarify why the transaction records should be considered more authoritative than the stored account balances in Figure 1? Neither the question nor the provided data explicitly establishes which source is more up-to-date or complete. For instance, it is also possible that the account balances have been updated while the transaction records have not yet been synchronized. Without additional assumptions about data completeness or source reliability, how can the intended answer be considered unambiguously correct?
- Could the authors provide stronger justification for the specific structural choices in GulliBench, particularly the use of two tables and two-step tasks? Although Section 2.4 provides some intuition for the two-step design, it remains unclear why these design choices are necessary for measuring over-reliance.
- In Section 2.3, the authors state that, in the table format, the agent "must do any filtering or arithmetic itself." However, it appears that agents in the document format also need to perform similar computations, whereas the SQL format allows these operations to be executed directly by the database. Could the authors clarify the computational capabilities available in each format?
- Section 4.1 states that the training and evaluation sets use disjoint shortcut types, while also acknowledging that some training shortcuts share underlying patterns with evaluation shortcuts, such as stale aggregates. I think these two things are somewhat contradictory. Could the authors clarify it more clearly?
- Could the authors provide an analysis of the inference-time efficiency of the post-trained model compared with the base model? Since increased verification may require additional computation and tool interactions, it would be valuable to understand whether the improvements in reliability come at a substantial efficiency cost.
3 comments
Thank you for the authors' feedback. I have reread the your revised manuscript, and it is now much clearer. However, this suggests that the authors are capable of write a good paper, but their initial submission did not reflect the same level of care and effort. I therefore do not think the authors demonstrated an appropriate level of commitment to the quality of their work in the original submission, and I will maintain my original score.
We thank the reviewer for rereading the revised manuscript, and we are glad it is now much clearer. We regret that the original submission did not present the work as clearly as it should have, and the reviewer's detailed comments are a large part of why the revision improved. If any concern about the method, the evidence or the claims remains, we would welcome the chance to address it before the discussion closes. Otherwise, we would be grateful if the score could reflect the paper as revised.
We are very grateful for such a thorough and thoughtful review. Each question pointed to a real gap in how we presented our work, and we have revised the paper substantially in response.
W1 (writing). We fully agree, and thank the reviewer for the concrete examples. We documented the process in detail, including the date at line 184, because a benchmark selected against a model invites the worry that it was tuned after the fact, and we wanted every step to be checkable. The reviewer is right that this belongs in an appendix. We have split the sentence at line 440, moved the process detail of §2.4 and §4.2 to the appendix, and added a diagram of the task structure (Fig. 3). We keep the date only to show that ten of the 24 entries were released after the selection was fixed, and now say so.
W2 and Q1 (selection). We agree the finding should not depend on the selection. As requested, we ran 22 of the 24 entries on all 93 eligible pairs (App. B gives every entry).
Table 1. Pass rates of 22 entries on the 50 kept pairs, the 43 excluded pairs and all 93, how many entries fail more than half their attempts, and how closely each set ranks the models like the kept pairs (Spearman).
The selection made the benchmark harder, as intended, but the finding holds without it. On all 93 pairs, 17 of the 22 entries fail more than half of their attempts, and the order is nearly the same, with the same top three (Fable 5.1, Fable 5, Opus 5). The selection does not favour Fable 5, since the excluded pairs are easier mostly for its own family (+43 points against +18). We also generated the 100 candidates before running any model, so the selection did not shape the tasks.
W3 and Q2 (recomputing versus selective verification). We agree, and we have tightened our claims (§6). An agent that always recomputes would score well, so a high score cannot show selective verification. Two results separate the behaviours. For frontier models, 68% of failures never computed the contradicting value (Q3), so they neither distrusted the stored value systematically nor verified it selectively. For the post-trained model, a recompute rule has little to act on in §5's third-party benchmarks, yet it tests its premises more often, even on Terminal-Bench 2.1, which has no stored values (8.9% to 22.2%). When a check confirms a value it keeps it, and when the check finds a problem it changes course in 68% to 100% of cases.
Q3 (why the records are authoritative). We thank the reviewer for this important question. The reviewer is right that, in general, records need not be more reliable than a stored value. We address it in three ways. First, each task shows which source is authoritative. For every other entity the stored value equals the value recomputed from its records (200 of 200), and only the shortcut's entity differs. Second, most failures never reach this choice. In 68% of the 14,019 failed attempts that reported the stored value, the agent never computed the contradicting value, and in 28% it never read the records. Third, when the convention is stated, models follow it. Told to reconcile from the records, with the tables in its prompt, DeepSeek-V4 Flash passes 87% of runs, against 16% without (App. D), and Fable 5.1 and GPT-6 Astra report the stored value in under 20% of attempts in which they had obtained the correct one. GulliBench thus measures whether an agent checks, not which source it prefers. We have added the first point to the caption of Fig. 1.
Q4 (two tables, two steps). Two tables are the smallest setting in which a derived value and its source coexist. Two steps make the first result an input the agent can check or trust, which one step cannot. Fig. 3 now shows this.
Q5 (formats). Thank you, the reviewer is right, and §2.3 is corrected. The table format offers schema, count_rows, page_rows and read_row, documents offers list_files, grep and read_file, and SQL offers schema and a read-only query. No format executes code, so only SQL computes for the agent (App. A.2).
Q6 (disjoint types, shared patterns). Thank you for pointing out this unclear wording. No evaluation shortcut type appears in training, but both sets draw on the same five families (e.g. stale stored values, 7 training and 8 evaluation types). The first rules out memorising GulliBench's shortcuts, and the second is why we test on third-party benchmarks (§5). §4.1 now says this.
Q7 (efficiency). Median steps per attempt go from 6 to 9 on GulliBench, 14 to 15 on DABstep and 5 to 6 on BIRD, and do not rise on coding (Terminal-Bench 2.1 17.5 to 17.5, SWE-bench Pro 76 to 66). The cost is a few steps on data tasks (App. K).