- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose a new benchmark RITE to study silent errors during LLM agents interactive tool calling process. The silent error is a tool return that has a valid schema but incorrect. This type of error can be passed to the next step without a warning and causing error in the final answer. The benchmark aim to evaluate LLM's resilience if such error occurs.
The RITE is made by nine e-commerce workflows from an industrial platform. The inputs come from production logs and each test unit is designed as follows:
- Step 1: The author stop a recorded workflow trace at on slot that is important for the task
- Step 2: They make three prefixes, a correct value, a silent wrong value, and an explicit failure for this slot
- Step 3: Then they select the wrong value from a taxonomy of seven error types. The benchmark uses six of these types
As described in paper, the final benchmark include 167 units and 501 instances. After conducting experiments on ten LLMs, three main finding are concluded as follow:
- Silent errors nearly halve result success and are consistently harder than explicit failures
- The agent can not aware the silent error. Even when an error is detected, the tool that cause the error will rarely been re-invoke for verification
- A prompt-based "State Checker" increases noticing and refetching, but improves final success slightly
Critical strengths
-
First of all, I think the proposed benchmark brings up an important and practical problem for multi-turn or multi-tool calls agentic scenarios. The silent error can affect the final result and hard to be detected in many industrial cases
-
The benchmark is built from nine human-authored workflows deployed in production and from inputs derived from production logs, not from synthetic toy tasks. This make the corrupted values plausible in the way real silent errors are
-
The research provide an informative decomposition analysis of the recognition, active verification, recovery of the correct evidence and final task success. A well illustration on the failure mechanisms
-
The result number are internally consistent and the validation is well reported. The metrics and human involved validation are well designed and well support the conclusion from this research
Critical weaknesses
-
The alignment between judge model and human validation seems week. The core results from this paper is rely on using GLM-5 as LLM judger. However, in human validation, the cohan's kappa score is 0.318 which consider as a weak correlation. The paper fail to demonstrate that the judge equally strict across three experimental conditions of correct, implicit and explicit
-
From the abstract, the author claim the benchmark is for end-to-end resilience test. But the reality is the benchmark measures recovery from an inherited prefix where expect LLM to fix the execution history up to the corrupted slot. This is reasonable setting for controlled comparison, but it changes the construct being measured. It is not a full autonomous agentic run
-
Unclear benchmark availability and reproducibility. The paper does not clearly establish whether the complete RITE benchmark, including evaluation instances, workflow definitions, executable environments, and evaluation artifacts, will be publicly available
Questions for authors
None
4 comments
W2: Starting from an inherited prefix is a measurement choice
Starting from a fixed prefix is a deliberate choice for controlled testing. In a fully free run [1, 2], the agent's own trajectory decides whether, when, and with what history it meets the error, so the error cannot be controlled. We therefore follow the common practice in recovery benchmarks of starting from a fixed point [3, 4]: the prefix is a real workflow execution, and one slot returns a correct, silently wrong, or explicitly failed value. After this point, the agent acts freely. Fixing the prefix lets us isolate how the agent handles the error, and keeps the scenario and task difficulty the same across conditions and models.
We will revise the abstract and introduction to state that a trajectory is a controlled prefix plus free continuation, explain this choice in §3–§4, and replace "end-to-end" in the title with "recovery".
[1] When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents. [2] Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability. [3] Recovery-Bench: Evaluating Agentic Recovery from Mistakes. [4] AgentRewind: Recoverable Execution for Long-Horizon LLM Agents
For the availability and reproducibility, the appendix already provides extensive detail for reproduction, including the error taxonomy, construction pipeline, evaluation protocol, and judge prompts, so the methodology can be applied to other workflows. We are actively pushing to release both the code and the data. Because RITE is built on the company's production workflows and real user interactions, release requires internal approval, and we cannot yet guarantee the outcome. If the data cannot be approved, we will file a separate application to release the code on its own.
Dear Reviewer, we have uploaded a revised PDF with changes highlighted in blue.
We hope the revision addresses your concerns and would appreciate your reconsideration of the score.
Thank you for your time.
We sincerely thank you for your valuable feedback. As the rebuttal period is approaching its end, we are glad to provide further clarifications regarding your remaining questions. We hope these responses adequately address your concerns, and we would appreciate it if you could reconsider the assessment in light of our revisions
Thank you for recognizing RITE's practical value on real production workflows, its informative stage-wise decomposition, and its internally consistent results. We address the weaknesses and questions you raised below. We will revise the manuscript accordingly and upload the revision to OpenReview as soon as possible, with changes highlighted in blue.
W1: The judge applies the same criteria across conditions
The judge itself is reliable: the lower κ in Appendix D, especially on explicit and implicit, comes from errors in human annotation. Once these are corrected, the judge's evaluation is consistent across all three conditions.
In Appendix D, the human majority almost always agrees when the judge marks an answer as solved, in every condition:
Yet κ is modest, so we examined where the disagreements come from. They concentrate in explicit and implicit, and 22 of the 24 are "judge unsolved, human solved". Manually inspecting these 22 cases, we found two reasons. First, humans miss small factual errors when the answer stays fluent and consistent with the polluted context: e.g., the reference age is 55, but the answer adopts the injected 44 and calls the user "in their forties", and all three annotators marked it solved. Second, rollouts span 7–10 turns, so annotators rarely trace a value back to its evidence. These problems arise mainly in the error conditions, so the original agreement understates the judge's reliability exactly where cross-condition comparison matters.
We therefore ran a new annotation study in which annotators judge against the relevant evidence rather than the full trajectory alone. We took 45 rollouts (15 per condition), prioritizing the 20 Appendix D disagreements (34 overlap with Appendix D), and six new participants recruited through campus channels labeled each with 3 annotators. Annotation has two stages:
A rollout counts as solved only if all three agree. The Stage 1 results are:
In Stage 2, annotators agreed that the judge's rationale was correct in 125 of 131 reviewed responses (95.4%; 3 disagree, 3 unsure), with similar rates across correct / explicit / implicit (95.3% / 97.8% / 93.0%).
For the 34 rollouts labeled in both studies, we compare the human majority label in Appendix D with that in the new study. Most changes are from solved to unsolved, and they occur mainly in explicit and implicit:
With the relevant evidence, judge–human agreement is similar across conditions: 93.3% / 100% / 93.3% for correct / explicit / implicit (Cohen κ 0.815 / 1.000 / 0.815). On the 34 overlapping rollouts, 14 of the 15 solved→unsolved changes occur in explicit and implicit, i.e., humans now catch errors they had missed there, while changes on correct go both ways (1 vs. 2), so annotators did not simply become stricter. The earlier gap across conditions thus came from human checking, not from the judge. By design, the judge also sees the same reference evidence for each paired slot and receives no condition label. We will add this study to Appendix D.