acceptodds
Under review as a conference paper at ICLR 2027

RITE: Evaluating End-to-End Resilience to Silent Tool Errors in E-Commerce Agentic Systems

Abstract

Tool-using LLM agents increasingly answer e-commerce queries through multi-step tool interactions. The expected output of a tool call can shift between agent development and execution, allowing a plausible but incorrect value to be returned without an explicit failure signal. These silent errors may propagate through subsequent decisions into final answers. To study agents’ resilience across this end-to-end process, we formulate Silent-error Resilience Task, an interactive task that evaluates whether they complete the original query after a silent error corrupts an intermediate state. We introduce RITE, a benchmark built from human-authored industrial workflows and production-log-derived inputs. To enable controlled and reproducible evaluation, we use a semantic error taxonomy to inject errors into fixed workflow prefixes, freeze recorded tool behavior, and let agents continue freely. It contains 167 matched units across nine e-commerce workflows, yielding 501 silent-error, correct-value, and explicit-failure instances. Across 10 recent LLMs, silent errors roughly halve task success and are consistently harder than explicit failures. Recognition, use of an available verification path, and integration of the resulting evidence form successive bottlenecks. We also propose a light State Checker as an intervention to test whether resilience can be improved.

Primary area:
datasets and benchmarks
Keywords:
Agentic System, tool use, silent error recovery, benchmark evaluation, e-commerce systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CFDSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a new benchmark RITE to study silent errors during LLM agents interactive tool calling process. The silent error is a tool return that has a valid schema but incorrect. This type of error can be passed to the next step without a warning and causing error in the final answer. The benchmark aim to evaluate LLM's resilience if such error occurs.

The RITE is made by nine e-commerce workflows from an industrial platform. The inputs come from production logs and each test unit is designed as follows:

  • Step 1: The author stop a recorded workflow trace at on slot that is important for the task
  • Step 2: They make three prefixes, a correct value, a silent wrong value, and an explicit failure for this slot
  • Step 3: Then they select the wrong value from a taxonomy of seven error types. The benchmark uses six of these types

As described in paper, the final benchmark include 167 units and 501 instances. After conducting experiments on ten LLMs, three main finding are concluded as follow:

  • Silent errors nearly halve result success and are consistently harder than explicit failures
  • The agent can not aware the silent error. Even when an error is detected, the tool that cause the error will rarely been re-invoke for verification
  • A prompt-based "State Checker" increases noticing and refetching, but improves final success slightly
Critical strengths
  • First of all, I think the proposed benchmark brings up an important and practical problem for multi-turn or multi-tool calls agentic scenarios. The silent error can affect the final result and hard to be detected in many industrial cases

  • The benchmark is built from nine human-authored workflows deployed in production and from inputs derived from production logs, not from synthetic toy tasks. This make the corrupted values plausible in the way real silent errors are

  • The research provide an informative decomposition analysis of the recognition, active verification, recovery of the correct evidence and final task success. A well illustration on the failure mechanisms

  • The result number are internally consistent and the validation is well reported. The metrics and human involved validation are well designed and well support the conclusion from this research

Critical weaknesses
  • The alignment between judge model and human validation seems week. The core results from this paper is rely on using GLM-5 as LLM judger. However, in human validation, the cohan's kappa score is 0.318 which consider as a weak correlation. The paper fail to demonstrate that the judge equally strict across three experimental conditions of correct, implicit and explicit

  • From the abstract, the author claim the benchmark is for end-to-end resilience test. But the reality is the benchmark measures recovery from an inherited prefix where expect LLM to fix the execution history up to the corrupted slot. This is reasonable setting for controlled comparison, but it changes the construct being measured. It is not a full autonomous agentic run

  • Unclear benchmark availability and reproducibility. The paper does not clearly establish whether the complete RITE benchmark, including evaluation instances, workflow definitions, executable environments, and evaluation artifacts, will be publicly available

Questions for authors

None

4 comments

Thank you for recognizing RITE's practical value on real production workflows, its informative stage-wise decomposition, and its internally consistent results. We address the weaknesses and questions you raised below. We will revise the manuscript accordingly and upload the revision to OpenReview as soon as possible, with changes highlighted in blue.

W1: The judge applies the same criteria across conditions

The judge itself is reliable: the lower κ in Appendix D, especially on explicit and implicit, comes from errors in human annotation. Once these are corrected, the judge's evaluation is consistent across all three conditions.

In Appendix D, the human majority almost always agrees when the judge marks an answer as solved, in every condition:

correctexplicitimplicitAll
Human agreement with judge's "solved" (precision)96.2%100%95.5%97.1%

Yet κ is modest, so we examined where the disagreements come from. They concentrate in explicit and implicit, and 22 of the 24 are "judge unsolved, human solved". Manually inspecting these 22 cases, we found two reasons. First, humans miss small factual errors when the answer stays fluent and consistent with the polluted context: e.g., the reference age is 55, but the answer adopts the injected 44 and calls the user "in their forties", and all three annotators marked it solved. Second, rollouts span 7–10 turns, so annotators rarely trace a value back to its evidence. These problems arise mainly in the error conditions, so the original agreement understates the judge's reliability exactly where cross-condition comparison matters.

We therefore ran a new annotation study in which annotators judge against the relevant evidence rather than the full trajectory alone. We took 45 rollouts (15 per condition), prioritizing the 20 Appendix D disagreements (34 overlap with Appendix D), and six new participants recruited through campus channels labeled each with 3 annotators. Annotation has two stages:

  • Stage 1 (blind labeling): annotators see the full trajectory, a summary of the context relevant to the task's key statements, and the answer values to be checked highlighted against it. A facilitator answers questions and provides background without revealing the judge's decision. Annotators then label whether the task is solved.
  • Stage 2 (re-check): annotators view the judge's rationale and, where it disagrees with their label, judge whether the rationale is correct.

A rollout counts as solved only if all three agree. The Stage 1 results are:

ConditionnAgreementCohen κAC1Fleiss κ
correct1593.3%0.8150.8960.722
explicit15100.0%1.0001.0000.911
implicit1593.3%0.8150.8960.689
All4595.6%0.9110.9110.821

In Stage 2, annotators agreed that the judge's rationale was correct in 125 of 131 reviewed responses (95.4%; 3 disagree, 3 unsure), with similar rates across correct / explicit / implicit (95.3% / 97.8% / 93.0%).

For the 34 rollouts labeled in both studies, we compare the human majority label in Appendix D with that in the new study. Most changes are from solved to unsolved, and they occur mainly in explicit and implicit:

ConditionnSolved→UnsolvedUnsolved→SolvedAgreement w/ judgeFleiss κ
correct111281.8% → 90.9%0.456 → 0.819
explicit96122.2% → 100.0%−0.038 → 0.841
implicit148128.6% → 92.9%0.028 → 0.630
All3415444.1% → 94.1%0.255 → 0.803

With the relevant evidence, judge–human agreement is similar across conditions: 93.3% / 100% / 93.3% for correct / explicit / implicit (Cohen κ 0.815 / 1.000 / 0.815). On the 34 overlapping rollouts, 14 of the 15 solved→unsolved changes occur in explicit and implicit, i.e., humans now catch errors they had missed there, while changes on correct go both ways (1 vs. 2), so annotators did not simply become stricter. The earlier gap across conditions thus came from human checking, not from the judge. By design, the judge also sees the same reference evidence for each paired slot and receives no condition label. We will add this study to Appendix D.

W2: Starting from an inherited prefix is a measurement choice

Starting from a fixed prefix is a deliberate choice for controlled testing. In a fully free run [1, 2], the agent's own trajectory decides whether, when, and with what history it meets the error, so the error cannot be controlled. We therefore follow the common practice in recovery benchmarks of starting from a fixed point [3, 4]: the prefix is a real workflow execution, and one slot returns a correct, silently wrong, or explicitly failed value. After this point, the agent acts freely. Fixing the prefix lets us isolate how the agent handles the error, and keeps the scenario and task difficulty the same across conditions and models.

We will revise the abstract and introduction to state that a trajectory is a controlled prefix plus free continuation, explain this choice in §3–§4, and replace "end-to-end" in the title with "recovery".

[1] When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents. [2] Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability. [3] Recovery-Bench: Evaluating Agentic Recovery from Mistakes. [4] AgentRewind: Recoverable Execution for Long-Horizon LLM Agents

For the availability and reproducibility, the appendix already provides extensive detail for reproduction, including the error taxonomy, construction pipeline, evaluation protocol, and judge prompts, so the methodology can be applied to other workflows. We are actively pushing to release both the code and the data. Because RITE is built on the company's production workflows and real user interactions, release requires internal approval, and we cannot yet guarantee the outcome. If the data cannot be approved, we will file a separate application to release the code on its own.

Dear Reviewer, we have uploaded a revised PDF with changes highlighted in blue.
We hope the revision addresses your concerns and would appreciate your reconsideration of the score.
Thank you for your time.

We sincerely thank you for your valuable feedback. As the rebuttal period is approaching its end, we are glad to provide further clarifications regarding your remaining questions. We hope these responses adequately address your concerns, and we would appreciate it if you could reconsider the assessment in light of our revisions

Reviewer_uJzCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates how tool-use agents answer e-commerce tasks when there is a plausible but incorrect tool return value without explicit error. This is motivated by possible shifts in tool calls between agent development and execution. RITE is built from e-commerce industrial workflows with injected errors with fixed prefixes to enable comparison across agent models on how they react to these injections. Across 10 models, silent error negatively impact task success and harder to detect and address compared to explicit errors.

Critical strengths

The paper constructs industry workflows in e-commerce with controlled injections that enable a useful comparison. It targets a practically useful problem with schema-valid errors in business-derived tasks and analyzes the consequences of different types of errors (e.g. explicit or silent).

Critical weaknesses

The current benchmark design doesn't clearly establish when the agent can reasonably detect the silent error. Although the timestamp example provided in the paper is one case of a silent error resulting in a detectable discrepancy, the taxonomy doesn't clarify how the different silent errors become observable to the agent. Thus, interpreting failure to catch the silent bad evidence as deficient agent behavior is not clearly supported. If the expected ideal agent behavior is to distrust tool outputs by default and verify them even without an observable inconsistency, this also seems overly strict without justification for when such verification is warranted.

The current scenario assumes one historical value is wrong and repeating the matching recorded tool call will return the clean result. It isn’t quite clear why this setting is motivated as a representative case for these silent tool errors. The motivating settings mentioned in the paper such as data change, tool implementation change, or version change are relevant in production but do not justify the correct-on-retry behavior. This limits how strongly the benchmark results support conclusions about resilience to the deployment failures used to motivate the paper.

Questions for authors

Please see the stated weaknesses.

6 comments

Thank you for recognizing RITE's industrial e-commerce workflows with controlled injections as enabling useful comparison, its schema-valid errors in business-derived tasks as targeting a practical problem, and its analysis of the different consequences of explicit and silent errors. We address the weaknesses and questions you raised below. We will revise the manuscript accordingly and upload the revision to OpenReview as soon as possible, with changes highlighted in blue.

W2: Retry recoverability is a measurement tradeoff

Correct-on-retry is a deliberate measurement choice. Each RITE slot lies on the task's necessary path, so if gold could never be retrieved, the task would be unsolvable by design, and RSR would measure solvability rather than how the agent handles the silent error. Keeping a clean path also lets us locate where agents fail along recognition, refetch, and recovery (F2–F3), which a persistent fault would conflate. The setting directly covers transient deployment failures, such as canary routing, stale caches or replicas, and write races, as well as the 20 tool_input units, where the tool is correct but receives a wrong argument. For the persistent faults the reviewer raises, such as data, implementation, or version changes, we test below whether this choice overstates our conclusions.

Even if retries never returned the correct value, the impact would be limited: the retry setting affects an agent only when it re-invokes the slot's producer, which most rollouts never do (implicit condition, 10 models):

StageRolloutsShareRSR
Not noticed, no producer refetch118075.9%0.276
Noticed, no producer refetch23014.8%0.387
Producer refetched1459.3%0.648

To verify this, we removed the correct-on-retry condition and reran 3 models (Claude-Opus-4.8, DeepSeek-V4-Pro, GLM-5.2) on 130 tool_output units: every producer call, with any arguments, now returns the same wrong value as the prefix, while everything else, including other tools for cross-checking, is unchanged. As in the main experiment, rollouts with tool execution exceptions are removed. We pair each rollout with the main-experiment rollout of the same model and slot to see what changes:

GroupRolloutsShareImplicit RSR: recoverable → persistent
Did not call the producer29790.5%0.273 → 0.263
Called the producer (got the same wrong value)319.5%0.516 → 0.323
All328100%0.296 → 0.268

Overall implicit RSR drops by only 2.7pp. The loss comes mainly from the 9.5% of rollouts that called the producer, whose RSR falls from 0.516 to 0.323. Most of those that still succeed do so by noticing the conflict and switching to another source in the context.

For our findings, F1 (silent errors hurt success) and F2 (low noticing, rare refetching) are unchanged, since most failures occur before any retry. For F3, recovery must come from an independent source rather than refetching the producer, so the main RSR is an optimistic upper bound for deployment.

W1: Silent errors in RITE leave checkable evidence in the context as recovery path

Silent errors in RITE are designed to have a detection path. The query, task input, and prefix come from the same real execution as the gold value, so this context is consistent with gold; once gold is replaced, the new value conflicts with the context, and this conflict is what lets the agent detect the error. We verify with a human study that this conflict is distinguishable, show that agents seeing only the polluted value can notice it, and analyze agent trajectories to show how each error type can be detected.

Human study. If humans can tell gold from the replacement using only the context, the evidence needed to detect the error is present in what the agent sees, so the detection path exists. We sampled 30 units from RITE; for each, 3 non-author participants independently saw the task, the preceding context, and the two values in random order, without knowing which was gold. We report the majority (≥2 of 3) answer, unanimous agreement, and Gwet's AC1 (chance-corrected inter-rater agreement; >0.8 indicates strong agreement).

Question to participantsMajority answerAll 3 agreeGwet's AC1
Q1: Which value supports a correct answer?The original value: 29/30 (96.7%)25/30 (83.3%)0.873
Q2: Is the other value wrong, or a valid change?Wrong: 29/30 (96.7%)23/30 (76.7%)0.814

Since both values are schema-valid and plausible on their own, humans could only tell them apart through the context, and they did so in 29 of 30 units with strong agreement. The detection path therefore exists; whether the agent uses this evidence is what RITE measures.

Agents can notice the conflict from the polluted value alone. Unlike the human study, an agent sees only one value. We therefore compare the correct and implicit conditions, which share the same task and context and differ only in whether this value is gold or polluted. Using GLM-5, we label whether the agent's reasoning explicitly doubts the value (aware). On 1,535 matched (model, slot) pairs across 10 models, agents are aware far more often for polluted values than for gold ones (0.220 vs. 0.062, McNemar p < 1e-40), significantly so for every model. The conflict is thus noticeable from a single value, and since gold values are rarely doubted, the suspicion is targeted rather than default distrust.

Trajectory analysis. Agent trajectories support the human study. We sampled 108 implicit-condition trajectories stratified by error type: 48 that missed the error and failed, and 60 judged as Noticed. GLM-5 annotated each trajectory, and the authors verified the annotations (85% agreement). In the 48 missed trajectories, evidence pointing to gold was present in the agent's context in 41 of the 45 successfully labeled cases (91%). Since the tasks are largely solvable (correct RSR 0.716–0.827), most misses reflect the agent not using available evidence.

Across all 108 trajectories, the extracted inconsistency shows what this evidence looks like for each type. The polluted value conflicts with:

  • Numeric bucket: the same quantity recorded in the task input or by another tool, which falls in a different range.
  • Entity near-duplicate: the entity identified elsewhere in the context, which the polluted value resembles but does not match.
  • Profile attribute: other observations of the same object, such as its behavior or history.
  • Semantic synonym: the scene described in the instruction or other observations, whose meaning the polluted wording shifts.
  • Category sibling / granularity: the content or interest indicated by the query, tags, or summaries, which a neighboring category or a wrong category level does not match.

On default distrust. We agree with the reviewer that distrusting every tool result is not a scientific goal, and RITE does not ask for it. It expects targeted verification: check a value when it conflicts with other evidence and the answer depends on it. Default distrust corresponds to the extreme probed by the State Checker in D3, which we use as a diagnostic intervention. On the same slots, it raises noticing and refetching substantially, yet implicit RSR improves by only 0–9pp, consistent with D3's finding that the bottleneck lies in integrating evidence rather than in suspicion (baseline → State Checker):

ModelImplicit NoticedImplicit refetchImplicit RSRCorrect RSR
Claude-Opus-4.80.348→0.4700.121→0.1520.415→0.4150.735→0.814
DeepSeek-V4-Pro0.219→0.4790.137→0.2330.343→0.4250.797→0.732
GLM-5.20.183→0.5630.085→0.2680.333→0.4230.821→0.721

Dear Reviewer, we have uploaded a revised PDF with changes highlighted in blue.
We hope the revision addresses your concerns and would appreciate your reconsideration of the score.
Thank you for your time.

Reviewer_7zp1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating how LLM agents handle silent tool errors, where tools return plausible but incorrect values without explicit failure signals. The benchmark is constructed from industrial e-commerce workflows and production-log-derived inputs. It contains 167 matched units across nine workflows, with each unit evaluated under correct-value, silent-error, and explicit-failure conditions. The authors evaluate 10 LLMs and find that silent errors reduce task success and are harder to handle than explicit failures. Further analysis shows that agents often fail to recognize incorrect tool outputs, and even when they retrieve correct information, they may still fail to use it properly in the final answer.

Critical strengths
  1. The paper studies an important reliability problem in tool-using agents. Unlike explicit tool failures, silent errors may appear valid and propagate through subsequent actions without being detected.

  2. The benchmark provides a controlled setup for comparing agent behavior under correct, silent-error, and explicit-failure conditions. The evaluation across 10 LLMs also provides useful insights into different stages of error recovery, including error recognition, verification, and the use of corrected evidence in final answers.

Critical weaknesses
  1. The benchmark treats values from the original workflow traces as correct and introduces artificial corruptions as silent errors. However, a different tool output, such as one caused by a model or tool update, is not necessarily incorrect. It is unclear how well these injected errors represent real-world silent failures.

  2. The paper relies on an LLM judge to evaluate final task success. However, the human validation in Appendix D reports only 76% agreement between the automatic Result labels and human majority labels, with a Cohen's kappa of 0.318. Since task success is the main metric used to support the paper's findings, the relatively low agreement raises concerns about the reliability of the reported results.

Questions for authors
  1. How do the authors determine that the original workflow outputs are correct? A change in a tool's output after an update does not necessarily indicate an error. How do the authors distinguish genuinely incorrect values from valid changes in tool behavior, and how representative are the artificially injected errors of real production failures?

  2. The benchmark assumes that re-querying the original tool returns the correct value. How would the results change if the tool consistently returned the same incorrect value, as might happen after a model or implementation update?

5 comments

Thank you for recognizing silent tool errors as an important reliability problem, and RITE's controlled comparison across 10 LLMs as offering stage-wise insights into how agents recognize, verify, and recover from them. We address the weaknesses and questions you raised below. We will revise the manuscript accordingly and upload the revision to OpenReview as soon as possible, with changes highlighted in blue.

W1 & Q1: The original value is correct, and the injected value is a genuine error by design

In RITE, a value is an error only relative to the query, not merely because it differs from the original. For each slot, we replace the original value (gold) with a schema-valid value (polluted) drawn from real same-field values of other cases, real category-tree nodes, or controlled shifts. We then append a requirement to the query that depends on this slot, and keep the unit only if it passes four gates:

  • Leak: the query contains neither value.
  • Coupling: the query cannot be answered without this slot.
  • Solvable: gold supports a correct answer.
  • Discriminative: polluted leads to a different, wrong answer.

Together, these gates make both values unambiguous. Leak and Coupling force the answer to come from this slot alone. Under that constraint, Solvable makes gold the correct value for the query, and Discriminative rules out polluted as an equally valid alternative: since it changes the conclusion the query depends on, it can only produce a wrong answer. The judge also accepts the agent's new observations as evidence, so an agent that obtains a valid updated value is not penalized.

Example. A live-stream analysis tool reads a 30-second audio transcript and labels what the streamer is doing. The query asks to "identify the current scene, and based on it decide whether the streamer is focused on selling products or on a talent performance." The tool's input transcript reads: "…let me walk you through the material details of this bed, with a raised and thickened design," with audio-event scores Speech 0.999 and Singing 0.0005.

  • Gold: "Introducing products", the label the tool actually returned. It is correct because it matches the transcript and the audio scores, so the answer concludes selling products.
  • Polluted: "Singing", another legal label of the same tool, so it looks like a normal output. It is wrong because it contradicts the tool's own input, where there is no singing, and it leads the answer to conclude talent performance.

Human verification of the gates. Since the gates are checked by an LLM, we ran an additional blind study to confirm that the resulting units indeed satisfy both properties above. We sampled 30 units from RITE. For each unit, 3 non-author participants independently saw the task, the preceding context, and the two values labeled A/B in random order, without knowing which was gold, and answered the two questions below. We report the majority (≥2 of 3) answer, unanimous agreement, and Gwet's AC1 (chance-corrected inter-rater agreement; >0.8 indicates strong agreement).

Question to participantsMajority answerAll 3 agreeGwet's AC1
Q1: Which value supports a correct answer?The original value: 29/30 (96.7%)25/30 (83.3%)0.873
Q2: Is the other value wrong, or a valid change?Wrong: 29/30 (96.7%)23/30 (76.7%)0.814

Humans thus confirm what the gates are designed to enforce: in 29 of 30 units, gold is correct and polluted is a genuine error rather than a valid change, with strong agreement. We will add the gates, this example, and the study to the paper.

W2: Low κ reflects the difficulty of manual checking, not judge errors

Although κ is modest, the human majority almost always agrees when the judge marks an answer as solved (Appendix D).

ConditioncorrectexplicitimplicitAll
Human agreement with judge's "solved" (precision)96.2%100%95.5%97.1%

RSR therefore does not overstate success. To understand where the low κ comes from, we split Appendix D by condition:

ConditionnAgreementCohen κAC1Judge solvedHuman solvedHuman–human agreement
correct2892.9%0.4620.9180.9290.92990.5%
explicit3574.3%0.4000.5750.6000.85788.6%
implicit3764.9%0.1750.4320.5950.89282.0%

The table shows that disagreements concentrate in explicit and implicit (on correct, judge–human agreement even exceeds human–human agreement), and 22 of the 24 are "judge unsolved, human solved". Manually inspecting these 22 cases, we found two reasons. First, humans miss small factual errors when the answer stays fluent and consistent with the polluted context: e.g., the reference age is 55, but the answer adopts the injected 44 and calls the user "in their forties", and all three annotators marked it solved. Second, rollouts span 7–10 turns, so annotators rarely trace a value back to its evidence.

We therefore ran a new annotation study in which annotators judge against the relevant evidence rather than the full trajectory alone. We took 45 rollouts (15 per condition), prioritizing the 20 Appendix D disagreements (34 overlap with Appendix D), and six new participants recruited through campus channels labeled each with 3 annotators. Annotation has two stages:

  • Stage 1 (blind labeling): annotators see the full trajectory, a summary of the context relevant to the task's key statements, and the answer values to be checked highlighted against it. A facilitator answers questions and provides background without revealing the judge's decision. Annotators then label whether the task is solved.
  • Stage 2 (re-check): annotators view the judge's rationale and, where it disagrees with their label, judge whether the rationale is correct.

A rollout counts as solved only if all three agree. The Stage 1 results are:

ConditionnAgreementCohen κAC1Fleiss κ
correct1593.3%0.8150.8960.722
explicit15100.0%1.0001.0000.911
implicit1593.3%0.8150.8960.689
All4595.6%0.9110.9110.821

In Stage 2, annotators agreed that the judge's rationale was correct in 125 of 131 reviewed responses (95.4%; 3 disagree, 3 unsure), with similar rates across correct / explicit / implicit (95.3% / 97.8% / 93.0%).

For the 34 rollouts labeled in both studies, we compare the human majority label in Appendix D with that in the new study. Most changes are from solved to unsolved:

A rollout counts as solved only if all three agree. The Stage 1 results are:

ConditionnSolved→UnsolvedUnsolved→SolvedAgreement w/ judgeFleiss κ
correct111281.8% → 90.9%0.456 → 0.819
explicit96122.2% → 100.0%−0.038 → 0.841
implicit148128.6% → 92.9%0.028 → 0.630
All3415444.1% → 94.1%0.255 → 0.803

In Stage 2, annotators agreed that the judge's rationale was correct in 125 of 131 reviewed responses (95.4%; 3 disagree, 3 unsure), with similar rates across correct / explicit / implicit (95.3% / 97.8% / 93.0%).

This confirms that the earlier disagreements came from human checking, not from the judge. Once annotators check against the relevant evidence, agreement with the judge reaches 95.6% (κ = 0.911), and agreement among annotators rises from 0.255 to 0.803. The 15 solved→unsolved changes (14 in explicit and implicit) move humans toward the judge, while changes on correct go both ways (1 vs. 2), so annotators did not simply become stricter. Thus κ = 0.318 reflects the difficulty of manually checking long trajectories, and with 97.1% precision on "solved", RSR is a conservative but reliable measure of task success. We will add this study to Appendix D.

Q2: Persistent faults cost only 2.7pp RSR, but block recovery by refetching

A persistent fault only matters when the agent re-invokes the slot's producer. Most rollouts never do (implicit condition, 10 models):

StageRolloutsShareRSR
Not noticed, no producer refetch118075.9%0.276
Noticed, no producer refetch23014.8%0.387
Producer refetched1459.3%0.648

Since only 9.3% of rollouts refetch the producer, a persistent fault should have limited impact. To verify this, we ran an additional test on 3 models (Claude-Opus-4.8, DeepSeek-V4-Pro, GLM-5.2) and 130 tool_output units: every producer call, with any arguments, returns the same wrong value as the prefix, while everything else, including other tools for cross-checking, is unchanged. We pair each rollout with the main-experiment rollout of the same model and slot (328 pairs).

GroupRolloutsShareImplicit RSR: main → persistent
Did not call the producer29790.5%0.273 → 0.263
Called the producer (got the same wrong value)319.5%0.516 → 0.323
All328100%0.296 → 0.268

As expected, overall RSR drops by only 2.7pp, and the loss is confined to the 9.5% of rollouts that called the producer (0.516 → 0.323). In these 31 rollouts, refetching no longer recovers the correct value: some agents keep the wrong value after seeing the identical result and fail, while most successes come from agents that notice the conflict and switch to an independent source, such as another tool or correct information in the task input or query.

For the paper's findings, F1 and F2 are unaffected, since most failures happen before any retry. For F3, recovery must come from an independent source rather than refetching the producer, so our main RSR is an optimistic upper bound for deployment. We will report both tables in the appendix and state this scope explicitly in §4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.