- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Bench2Robust augments tool-use benchmarks with retry-solvable, switch-required, and unsolvable scenarios, alongside explicit failures and silent response corruptions. Bayesian Tool Memory (BTM) supplies recovery instructions, fallback relations, and empirical recovery statistics, and is combined with RL training. On 402 held-out Combined Retail tasks without added alternatives, online BTM raises the 4B base model's injected pass rate from 20.1% to 36.9%; RL without inference-time BTM reaches 26.4%, and the full system reaches 40.8%. S3 is a training condition, not a demonstrated independently learned abstention capability.
Critical strengths
-
Controlling whether the original path remains viable or switching is required is a useful way to study recovery. The proposed design distinguishes choosing an appropriate recovery strategy from benefiting when a random failure disappears. The distinction between execution-suppressing failures and post-execution observation corruption also makes the modeling assumptions explicit.
-
Tables 2 and 12 provide useful held-out evidence for runtime recovery context and behavior retained after training. Both 4B and 8B improve without inference-time BTM, with further gains when it is supplied. Clean-performance results, severity sweeps, and inference-cost reporting characterize practical trade-offs. The paper appropriately acknowledges that online BTM uses completed-episode outcomes and that evaluation seeds do not measure training-run variability.
Critical weaknesses
The supplied implementation conflicts with the central scenario definition. Section 2.1 and Algorithm 1 place episode-persistent blocking outside the stochastic injection budget and consecutive-failure limits. However, GenericInjectionEngine.should_inject passes blocked tools through InjectionPolicy.can_inject, which enforces both limits. The supplied test test_permanent_block_respects_write_protection_and_budget explicitly expects blocking to stop once the budget is exhausted, consistent with the README. If these semantics were used experimentally, a supposedly switch-required episode could instead be solved by retrying, changing what the evaluation measures. The README says the modified training fork is omitted, so this discrepancy does not establish that the reported experiments used incorrect semantics. Nevertheless, the connection between the stated intervention and the experimental implementation remains unresolved. This implementation gap limits how confidently the gains can be interpreted as recovery under the stated scenario controls.
Questions for authors
Which implementation and revision generated the scenario-controlled results? Please reconcile it with the supplied engine and test, and provide a trace showing that an S2/S3-blocked tool remains unavailable after both stochastic limits are reached. A separate experimental implementation enforcing Algorithm 1 could resolve the concern; if the supplied semantics were used, which scenario labels and results are affected?
4 comments
Follow-up on our responses to your review
Dear Reviewer 67Fa,
We want to thank you again for your thoughtful review and constructive feedback. As the interactive discussion period is currently active, we wanted to gently follow up to see if our replies and proposed revisions have successfully addressed your concerns. If there are any remaining questions or areas where further clarification would be helpful, please don't hesitate to let us know. We would be more than happy to discuss them further!
Best regards,
The Authors
Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE
Best, AC
Follow-up on our responses to your review
Dear Reviewer 67Fa,
With approximately 12 hours remaining in the discussion period, we would greatly appreciate your feedback on whether our implementation clarification, corrected code, and verification traces have resolved your concern.
If so, we would be grateful if you could consider updating your rating to reflect this resolution. We are happy to provide any further clarification.
Thank you again for your time and consideration!
Best regards, The Authors
Response to Reviewer 67Fa
We thank the reviewer for recognizing the value of our scenario-controlled evaluation and empirical results.
We traced the implementation discrepancy issue to an error introduced during anonymization and code packaging. In our original implementations, both the evaluation harness (
eval/src/run_inference.py:367-388) and the RL training environment (training/bench2robust_rl/rollout.py:982-994) check the blocked set before consulting the stochastic policy, and never call it for blocked tools. Blocking therefore cannot lapse afterBorK_maxis reached, and the issue in the releasedGenericInjectionEnginedoes not affect our reported results. We verified this with execution traces: underB=1, K_max=1, all blocked calls across both implementations remained blocked.We have corrected the packaged library and provide an updated Anonymous GitHub repository with the original source, the corrected library, tests, and verification traces.