- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ContractRL, a method for repairing incorrect tool calls by editing selected fields rather than generating the entire call again. The model receives error feedback, can make a limited number of repair attempts, and can decline a repair. Proposed changes are checked before the tool is allowed to run. The training reward encourages successful repairs while penalizing unnecessary edits, invalid actions, retries, and generated text. The authors compare ContractRL with supervised patch generation, full-call regeneration, and generation with formatting constraints, reporting higher task success and lower generation costs.
Critical strengths
The paper addresses a useful problem. When only a small part of a tool call is incorrect, repairing that part rather than rewriting everything is a sensible approach. But I am not sure if this is a novel approach or a significant advancement.
Critical weaknesses
The main success measure does not show how many tasks are successfully completed overall. Table 24 calculates semantic success only among calls that are actually executed. Because ContractRL can decline repairs or have them rejected, it can improve this success rate by avoiding difficult cases, rather than by successfully completing more tasks. This is particularly important when comparing the full method with the version that cannot decline repairs, as in Table 29.
The reinforcement-learning procedure is not sufficiently explained. Section 3 gives a reward function, but does not specify how alternative repair sequences are sampled and compared, or how their rewards are used to update the model. Since reinforcement learning is central to the claimed contribution, the manuscript needs the actual update procedure and a clear explanation of how the repair checks interact with training. It is also unclear whether prohibited edits are prevented during generation or rejected after generation.
Questions for authors
Please see above.