- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RePair, a rubric-guided framework that converts recurring errors in failed agent trajectories into locally validated preference pairs for offline training, without additional rollouts. Experiments on BrowseComp-Plus show that RePair improves Qwen3-14B accuracy from 42.6% to 53.2% with only 2,097 pairs, and supports iterative self-repair.
Critical strengths
The paper addresses an interesting and important problem with a well-motivated and natural approach. The proposed method achieves strong empirical results and provides several interesting insights into improving tool-using agents.
Critical weaknesses
- The experimental evaluation could be strengthened by comparing RePair with more existing methods.
- The experiments are conducted only on BrowseComp-Plus. Evaluating RePair on more diverse tasks and environments would help demonstrate its generalizability.
- The contribution of rubric-based validation is not sufficiently explored. The current no-rubric baseline removes both rubric-guided repair and rubric-based validation, making it difficult to isolate their individual contributions.
- The presentation could be improved, particularly Figures.
Questions for authors
See Weaknesses. If the authors can address these concerns through further clarification or additional experiments, I would be willing to raise my score accordingly.
Follow-up on our response
Dear Reviewer kmxR,
Thank you again for your constructive feedback. With less than four hours remaining before the deadline, we wanted to kindly follow up on our responses and the revised manuscript.
In response to your concerns, we added evaluations on four held-out multi-hop QA benchmarks and a second model family, an ablation that removes only the LLM validator while retaining rubric-guided repair, and a discussion of published agent-RL results with explicit caveats about differences in evaluation settings. These additions are detailed in our responses above.
We would greatly appreciate any feedback on whether these additions address your main concerns, or whether there are remaining issues we could clarify before the deadline. We understand that this is a busy period and would be grateful even for a brief update.
Thank you for your time and consideration.