- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies three failure modes: under-protection, over-refusal, and coordination breakdown. It introduces InterlockBench, a benchmark constructed within the AgentDyn framework. The authors also propose CONCORD, which jointly evolves complementary procedural memories based on shared failures. CONCORD improves task completion and achieves higher average joint success rates. The authors also report that the learned procedures transfer to new safety and general-purpose tasks.
Critical strengths
-
The paper identifies the coordination breakdown problem, which is practical and relevant to real-world agentic systems.
-
The paper highlights the importance of jointly evolving the executor and guard rather than developing them independently.
Critical weaknesses
-
Lack of repeated experiments. The experiments appear to have been run only once, without repeated trials. Given the relatively small test sets, it is difficult to assess the stability and reliability of the reported results.
-
Limited evidence of cross-environment generalization. InterlockBench appears to be constructed within the AgentDyn framework, which is also used to evolve CONCORD's procedural memories. Therefore, the strong performance on InterlockBench may partly result from similarities between the training and evaluation environments. This raises concerns about how well the results generalize to substantially different environments.
Questions for authors
-
Failure attribution. Is failure attribution necessary for CONCORD's performance improvements? Could the authors provide an ablation study without failure attribution to demonstrate its contribution?
-
Comparison with existing methods. Is the comparison with StepGuard and ToolSafe fair, given that they use their own guard models? Could the authors clarify how differences in the underlying guard models affect the comparison?
-
Fairness of the ablation study. Could the authors control for the same number of learning and evaluation rounds while varying which role (executor, guard, or both) is updated? The current setups appear to differ in their learning and evaluation procedures, making it difficult to isolate the benefits of joint evolution.
2 comments
Dear Reviewer 6tVx,
We have uploaded a revised PDF with changes highlighted in blue; the new Appendix H includes the analyses added in our responses. We would be grateful for any further feedback.
A Gentle Reminder
Dear Reviewer 6tVx,
Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.
We sincerely appreciate your time and valuable feedback.