- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies origin attribution in self-evolving multi-agent coding systems, aiming to identify the specific agent and program version that first introduced an unsafe behavior. The authors contribute a controlled benchmark with manually injected faults and propose an attribution framework that traces unsafe behaviors through historical localization, behavioral matching, and recursive tracing.
Critical strengths
Timely Problem and Practical Importance: Studying safety and accountability in self-evolving multi-agent systems is a novel and promising research direction. Tracing the origin of unsafe behaviors across complex, multi-agent code evolution histories is a critical problem for ensuring the reliability of autonomous coding systems.
Critical weaknesses
Limited Methodological Novelty and Weak Baselines: The proposed framework (combining historical localization, behavioral matching, and recursive tracing) is relatively straightforward and lacks significant technical innovation. The evaluation lacks strong baselines designed for agent fault attribution.
Unrealistic Fault Injection and Limited Agent Autonomy: Unsafe behaviors are introduced via manual, direct fault injection rather than emerging naturally from the agents' self-evolution. This intervention undermines "self-evolving" and fails to reflect true multi-agent autonomy.
Limited Diversity of Risks and Tasks: The benchmark only covers a narrow set of fault types: literal-string routing bypasses, boundary errors, and field-substitution faults. Because these faults have distinct syntactic or behavioral signatures, they may be easy to detect and trace. Evaluating against a broader and more realistic spectrum of vulnerabilities (e.g., more CWE categories) is necessary to demonstrate generalizability.
Questions for authors
Cross-Agent Code Adoption and Autonomy: Under what exact conditions do agents adopt code from one another? In realistic multi-agent systems, agents autonomously decide when and how to integrate peers' code. How does your benchmark and attribution method account for this autonomy?
Generalizability to Diverse Vulnerabilities: Have you considered evaluating your method on a broader set of security risks, such as CWE vulnerabilities?