- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is an empirical analysis paper on coding agent/framework’s ability to detect the CVE patch existence in the binary. The authors made three discoveries: when debug symbols are available, coding agents have similar performance compared to specialized agent system; stripping and build-configuration variation have limited impact on the performance; exposing additional binary-analysis interfaces introduces regressions and higher analysis cost, whereas enriched patch metadata raises performance at lower cost. It shows that coding agents are credible alternatives for some binary CVE patch-detection tasks and identifies which resources help them and which can make them worse.
Critical strengths
This paper aims to address an important question, in an emerging area to LLM and agent systems to understand the vulnerability, while patch detection will need a better and deeper understanding of the vulnerability and how to fix, which adds up the difficulty.
The authors highlight elements that improve the performance, showing that incorporating extra analysis toolkits fails to guarantee a better outcome, whereas supplying augmented patch context enhances identification accuracy and lowers computation expense, which is a good insight.
The evaluation is thorough, assessing the impact of binary-analysis toolkits, symbol presence, and varied compilation options on identification capabilities, and simple command-line tools can help agent on this task a lot.
Critical weaknesses
The evaluation does not provide sufficient investigation into how much the base LLM versus the agentic framework drives performance. Specifically, the reasons behind Codex achieving comparable or better results remain ambiguous, leaving open what distinct perspectives are gained relative to the ISSTA work (A Comprehensive Empirical Analysis of Patch Presence Testing: Capabilities, Limitations, and Paths Forward). The study lacks a finer-grained breakdown of toolcall step distributions, execution workflows, and specific failure modes. Furthermore, variations in foundational model strengths along with potential model- or harness-specific training could introduce confounding factors, complicating efforts to isolate whether performance gains truly stem from the agent framework itself.
This work doesn’t introduce a new dataset or benchmark approach, only tests established coding agents with existing dataset. This point doesn’t imply the insights are not helpful, but it raises a question, is the main contribution a new understanding of patch-presence testing, or is it evaluating another type of agent using an existing benchmark.
While the evaluation yields helpful empirical observations regarding coding agent efficacy in binary patch detection, the scope mostly expands upon existing evaluation setups like the recent ISSTA 2026 study rather than offering fundamentally new paradigms. Several takeaways, including the positive impact of enriched patch metadata and the counterproductive nature of extra analysis toolkits, are not surprising and mostly expected. Although these trends are empirically grounded, the work lacks a deeper breakdown of specific agent failure modes, toolcall step distributions, or distinct behavior patterns that would meaningfully push beyond established literature.
Questions for authors
Can you provide a more fine-grained analysis of how agents perform patch presence testing, such as intermediate steps, tool calls, and successful or failed trajectories, rather than only reporting aggregated results?
Can you isolate the influence of the base LLM from the agentic framework, especially regarding Codex, and clarify whether underlying model capabilities or harness-specific training factors might account for these performance gaps?