acceptodds
Under review as a conference paper at ICLR 2027

Understanding Coding Agents for Patch Presence Testing

88 views

Abstract

Patch presence testing determines whether a binary incorporates the security fix for a known vulnerability. Existing systems generally require symbols and cannot operate end to end on stripped binaries, limiting their practical applicability. Recent advances in large language models and coding agents make it possible to use agents directly for patch presence testing. We conduct a systematic empirical study of agent systems for end-to-end patch presence testing. Our findings show that: (1) with symbols available, a coding agent using only binutils tools achieves performance comparable to the state-of-the-art specialized system; (2) stripping and varying the build configurations have limited impact, whereas patch evolution remains challenging; (3) exposing additional binary-analysis interfaces introduces regressions and higher analysis cost, while enriched patch metadata raises performance at lower cost. Finally, we evaluate 12 combinations of four model configurations and three agent implementations, and derive a practical deployment recommendation that balances decision success and cost.

Primary area:
datasets and benchmarks
Keywords:
Patch Presence Testing, Coding Agents, Binary Analysis, Large Language Models, Software Security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5i9XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is an empirical analysis paper on coding agent/framework’s ability to detect the CVE patch existence in the binary. The authors made three discoveries: when debug symbols are available, coding agents have similar performance compared to specialized agent system; stripping and build-configuration variation have limited impact on the performance; exposing additional binary-analysis interfaces introduces regressions and higher analysis cost, whereas enriched patch metadata raises performance at lower cost. It shows that coding agents are credible alternatives for some binary CVE patch-detection tasks and identifies which resources help them and which can make them worse.

Critical strengths

This paper aims to address an important question, in an emerging area to LLM and agent systems to understand the vulnerability, while patch detection will need a better and deeper understanding of the vulnerability and how to fix, which adds up the difficulty.

The authors highlight elements that improve the performance, showing that incorporating extra analysis toolkits fails to guarantee a better outcome, whereas supplying augmented patch context enhances identification accuracy and lowers computation expense, which is a good insight.

The evaluation is thorough, assessing the impact of binary-analysis toolkits, symbol presence, and varied compilation options on identification capabilities, and simple command-line tools can help agent on this task a lot.

Critical weaknesses

The evaluation does not provide sufficient investigation into how much the base LLM versus the agentic framework drives performance. Specifically, the reasons behind Codex achieving comparable or better results remain ambiguous, leaving open what distinct perspectives are gained relative to the ISSTA work (A Comprehensive Empirical Analysis of Patch Presence Testing: Capabilities, Limitations, and Paths Forward). The study lacks a finer-grained breakdown of toolcall step distributions, execution workflows, and specific failure modes. Furthermore, variations in foundational model strengths along with potential model- or harness-specific training could introduce confounding factors, complicating efforts to isolate whether performance gains truly stem from the agent framework itself.

This work doesn’t introduce a new dataset or benchmark approach, only tests established coding agents with existing dataset. This point doesn’t imply the insights are not helpful, but it raises a question, is the main contribution a new understanding of patch-presence testing, or is it evaluating another type of agent using an existing benchmark.

While the evaluation yields helpful empirical observations regarding coding agent efficacy in binary patch detection, the scope mostly expands upon existing evaluation setups like the recent ISSTA 2026 study rather than offering fundamentally new paradigms. Several takeaways, including the positive impact of enriched patch metadata and the counterproductive nature of extra analysis toolkits, are not surprising and mostly expected. Although these trends are empirically grounded, the work lacks a deeper breakdown of specific agent failure modes, toolcall step distributions, or distinct behavior patterns that would meaningfully push beyond established literature.

Questions for authors

Can you provide a more fine-grained analysis of how agents perform patch presence testing, such as intermediate steps, tool calls, and successful or failed trajectories, rather than only reporting aggregated results?

Can you isolate the influence of the base LLM from the agentic framework, especially regarding Codex, and clarify whether underlying model capabilities or harness-specific training factors might account for these performance gaps?

Reviewer_wHjQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates coding agents for determining whether a known security patch is present in an ELF binary. Using four evaluation sets based on a prior patch-presence benchmark, it compares Codex with specialized systems on binaries with symbols, tests robustness to stripping, build variation, patch evolution, and Ubuntu packaging, and compares model–agent configurations by decision success rate (DSR) and cost. It reports high DSR in most settings, identifies patch localization as a common failure mode for weaker models, and finds that enriched patch metadata helps while the tested Ghidra interfaces do not.

Critical strengths

The paper studies a relevant practical task and evaluates stripped binaries rather than relying exclusively on supplied binary function locations. Its use of DSR appropriately counts inconclusive analyses as unsuccessful. These choices make the reported feasibility results useful, even though they do not resolve my concerns about the contribution’s significance.

Critical weaknesses
  1. Limited contribution. The evaluation draws largely on a prior benchmark and mostly tests existing agents. Strong systems already achieve 0.99 DSR with symbols and 0.95–0.98 in most other settings, so these experiments provide limited new insight. Patch evolution is harder (0.86 DSR), but receives little systematic analysis.
  2. Conclusions need stronger evidence. Enriched metadata combines several changes, making the source of its benefit unclear. The Ghidra result comes from one model tier and one tool interface. The paper reports no repeated-run results or uncertainty estimates, despite a 0.005 DSR difference between its recommended configurations.
Questions for authors

Can you separate the effects of the components of enriched metadata and show whether the metadata-versus-Ghidra result holds for stronger models? A systematic account of failures across the 73 patch-evolution cases would also materially affect my assessment of the paper’s insight.

Reviewer_K3XTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study of coding agents for Patch Presence Testing in binaries. The evaluation gives coding agents (Codex, MiniSWEAgent, and a custom harness called PPTAgent) standard binutils tools and checks whether they can locate and verify patches suggested by CVE metadata. The results show that such agents + binutils match existing symbolic tools, even when the binaries are stripped. Counterintuitively the results show that providing richer tools (such as the Ghidra decompiler) negatively impacts the accuracy.

Critical strengths
  • This is a practical case study in realistic settings. Most impressive is the result showing that general purpose agents reach similar accuracy to task-specific tools.

  • The ablations and the associated analyses provide new insights to this domain.

Critical weaknesses
  • Reported numbers are often confusing, as distinctions between patched/vulnerable and non applicable make direct comparison hard. It may be easier to completely separate the experiments (one experiment with should-be-patched software where the task is to detect patch presence; one experiment with both should-be-patched and not-affected software where the the task is to detect need of patching).

  • In terms of confusion, Section 5.1 talks about 13 "new failures" introduced by stripping the binaries, while Section 6.1 report a total of 12 failures on stripped binaries.

  • Terminology differs between the prompt (Appendix C -- "not_affected" and "absent") and the main body of the text ("not applicable" and "vulnerable"). I am assuming these two categories are the same.

  • Deeper root-cause analysis of the Ghidra failures would be interesting to determine (through measurements) whether it is an issue of context pollution/exhaustion, incorrect information, or other challenges.

Questions for authors
  • The PPTAgent describing in Section 7 talks about "read-only source guidance" as an input to the agent. Earlier sections talk about "bounded source guidance". Can you explain more what this means?

  • Does PPTAgent have access to function names?

  • Can you clarify on how thorough the stripped-binary dataset is? Is the patch still provided to the agent? Are names removed from the patch and the commit message?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.