- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SRE-Bench for evaluating agentic reverse engineering on privately developed binary programs. The benchmark contains 19 programs across five domains and four implementation languages. Compilation settings and an in-house suite of 44 protection primitives produce 262 binary instances, each with six deterministically graded objectives. The evaluation covers standardized public-facing agents and provider-internal configurations, with analyses of build settings, protection mechanisms, reasoning effort, and repeated attempts. The results show substantial differences in binary-analysis capability and distinguish generating a correct solution from reliably selecting one.
Critical strengths
-
The benchmark makes a substantial contribution to evaluating binary-level program understanding. Its original programs are considerably larger than typical toy RE tasks, and the protection suite introduces layered obstacles beyond simple obfuscation. The tasks require concrete outcomes such as protocol recovery, archive decoding, targeted malware cleanup, and firmware control. Private reference solutions and adversarial shortcut audits further support task solvability and the validity of the grading design.
-
The experimental analysis goes beyond reporting a model ranking. The standardized public evaluation uses a common scaffold, and the build and protection analyses examine factors that affect RE performance. The differences between symbol stripping, optimization, and protection are informative. The authors also clearly distinguish scores conditional on gradable runs and explain why high pass@4 does not establish reliable solution selection. These qualifications make the empirical findings easier to interpret.
Critical weaknesses
-
The contamination ablation provides suggestive evidence, but does not isolate the effect of prior training exposure. Table 5 compares three different compressor implementations, which differ in provenance, complexity, and program structure. The performance gap could therefore reflect differences in the underlying analysis task as well as recognition of public code. The clean-room construction remains well motivated, but the stronger claim that this experiment confirms contamination-induced performance inflation should be qualified or supported by additional evidence from the execution traces.
-
External verification of the benchmark remains insufficiently specified. Retaining private instances is a reasonable way to reduce future contamination, and the proposed evaluation service could support continued comparisons. However, the paper does not explain how external researchers can access a versioned evaluation, inspect sufficiently detailed results, or independently assess grader correctness without exposing task secrets. Since the benchmark is the main contribution, a concrete access and audit procedure would strengthen confidence in its reproducibility and long-term research value.
Questions for authors
-
Do the Gzip-variant execution traces show agents recognizing the implementation or reusing known program details? Please clarify what evidence attributes its lower difficulty to prior exposure, rather than differences in algorithms, structure, or task complexity.
-
How can external researchers access the evaluation service and obtain versioned configurations and per-instance outcomes? Is independent inspection of the grading pipeline possible under restricted access while preserving the confidentiality of the benchmark instances?
4 comments
Q1. Contamination and scale ablation
We thank the reviewer for the thoughtful feedback. To better isolate the effects of program size, potential training-data contamination, and inherent task difficulty, we conducted a more fine-grained, controlled ablation study as our quantitative analysis.
Recognizing that these factors are difficult to disentangle completely, we also examined agent trajectories as our qualitative analysis to better understand how familiarity with public implementations may affect performance.
Quantitative analysis
We constructed four variants with progressively different implementations and functionality:
- Modified gzip: The original gzip implementation, with changes to its Huffman coder and selected constants.
- gzip with a custom coder: Retains gzip's surrounding file-format handling and program structure, but replaces its coder with RevCompress's in-house context-mixing (CM) coder.
- Minimal RevCompress: Retains RevCompress's file format and surrounding structure, but includes only the CM coder, removing the other coders and functionality.
- Full RevCompress: The complete in-house implementation, including all coders and functionality.
Variants (1) - (3) provide a more fine-grained comparison as the potential influence of training-data contamination gradually decreases. Comparing variants (3) and (4) further allows us to examine the effects of additional program functionality and complexity. All variants were given the same task and grader.
We evaluated GPT-5.6 Sol on 8 compilation variants of each program, obtaining the following results:
| Variant | Mean solving steps | Result |
|---|---|---|
| (1) Modified gzip | 17.8 | All 8 fully solved |
| (2) gzip with custom coder | 30.1 | All 8 fully solved |
| (3) Minimal RevCompress | 45.8 | All 8 fully solved |
| (4) Full RevCompress | 199.3 | 7/8 variants fully solved; remaining variant scored 3/6 |
The gradual increase in solving steps across variants (1) - (3) is consistent with a diminishing benefit from training-data contamination. The substantial increase from (3) to (4) further suggests that additional program functionality and complexity contribute to the difficulty.
We acknowledge that these comparisons do not perfectly isolate the individual factors, particularly because program scale and functionality are closely related. We will clarify this limitation and avoid attributing the observed differences to a single cause.
Trajectory Analysis
We further examined the agent trajectories for variants (1) and (2) to understand how prior knowledge of public implementations may influence the solving process. The trajectories provide additional evidence of why the solving steps differ and motivate the use of private programs in our benchmark.
For variant (1), we observed several notable behaviors:
-
In every trajectory, within the agent’s first 14 steps it identified and recognized gzip/zlib.
-
Even in symbol-stripped builds, agents used zlib-specific identifiers such as
bl_order,send_tree, andtrees.c, which were not available as symbols in the binary. -
In most trajectories, agents reproduced DEFLATE's stock Huffman code-length constants from memory, before any of these constants appeared in a tool output, e.g.,
order=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15] … 'HLIT',val(3,5)+257,'HDIST',val(8,5)+1,'HCLEN',val(13,4)+4 -
In one of the trajectories after finding the standard order missing, the agent describes the transformation in contrast to RFC 1951.
One representative trajectory stated:
The outer layer is standard gzip, so the main unknown is likely the uncompressed payload/container rather than the entropy codec itself.
For variant (2), we observed that agents initially attempted similar approaches based on their knowledge of gzip/DEFLATE. However, when these approaches failed, they shifted to direct binary analysis.
One representative trajectory stated:
The compressed blocks are byte-framed rather than DEFLATE bitstreams. … I’m decompiling only the small non-standard region that constructs those two block types, which should give an exact inverse without needing to infer it statistically.
We will include these qualitative findings and the corresponding discussion in the revised paper.
Q2. External access and grading verification
Thank you for raising this concern.
First, we would like to reiterate why we keep the benchmark private. As we have explained and justified in the paper, even partial exposure of a program's identity or high-level functionality can allow models to bypass substantial reverse-engineering effort. Publicly releasing our instances would risk their inclusion in the training data of future models, thereby undermining the benchmark's long-term validity. We therefore consider preserving private evaluation instances essential to SRE-Bench's practical value.
While we make efforts to avoid future data contamination, we also welcome external evaluation: we have already supported private evaluations for external researchers. Researchers were able to provide their agent harness and/or model API, and we run the evaluation and return per-instance scores. To date, we have conducted such evaluations for eight external companies and research groups.
To ensure the credibility and independent verifiability of SRE-Bench grading infrastructure while preserving its contamination-free nature, we have developed separate sample tasks using different binaries but the same grading infrastructure, which we will release publicly. This infrastructure has already been independently tested by five companies.
Additionally, to ensure the correctness and solvability of each challenge, we conducted a separate validation by providing agents (GPT-5.6 Sol in this case) with the corresponding source code. All challenges were fully solved by these source-access agents.
We will clarify these procedures in the revised paper.
Thanks for your detailed reply. I will maintain my rating.
We thank the reviewer for the thoughtful feedback. We have conducted additional analyses to address the concerns raised, which we present below. We also provide further details on external access and grading verification. We will incorporate these findings and clarifications into the revised paper.