acceptodds
Under review as a conference paper at ICLR 2027

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

Abstract

AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries. Analyzing such software requires reverse engineering (RE): recovering program semantics before the analysis can be meaningfully performed. However, evaluating agentic RE poses a fundamental challenge: realistic and useful benchmark instances must (1) be absent from LLMs’ training data to prevent models from taking shortcuts by memorization, and (2) represent the scale and anti-analysis protections of real-world binaries. To faithfully evaluate the RE capability of frontier models, we introduce SRE-Bench, the first realistic, contamination-free RE benchmark. Built entirely from scratch by RE experts with over 5,000 expert hours, SRE-Bench comprises 19 private, real-world-scale programs averaging 16.9K lines of code. We further developed 44 in-house anti-analysis mechanisms, yielding 262 binary instances and 1,572 deterministically graded tasks. We evaluated 13 agentic settings across 11 models: eight in public-facing settings and five in internal unconstrained settings with cyber safeguards disabled and no budget cap. Our results show that realistic RE remains challenging for frontier agents: GPT-5.6-Sol and Claude-Fable-5.1, despite their strong source-code cybersecurity capabilities, fully solve only 31.5% and 26.9% of graded instances, respectively. This result suggests that success in source-code security does not necessarily translate into effective binary analysis. Without a budget cap and safety guard, GPT-6-Astra achieves a near-perfect pass@4 score; however, reliably identifying the correct candidate still poses a significant challenge. Our analysis further reveals that agents are relatively insensitive to compiler optimization and static linking, and controlled ablations confirm that both contamination control and realistic scale are essential to understand agents’ RE capability. Together, these findings highlight RE as a distinct frontier for agentic cybersecurity and establish SRE-Bench as a rigorous testbed for measuring progress.

Primary area:
datasets and benchmarks
Keywords:
Reverse Engineering, Cybersecurity, Benchmark, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fbVCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SRE-Bench for evaluating agentic reverse engineering on privately developed binary programs. The benchmark contains 19 programs across five domains and four implementation languages. Compilation settings and an in-house suite of 44 protection primitives produce 262 binary instances, each with six deterministically graded objectives. The evaluation covers standardized public-facing agents and provider-internal configurations, with analyses of build settings, protection mechanisms, reasoning effort, and repeated attempts. The results show substantial differences in binary-analysis capability and distinguish generating a correct solution from reliably selecting one.

Critical strengths
  1. The benchmark makes a substantial contribution to evaluating binary-level program understanding. Its original programs are considerably larger than typical toy RE tasks, and the protection suite introduces layered obstacles beyond simple obfuscation. The tasks require concrete outcomes such as protocol recovery, archive decoding, targeted malware cleanup, and firmware control. Private reference solutions and adversarial shortcut audits further support task solvability and the validity of the grading design.

  2. The experimental analysis goes beyond reporting a model ranking. The standardized public evaluation uses a common scaffold, and the build and protection analyses examine factors that affect RE performance. The differences between symbol stripping, optimization, and protection are informative. The authors also clearly distinguish scores conditional on gradable runs and explain why high pass@4 does not establish reliable solution selection. These qualifications make the empirical findings easier to interpret.

Critical weaknesses
  1. The contamination ablation provides suggestive evidence, but does not isolate the effect of prior training exposure. Table 5 compares three different compressor implementations, which differ in provenance, complexity, and program structure. The performance gap could therefore reflect differences in the underlying analysis task as well as recognition of public code. The clean-room construction remains well motivated, but the stronger claim that this experiment confirms contamination-induced performance inflation should be qualified or supported by additional evidence from the execution traces.

  2. External verification of the benchmark remains insufficiently specified. Retaining private instances is a reasonable way to reduce future contamination, and the proposed evaluation service could support continued comparisons. However, the paper does not explain how external researchers can access a versioned evaluation, inspect sufficiently detailed results, or independently assess grader correctness without exposing task secrets. Since the benchmark is the main contribution, a concrete access and audit procedure would strengthen confidence in its reproducibility and long-term research value.

Questions for authors
  1. Do the Gzip-variant execution traces show agents recognizing the implementation or reusing known program details? Please clarify what evidence attributes its lower difficulty to prior exposure, rather than differences in algorithms, structure, or task complexity.

  2. How can external researchers access the evaluation service and obtain versioned configurations and per-instance outcomes? Is independent inspection of the grading pipeline possible under restricted access while preserving the confidentiality of the benchmark instances?

4 comments

We thank the reviewer for the thoughtful feedback. We have conducted additional analyses to address the concerns raised, which we present below. We also provide further details on external access and grading verification. We will incorporate these findings and clarifications into the revised paper.

AuthorsOpenReview ↗

Q1. Contamination and scale ablation

We thank the reviewer for the thoughtful feedback. To better isolate the effects of program size, potential training-data contamination, and inherent task difficulty, we conducted a more fine-grained, controlled ablation study as our quantitative analysis.

Recognizing that these factors are difficult to disentangle completely, we also examined agent trajectories as our qualitative analysis to better understand how familiarity with public implementations may affect performance.

Quantitative analysis

We constructed four variants with progressively different implementations and functionality:

  1. Modified gzip: The original gzip implementation, with changes to its Huffman coder and selected constants.
  2. gzip with a custom coder: Retains gzip's surrounding file-format handling and program structure, but replaces its coder with RevCompress's in-house context-mixing (CM) coder.
  3. Minimal RevCompress: Retains RevCompress's file format and surrounding structure, but includes only the CM coder, removing the other coders and functionality.
  4. Full RevCompress: The complete in-house implementation, including all coders and functionality.

Variants (1) - (3) provide a more fine-grained comparison as the potential influence of training-data contamination gradually decreases. Comparing variants (3) and (4) further allows us to examine the effects of additional program functionality and complexity. All variants were given the same task and grader.

We evaluated GPT-5.6 Sol on 8 compilation variants of each program, obtaining the following results:

VariantMean solving stepsResult
(1) Modified gzip17.8All 8 fully solved
(2) gzip with custom coder30.1All 8 fully solved
(3) Minimal RevCompress45.8All 8 fully solved
(4) Full RevCompress199.37/8 variants fully solved; remaining variant scored 3/6

The gradual increase in solving steps across variants (1) - (3) is consistent with a diminishing benefit from training-data contamination. The substantial increase from (3) to (4) further suggests that additional program functionality and complexity contribute to the difficulty.

We acknowledge that these comparisons do not perfectly isolate the individual factors, particularly because program scale and functionality are closely related. We will clarify this limitation and avoid attributing the observed differences to a single cause.

Trajectory Analysis

We further examined the agent trajectories for variants (1) and (2) to understand how prior knowledge of public implementations may influence the solving process. The trajectories provide additional evidence of why the solving steps differ and motivate the use of private programs in our benchmark.

For variant (1), we observed several notable behaviors:

  • In every trajectory, within the agent’s first 14 steps it identified and recognized gzip/zlib.

  • Even in symbol-stripped builds, agents used zlib-specific identifiers such as bl_order, send_tree, and trees.c, which were not available as symbols in the binary.

  • In most trajectories, agents reproduced DEFLATE's stock Huffman code-length constants from memory, before any of these constants appeared in a tool output, e.g., order=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15] … 'HLIT',val(3,5)+257,'HDIST',val(8,5)+1,'HCLEN',val(13,4)+4

  • In one of the trajectories after finding the standard order missing, the agent describes the transformation in contrast to RFC 1951.

One representative trajectory stated:

The outer layer is standard gzip, so the main unknown is likely the uncompressed payload/container rather than the entropy codec itself.

For variant (2), we observed that agents initially attempted similar approaches based on their knowledge of gzip/DEFLATE. However, when these approaches failed, they shifted to direct binary analysis.

One representative trajectory stated:

The compressed blocks are byte-framed rather than DEFLATE bitstreams. … I’m decompiling only the small non-standard region that constructs those two block types, which should give an exact inverse without needing to infer it statistically.

We will include these qualitative findings and the corresponding discussion in the revised paper.

AuthorsOpenReview ↗

Q2. External access and grading verification

Thank you for raising this concern.

First, we would like to reiterate why we keep the benchmark private. As we have explained and justified in the paper, even partial exposure of a program's identity or high-level functionality can allow models to bypass substantial reverse-engineering effort. Publicly releasing our instances would risk their inclusion in the training data of future models, thereby undermining the benchmark's long-term validity. We therefore consider preserving private evaluation instances essential to SRE-Bench's practical value.

While we make efforts to avoid future data contamination, we also welcome external evaluation: we have already supported private evaluations for external researchers. Researchers were able to provide their agent harness and/or model API, and we run the evaluation and return per-instance scores. To date, we have conducted such evaluations for eight external companies and research groups.

To ensure the credibility and independent verifiability of SRE-Bench grading infrastructure while preserving its contamination-free nature, we have developed separate sample tasks using different binaries but the same grading infrastructure, which we will release publicly. This infrastructure has already been independently tested by five companies.

Additionally, to ensure the correctness and solvability of each challenge, we conducted a separate validation by providing agents (GPT-5.6 Sol in this case) with the corresponding source code. All challenges were fully solved by these source-access agents.

We will clarify these procedures in the revised paper.

Thanks for your detailed reply. I will maintain my rating.

Reviewer_KNu6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SRE-Bench, a realistic, contamination-free benchmark for evaluating AI agents' reverse-engineering capabilities. The benchmark consists of 262 binary instances derived from 19 independently developed programs across five domains, incorporating 44 anti-analysis mechanisms and 1,572 deterministically graded tasks. Through extensive evaluations of frontier AI models, the authors show that strong source-code cybersecurity capabilities do not necessarily translate to effective binary reverse engineering. The results also highlight how different compilation settings and binary protection mechanisms affect agent performance, providing useful insights into the current capabilities and limitations of agentic reverse engineering.

Critical strengths
  1. Presentation and Soundness: The paper is well-written, clearly structured, and easy to follow. I particularly liked how the authors motivate the need for a realistic reverse-engineering benchmark and connect it to the limitations of existing evaluations. The overall methodology is technically sound, and the benchmark construction and evaluation procedures are explained in sufficient detail.

  2. Realistic and Comprehensive Benchmark: Building 19 programs from scratch, with an average of around 16.9K lines of code, is a substantial effort. I appreciate the authors' attention to realistic program complexity, diverse RE domains, and the 44 anti-analysis mechanisms. The deterministic grading and adversarial reward-hacking audits are also valuable additions that strengthen the benchmark's reliability.

  3. Extensive Evaluation and Interesting Findings: The evaluation covers a wide range of frontier models under both public-facing and unconstrained settings. I found the analysis of different compilation configurations and protection mechanisms particularly interesting, especially the finding that symbol stripping and binary hardening affect models much more than compiler optimization or static linking. These results provide useful insights into the current strengths and limitations of agentic reverse engineering.

Critical weaknesses

Minor Concerns:

  1. I appreciate the authors' effort to control data contamination, but I am not fully convinced that the ablation in Table 5 demonstrates its impact on RE performance. The three programs differ in size, implementation, and potentially inherent difficulty, making it hard to isolate the effect of contamination. For example, both the clean-room Minimal program and the public-code-derived Gzip-variant are solved perfectly, while the larger clean-room RevCompress is significantly harder. To me, these results provide stronger evidence for the importance of program complexity than for the impact of contamination itself. I would have liked to see a more controlled ablation that separates these two factors and better supports the authors' claim that contamination inflates RE performance.

  2. The related work section provides a good overview of existing reverse-engineering benchmarks, but I think it could be strengthened by discussing a few additional relevant works. In particular, ReSym (CCS 2024), LLM4Decompile (EMNLP 2024), and PalmTree (CCS 2021) are important contributions to learning-based binary analysis and semantic recovery that would help better contextualize SRE-Bench. Additionally, CyberSecEval 2 (2024) and JAWS-Bench (TACL 2026) explore complementary aspects of cybersecurity evaluation, including security capabilities and vulnerabilities of AI agents. Citing them would provide a more comprehensive picture of the existing literature and better position SRE-Bench within the broader landscape of agentic cybersecurity research.

Questions for authors

N/A

5 comments

We thank the reviewer for the thoughtful and constructive feedback. We have conducted additional analyses and expanded our discussion of related work to address the concerns raised below. We will incorporate these findings and discussions into the revised paper.

AuthorsOpenReview ↗

Q1: Impact of data contamination (Table 5)

We thank the reviewer for the thoughtful feedback. To better isolate the effects of program size, potential training-data contamination, and inherent task difficulty, we conducted a more fine-grained, controlled ablation study as our quantitative analysis.

Recognizing that these factors are difficult to disentangle completely, we also examined agent trajectories as our qualitative analysis to better understand how familiarity with public implementations may affect performance.

Quantitative analysis

We constructed four variants with progressively different implementations and functionality:

  1. Modified gzip: The original gzip implementation, with changes to its Huffman coder and selected constants.
  2. gzip with a custom coder: Retains gzip's surrounding file-format handling and program structure, but replaces its coder with RevCompress's in-house context-mixing (CM) coder.
  3. Minimal RevCompress: Retains RevCompress's file format and surrounding structure, but includes only the CM coder, removing the other coders and functionality.
  4. Full RevCompress: The complete in-house implementation, including all coders and functionality.

Variants (1) - (3) provide a more fine-grained comparison as the potential influence of training-data contamination gradually decreases. Comparing variants (3) and (4) further allows us to examine the effects of additional program functionality and complexity. All variants were given the same task and grader.

We evaluated GPT-5.6 Sol on 8 compilation variants of each program, obtaining the following results:

VariantMean solving stepsResult
(1) Modified gzip17.8All 8 fully solved
(2) gzip with custom coder30.1All 8 fully solved
(3) Minimal RevCompress45.8All 8 fully solved
(4) Full RevCompress199.37/8 variants fully solved; remaining variant scored 3/6

The gradual increase in solving steps across variants (1) - (3) is consistent with a diminishing benefit from training-data contamination. The substantial increase from (3) to (4) further suggests that additional program functionality and complexity contribute to the difficulty.

We acknowledge that these comparisons do not perfectly isolate the individual factors, particularly because program scale and functionality are closely related. We will clarify this limitation and avoid attributing the observed differences to a single cause.

Trajectory Analysis

We further examined the agent trajectories for variants (1) and (2) to understand how prior knowledge of public implementations may influence the solving process. The trajectories provide additional evidence of why the solving steps differ and motivate the use of private programs in our benchmark.

For variant (1), we observed several notable behaviors:

  • In every trajectory, within the agent’s first 14 steps it identified and recognized gzip/zlib.

  • Even in symbol-stripped builds, agents used zlib-specific identifiers such as bl_order, send_tree, and trees.c, which were not available as symbols in the binary.

  • In most trajectories, agents reproduced DEFLATE's stock Huffman code-length constants from memory, before any of these constants appeared in a tool output, e.g., order=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15] … 'HLIT',val(3,5)+257,'HDIST',val(8,5)+1,'HCLEN',val(13,4)+4

  • In one of the trajectories after finding the standard order missing, the agent describes the transformation in contrast to RFC 1951.

One representative trajectory stated:

The outer layer is standard gzip, so the main unknown is likely the uncompressed payload/container rather than the entropy codec itself.

For variant (2), we observed that agents initially attempted similar approaches based on their knowledge of gzip/DEFLATE. However, when these approaches failed, they shifted to direct binary analysis.

One representative trajectory stated:

The compressed blocks are byte-framed rather than DEFLATE bitstreams. … I’m decompiling only the small non-standard region that constructs those two block types, which should give an exact inverse without needing to infer it statistically.

We will include these qualitative findings and the corresponding discussion in the revised paper.

AuthorsOpenReview ↗

Q2: Additional related work

Thank you for these helpful suggestions! We agree that expanding the related work would better position SRE-Bench within the development of AI-based binary analysis and cybersecurity evaluation.

We will first expand our discussion of AI-based binary analysis, covering learning-based disassembly (XDA, NDSS'21; DeepDi, Security'22; Tady, Security'25; DISA, CCS'25), binary representation and similarity (Gemini, CCS'17; PalmTree, CCS'21; jTrans, ISSTA'22), and semantic recovery and decompilation (DIRTY, Security'22; ReSym, CCS'24; LLM4Decompile, EMNLP'24). We will also discuss AI-based obfuscation and deobfuscation, including ALMOND (ISSTA'25) and CodeSteer (ASE'26), to better contextualize the challenges posed by obfuscation.

While closely related to our work, these two lines of research primarily focus on developing AI-based binary analysis techniques. SRE-Bench takes a complementary direction, focusing on evaluating the end-to-end reverse-engineering capabilities of AI agents on realistic-scale, contamination-free binary programs. We fully agree that these works are valuable for illustrating the evolution of AI-based binary analysis and providing important context for understanding today's agentic reverse-engineering capabilities. We appreciate the reviewer's suggestion and will provide a more detailed discussion of these related works in the revised paper.

Additionally, we will broaden our discussion of cybersecurity evaluation to include CyberSecEval 2 (2024), JAWS-Bench (TACL'26), and CyberDual (arXiv'26). These benchmarks examine related aspects of AI cybersecurity, including security risks, misuse, and attack/defense capabilities. While SRE-Bench focuses more specifically on the end-to-end reverse-engineering and program comprehension capabilities, these works provide valuable complementary perspectives on evaluating AI capabilities and their cybersecurity implications. We will provide a more detailed discussion in the revised paper.

Given the breadth of this research area, we would also welcome any further suggestions from the reviewer. Thank you again for helping us improve the paper!

Reviewer_bTYGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SRE-Bench, a reverse-engineering benchmark built from 19 privately developed programs across five domains, with 44 protection primitives, 262 binary instances, and 1,572 deterministically graded tasks. It evaluates 13 agentic settings across 11 models, distinguishing standardized public-facing evaluations from unconstrained provider evaluations. The analyses examine build configurations, protection mechanisms, reasoning effort, repeated attempts, and the effects of program provenance and scale.

Critical strengths
  1. The benchmark represents substantial engineering work on an important evaluation gap: recovering semantics from unfamiliar binaries. Private program construction, deterministic grading, reference solutions, and adversarial reward-hacking audits provide a stronger basis for assessing reverse engineering than small challenge-style tasks alone.

  2. The evaluation covers multiple domains, implementation languages, and protection configurations, and reports resource usage alongside performance. The explicit distinction between single-attempt success and oracle pass@4 usefully separates reliable execution from the ability to produce a correct solution across repeated attempts.

Critical weaknesses
  1. The public-facing comparison conditions capability scores on gradable runs, but the included instances differ substantially across models (181-262 of 262 in Table 4). Although disclosed, this can entangle capability differences with which tasks trigger refusals or context-window failures, and complicates the build and protection comparisons. Performance over all attempted instances, separate failure categories, and matched-instance comparisons are needed to assess the robustness of these conclusions.

  2. The contamination and scale ablation does not isolate the proposed causes. Table 5 compares three different compression programs, varying implementation and intrinsic task difficulty alongside program size and presumed training exposure. Holding the build configurations fixed does not remove these confounds. Stronger matched controls, or more qualified conclusions, are needed to substantiate the claimed effects of contamination and scale.

Questions for authors
  1. How do the public-facing success rates and estimated effects of hardening, symbol stripping, and linking change when reporting all attempted instances and comparing matched subsets? Please separate safeguard refusals from context-window failures.

  2. Can the authors provide semantically matched controls for the contamination and scale analysis, or clarify which conclusions Table 5 supports independently of differences among the three target programs?

5 comments

We thank the reviewer for the thoughtful feedback. We have conducted additional analyses to address the reviewer's concerns, which we present in the following messages. We will also incorporate these findings into the revised paper.

AuthorsOpenReview ↗

Q1. Non-gradable runs and matched-instance comparisons

Thank you for this suggestion. We first report the distribution of non-gradable runs, separating safeguard refusals from context-window and timeout failures.

As shown below, only Grok 4.5 encountered context-window or timeout failures. All other non-gradable runs resulted from safeguard refusals.

ModelGraded / 262Safeguard refusalsContext window / timeout
GPT-6 Astra181810
GPT-5.6 Sol25480
Claude Fable 5.1223390
Claude Opus 525660
GPT-5.526200
DeepSeek V4.1 Flash26200
Grok 4.5218044
GLM-5.226200

We further report model performance under three additional evaluation settings:

  • (A) All 262 instances, counting safeguard refusals and other failures as zero scores.
  • (B) 96 matched instances successfully graded for all eight models.
  • (C) 135 matched instances successfully graded for the top-5 models.

These additional comparisons allow us to assess performance on identical instances: overall, the relative model ranking remains unchanged across all three settings.

(A) All 262 instances, with non-gradable runs counted as zero

ModelScore / 6 (%)Solved # (% of 262)Zeros # (%)
GPT-6 Astra3.83 (63.9)149 (56.9)88 (33.6)
GPT-5.6 Sol3.57 (59.5)80 (30.5)49 (18.7)
Claude Fable 5.12.20 (36.6)60 (22.9)135 (51.5)
Claude Opus 51.87 (31.1)32 (12.2)124 (47.3)
GPT-5.51.02 (17.0)10 (3.8)148 (56.5)
DeepSeek V4.1 Flash0.85 (14.2)2 (0.8)166 (63.4)
Grok 4.50.38 (6.3)2 (0.8)204 (77.9)
GLM-5.20.21 (3.4)0 (0.0)217 (82.8)
ModelUnopt.Opt.Sym.Strip.Dyn.StaticUnhard.Hard.
GPT-6 Astra3.734.083.883.943.734.084.383.94
GPT-5.6 Sol4.754.674.954.474.734.694.692.37
Claude Fable 5.14.254.054.274.034.124.174.000.27
Claude Opus 53.393.163.533.023.383.172.880.33
GPT-5.52.001.672.531.141.841.830.940.11
DeepSeek V4.1 Flash1.841.382.201.021.611.610.810.08
Grok 4.50.860.470.920.410.720.610.310.02
GLM-5.20.390.380.590.170.390.380.250.00

(B) 96 matched instances graded for all eight models

ModelScore / 6 (%)Solved # (% of 96)Zeros # (%)
GPT-6 Astra5.31 (88.5)74 (77.1)6 (6.2)
GPT-5.6 Sol4.14 (68.9)56 (58.3)19 (19.8)
Claude Fable 5.12.92 (48.6)42 (43.8)39 (40.6)
Claude Opus 52.47 (41.1)24 (25.0)37 (38.5)
GPT-5.51.27 (21.2)6 (6.2)48 (50.0)
DeepSeek V4.1 Flash1.22 (20.3)2 (2.1)51 (53.1)
Grok 4.50.52 (8.7)1 (1.0)67 (69.8)
GLM-5.20.35 (5.9)0 (0.0)69 (71.9)
ModelUnopt.Opt.Sym.Strip.Dyn.StaticUnhard.Hard.
Instances332229263124641
GPT-6 Astra4.915.325.035.124.815.425.675.63
GPT-5.6 Sol5.365.645.835.085.455.505.672.34
Claude Fable 5.14.585.365.414.314.235.756.000.27
Claude Opus 53.704.594.343.733.814.384.330.34
GPT-5.52.242.092.931.352.292.041.000.05
DeepSeek V4.1 Flash2.032.142.521.582.102.041.330.07
Grok 4.51.030.731.140.650.940.880.500.00
GLM-5.20.580.680.860.350.580.670.500.00

(C) 135 matched instances graded for the five highest-performing models

ModelScore / 6 (%)Solved # (% of 135)Zeros # (%)
GPT-6 Astra5.51 (91.9)113 (83.7)6 (4.4)
GPT-5.6 Sol4.01 (66.9)68 (50.4)22 (16.3)
Claude Fable 5.12.76 (45.9)57 (42.2)61 (45.2)
Claude Opus 52.13 (35.6)27 (20.0)57 (42.2)
GPT-5.51.00 (16.7)6 (4.4)74 (54.8)
ModelUnopt.Opt.Sym.Strip.Dyn.StaticUnhard.Hard.
Instances393133373832865
GPT-6 Astra5.085.525.155.385.035.565.755.77
GPT-5.6 Sol5.315.325.854.845.215.445.502.62
Claude Fable 5.14.795.555.484.814.555.816.000.20
Claude Opus 53.674.034.183.513.554.164.120.31
GPT-5.52.001.742.701.162.031.720.880.05

We will include these additional results in the revised paper.

AuthorsOpenReview ↗

Q2. Contamination and scale ablation

We thank the reviewer for the thoughtful feedback. To better isolate the effects of program size, potential training-data contamination, and inherent task difficulty, we conducted a more fine-grained, controlled ablation study as our quantitative analysis.

Recognizing that these factors are difficult to disentangle completely, we also examined agent trajectories as our qualitative analysis to better understand how familiarity with public implementations may affect performance.

Quantitative analysis

We constructed four variants with progressively different implementations and functionality:

  1. Modified gzip: The original gzip implementation, with changes to its Huffman coder and selected constants.
  2. gzip with a custom coder: Retains gzip's surrounding file-format handling and program structure, but replaces its coder with RevCompress's in-house context-mixing (CM) coder.
  3. Minimal RevCompress: Retains RevCompress's file format and surrounding structure, but includes only the CM coder, removing the other coders and functionality.
  4. Full RevCompress: The complete in-house implementation, including all coders and functionality.

Variants (1) - (3) provide a more fine-grained comparison as the potential influence of training-data contamination gradually decreases. Comparing variants (3) and (4) further allows us to examine the effects of additional program functionality and complexity. All variants were given the same task and grader.

We evaluated GPT-5.6 Sol on 8 compilation variants of each program, obtaining the following results:

VariantMean solving stepsResult
(1) Modified gzip17.8All 8 fully solved
(2) gzip with custom coder30.1All 8 fully solved
(3) Minimal RevCompress45.8All 8 fully solved
(4) Full RevCompress199.37/8 variants fully solved; remaining variant scored 3/6

The gradual increase in solving steps across variants (1) - (3) is consistent with a diminishing benefit from training-data contamination. The substantial increase from (3) to (4) further suggests that additional program functionality and complexity contribute to the difficulty.

We acknowledge that these comparisons do not perfectly isolate the individual factors, particularly because program scale and functionality are closely related. We will clarify this limitation and avoid attributing the observed differences to a single cause.

Trajectory Analysis

We further examined the agent trajectories for variants (1) and (2) to understand how prior knowledge of public implementations may influence the solving process. The trajectories provide additional evidence of why the solving steps differ and motivate the use of private programs in our benchmark.

For variant (1), we observed several notable behaviors:

  • In every trajectory, within the agent’s first 14 steps it identified and recognized gzip/zlib.

  • Even in symbol-stripped builds, agents used zlib-specific identifiers such as bl_order, send_tree, and trees.c, which were not available as symbols in the binary.

  • In most trajectories, agents reproduced DEFLATE's stock Huffman code-length constants from memory, before any of these constants appeared in a tool output, e.g., order=[16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15] … 'HLIT',val(3,5)+257,'HDIST',val(8,5)+1,'HCLEN',val(13,4)+4

  • In one of the trajectories after finding the standard order missing, the agent describes the transformation in contrast to RFC 1951.

One representative trajectory stated:

The outer layer is standard gzip, so the main unknown is likely the uncompressed payload/container rather than the entropy codec itself.

For variant (2), we observed that agents initially attempted similar approaches based on their knowledge of gzip/DEFLATE. However, when these approaches failed, they shifted to direct binary analysis.

One representative trajectory stated:

The compressed blocks are byte-framed rather than DEFLATE bitstreams. … I’m decompiling only the small non-standard region that constructs those two block types, which should give an exact inverse without needing to infer it statistically.

We will include these qualitative findings and the corresponding discussion in the revised paper.

Comments on the paper

2 comments

Dear Reviewers,

With the discussion period closing on November 18, please take a moment to read the authors’ responses. Please also indicate whether any further clarification or follow-up is needed.

Thank you for your time and contributions!
AC

AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank all three reviewers for their constructive feedback and thoughtful discussions.

A central concern shared by all three reviewers was the ablation study. In response, we have redesigned the experiments to isolate the effects of data contamination and program scale, combining quantitative results with detailed analyses of agent trajectories. These new results provide a much clearer understanding of the factors affecting agent performance. We are pleased that all three reviewers have acknowledged these improvements in their follow-up discussions.

We also addressed each reviewer's individual concerns, including clarifying matched-instance comparisons, expanding the discussion of related work, and providing additional details on external verification. We are glad that these clarifications were also positively received.

We have uploaded the revised manuscript, with all changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.