acceptodds
Under review as a conference paper at ICLR 2027

BedrockBench: Can LLMs Speak Machine Code?

Abstract

Large language models achieve strong results in high-level code generation, but this success does not establish whether they can directly construct executable machine-code bytes. We introduce BedrockBench, the first benchmark for direct machine-code function synthesis from natural-language specifications, with 100 tasks targeting x86-64. Models submit hexadecimal function bytes without external tools or execution feedback. A shared evaluator executes these bytes unchanged and checks functional behavior, memory accesses, return state, and instruction and resource constraints across legal execution environments. Diagnostics distinguish submission failures, execution violations, and wrong answers. Across seven model configurations, pass@1 ranges from 13% to 100%, with GPT-6-Astra passing all 100 tasks. Among 240 format-valid unsuccessful candidates, 160 (66.7%) exhibit unauthorized memory accesses, making memory-contract violations a prominent observed failure mode. BedrockBench provides an explicit and auditable basis for comparing models and studying how to improve the reliability of direct machine-code synthesis. The benchmark data and code are publicly available.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmarking, code generation, machine code

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_amtWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BedrockBench, a benchmark for evaluating whether LLMs can directly generate executable x86-64 machine code from natural-language specifications without external tools or execution feedback. It adapts 100 LiveCodeBench problems into function-level tasks with explicit interfaces and execution constraints. The evaluator executes submitted bytes unchanged and checks functional correctness, memory accesses, ABI compliance, and instruction and resource limits under three machine environments. Across seven model configurations, reported pass@1 ranges from 13% to 100%. GPT-6-Astra passes all tasks, and all 93 format-valid Claude submissions succeed. The paper also analyzes failure categories and environment sensitivity, identifying unauthorized memory accesses as a common observed violation among format-valid unsuccessful submissions.

Critical strengths
  1. The evaluation target is clearly defined. The paper requires models to generate executable machine-code bytes directly, without external tools or execution feedback. This makes instruction encoding part of the evaluation and provides a concrete setting for studying low-level implementation capabilities beyond source-code generation.

  2. The evaluation protocol is carefully specified. In addition to functional correctness, the evaluator checks logical memory boundaries, ABI return state, instruction restrictions, and resource use. It also runs the same inputs under three legal machine environments. These checks help identify machine-level errors that output-only evaluation may miss.

  3. The results are interpreted with appropriate care. The paper distinguishes refusals, formatting errors, and execution failures, and explains that overlapping failure categories do not establish independent root causes. It also reports that environment variation adds diagnostic information without reducing task-level success in the available complete execution reports.

Critical weaknesses
  1. I have some concerns about the extent of the contribution, even when considering the paper primarily as a benchmark study. The problems are adapted from LiveCodeBench, so the main contribution lies in a new evaluation setting for existing problems, the adapted benchmark, and the corresponding results. The application scope is relatively narrow, and the current study does not clearly establish what additional insights direct machine-code generation provides into the limits of LLM capabilities.

  2. The empirical analysis could be strengthened to better support the purpose of the benchmark. For a LiveCodeBench problem, a model may fail because of errors specific to machine-code generation or because its algorithmic solution is incorrect. A comparison with the same models' source-code generation performance on the same tasks would help distinguish these factors and clarify what the benchmark measures. In addition, GPT-6-Astra already achieves a 100% pass rate, which raises concerns about the benchmark's ability to differentiate among stronger models.

  3. Task adaptation lacks independent validation. According to Sections 2.2 and 2.4, GPT-6-Astra participates in constructing the task interfaces, checkers, and resource limits, while semantic review is also performed by the constructing agent itself. Given that the same model is also evaluated on the benchmark, I believe human expert involvement is necessary.

Questions for authors

Please see weaknesses.

Reviewer_oNhROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BedrockBench, with x86-64 function-synthesis tasks adapted from LiveCodeBench release_v6. Models get a natural-language specification, examples, a System V ABI interface and an explicit execution contract. They must return raw hexadecimal machine-code bytes in a single response, with no tools and no feedback. Seven frontier configurations score from 13% (GLM-5.3) to 100% (GPT-6-Astra) pass@1.

Critical strengths
  • The benchmark is fully verifiable and carefully designed, covering multiple domains and with a carefully designed evaluator.
  • The paper introduces a new, clearly-defined problem that extends previous low-level code generation tasks.
Critical weaknesses
  • The benchmark was built and validated by the top-scoring model, and it is saturated at release. Since GPT 6 astra essentially built the benchmark, this confounds the result since the final tasks may favor Astra more. Additionally, Opus 5.5 also achieves 93/93. Thus, the benchmark does not provide any useful signals with it being saturated without human review.
  • There is no control separating machine-code skill from algorithmic-level skill. Additionally, the task selection (livecodebench) poses a hard limitation on the diversity of the programs.
Questions for authors
  • Did any human, or any model other than GPT-6-Astra, check the task specifications, checkers and budgets? Can you rule out that GPT-authored prompts favor GPT?
  • Of the 160 memory-violating candidates, how many would produce correct outputs under page-granular, hardware-like permissions?
  • Does the pipeline extend to other tasks like in SWE bench or other more repo-level benchmarks?
Reviewer_yen6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

BedrockBench evaluates direct synthesis of x86-64 machine code bytes from language specifications without external tools or execution feedback. It adapts 100 LiveCodeBench problems and introduces an evaluator checking functional correctness, memory accesses, and resource constraints across three execution environments. Validation includes conversion consistency checks and evaluator regression fixtures. Seven model configurations achieve 13–100% pass@1, unauthorized memory accesses occur in 160 of 240 valid unsuccessful candidates.

Critical strengths
  1. This paper requires executable bytes places instruction encoding and machine interface compliance directly on the model. This extends existing source code and generation evaluations, and provides a concrete setting for studying low-level synthesis.
  2. The evaluator checks logical memory boundaries, preserved machine state, and functional outputs while keeping submitted bytes unchanged. Failure reporting explicitly accounts for overlapping categories and unobserved downstream checks. These choices strengthen the interpretability of the reported results.
Critical weaknesses
  1. Task adaptation and semantic review are performed by the same GPT-6-Astra constructing agent in section 2.4. Mechanical checks establish consistency but do not independently establish semantic fidelity. These limitations leave uncertainty about adaptation errors and incorrect implementations accepted by the retained tests, which directly affects confidence in benchmark scores.
  2. GPT-6-Astra solves all 100 tasks, while Claude solves every task for which it provides a valid submission. Although the suite distinguishes weaker configurations, these results suggest limited resolution among the strongest evaluated models, reducing its prospective value as a benchmark.
Questions for authors
  1. Was selection of the 100 tasks fixed before model evaluation? Were any tasks, interfaces, or resource limits revised in response to evaluated model outcomes? This is particularly relevant because GPT-6-Astra both constructs the tasks and achieves a perfect score.
  2. Are results available on a harder, independently constructed subset or under smaller generation budgets that distinguish the strongest models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.