- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BedrockBench, a benchmark for evaluating whether LLMs can directly generate executable x86-64 machine code from natural-language specifications without external tools or execution feedback. It adapts 100 LiveCodeBench problems into function-level tasks with explicit interfaces and execution constraints. The evaluator executes submitted bytes unchanged and checks functional correctness, memory accesses, ABI compliance, and instruction and resource limits under three machine environments. Across seven model configurations, reported pass@1 ranges from 13% to 100%. GPT-6-Astra passes all tasks, and all 93 format-valid Claude submissions succeed. The paper also analyzes failure categories and environment sensitivity, identifying unauthorized memory accesses as a common observed violation among format-valid unsuccessful submissions.
Critical strengths
-
The evaluation target is clearly defined. The paper requires models to generate executable machine-code bytes directly, without external tools or execution feedback. This makes instruction encoding part of the evaluation and provides a concrete setting for studying low-level implementation capabilities beyond source-code generation.
-
The evaluation protocol is carefully specified. In addition to functional correctness, the evaluator checks logical memory boundaries, ABI return state, instruction restrictions, and resource use. It also runs the same inputs under three legal machine environments. These checks help identify machine-level errors that output-only evaluation may miss.
-
The results are interpreted with appropriate care. The paper distinguishes refusals, formatting errors, and execution failures, and explains that overlapping failure categories do not establish independent root causes. It also reports that environment variation adds diagnostic information without reducing task-level success in the available complete execution reports.
Critical weaknesses
-
I have some concerns about the extent of the contribution, even when considering the paper primarily as a benchmark study. The problems are adapted from LiveCodeBench, so the main contribution lies in a new evaluation setting for existing problems, the adapted benchmark, and the corresponding results. The application scope is relatively narrow, and the current study does not clearly establish what additional insights direct machine-code generation provides into the limits of LLM capabilities.
-
The empirical analysis could be strengthened to better support the purpose of the benchmark. For a LiveCodeBench problem, a model may fail because of errors specific to machine-code generation or because its algorithmic solution is incorrect. A comparison with the same models' source-code generation performance on the same tasks would help distinguish these factors and clarify what the benchmark measures. In addition, GPT-6-Astra already achieves a 100% pass rate, which raises concerns about the benchmark's ability to differentiate among stronger models.
-
Task adaptation lacks independent validation. According to Sections 2.2 and 2.4, GPT-6-Astra participates in constructing the task interfaces, checkers, and resource limits, while semantic review is also performed by the constructing agent itself. Given that the same model is also evaluated on the benchmark, I believe human expert involvement is necessary.
Questions for authors
Please see weaknesses.