- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Overall, I find the paper insufficiently clear in its presentation and limited in technical novelty. First, the motivation and the specific problem being addressed are not clearly articulated in the abstract and introduction, making the paper somewhat difficult to follow. The authors begin by emphasizing that many open-source models are not readily usable due to heterogeneous dependencies, architectures, interfaces, and the lack of ready-to-run inference code. This framing naturally suggests that the primary challenge is how to generate reliable inference programs for these models. However, the discussion then shifts to the potential errors introduced by human developers or coding agents, which raises a different problem: how to verify whether the generated programs faithfully implement the intended inference behavior. Based on my understanding of the proposed method, the paper primarily focuses on the latter problem, namely verification. I believe the authors could substantially improve the presentation by clarifying this distinction and establishing a more coherent motivation. Second, I find the proposed methodology rather straightforward and not particularly compelling in terms of technical novelty. The framework consists of two main components: (1) Behavioral Verification, which checks whether program outputs satisfy the expected requirements, and (2) Content Verification, which examines whether the implementation contains deviations from the intended behavior. However, executing a program to inspect its outputs and examining its implementation are arguably the most natural approaches to assessing program correctness. The specific technical designs also appear relatively conventional. Behavioral Verification essentially requires all probes to execute successfully and pass their corresponding checks, while Content Verification relies on examining the implementation along five predefined dimensions. I find it difficult to identify a sufficiently novel technical contribution in these designs. There are also some limitations in the experiments. Overall, while the problem has practical relevance, I believe the paper has considerable room for improvement, particularly in terms of problem formulation, technical novelty, and experimental validation.
Critical strengths
-
The paper discusses an interesting and practically relevant problem: how to verify whether agent-generated inference programs faithfully follow their intended design and behavior.
-
The experimental evaluation is relatively extensive in scale and demonstrates some promising results, highlighting the practical and engineering value of the proposed system.
Critical weaknesses
-
Unclear motivation and problem formulation. The motivation and the specific problem being addressed are not clearly articulated in the abstract and introduction, making the paper somewhat difficult to follow. The authors initially emphasize that many open-source models are not readily usable due to heterogeneous dependencies, architectures, interfaces, and the lack of ready-to-run inference code. This framing naturally suggests that the primary challenge is how to generate reliable inference programs. However, the discussion then shifts to the potential errors introduced by human developers or coding agents, which raises a different problem: how to verify whether the generated programs faithfully implement the intended inference behavior. Based on my understanding, the paper primarily focuses on the latter problem. I believe the authors should better distinguish these two problems and establish a clearer and more coherent motivation.
-
Limited technical novelty. I find the proposed methodology rather straightforward and not particularly compelling in terms of technical novelty. The framework consists of two main components: (1) Behavioral Verification, which checks whether program outputs satisfy the expected requirements, and (2) Content Verification, which examines whether the implementation contains deviations from the intended behavior. However, executing a program to inspect its outputs and examining its implementation are arguably the most natural approaches to assessing program correctness. The specific technical designs also appear relatively conventional. Behavioral Verification essentially requires all probes to execute successfully and pass their corresponding checks, while Content Verification relies on examining the implementation along five predefined dimensions. I find it difficult to identify a sufficiently novel technical contribution in these designs.
-
Insufficient experimental evidence for verification effectiveness. Although the experiments are relatively extensive in scale and demonstrate the practical utility of the system, I do not think they sufficiently establish the effectiveness and reliability of the proposed verification mechanism. In particular, the improvements in model coverage and downstream agent performance do not directly demonstrate the contribution of the verification components, as these improvements may largely result from providing ready-to-run inference programs. Furthermore, the verification evaluation primarily compares the proportion of programs judged clean by Bolo and a binary LLM judge, but a higher clean rate does not necessarily indicate more accurate verification. More importantly, only 20 programs are manually inspected, which seems insufficient to support general conclusions about verification reliability across 2,019 programs. As the authors themselves acknowledge, the proposed verification provides evidence rather than formal guarantees of correctness. While formal verification may not be the intended goal, I believe more comprehensive empirical evidence is needed to establish how reliably the proposed approach identifies genuine implementation deviations and avoids missing errors.
Questions for authors
-
I recognize that the paper discusses a meaningful and practically relevant problem. However, I found the motivation and the specific research problem somewhat unclear throughout the presentation. Could the authors provide a more concise and explicit statement of the core problem and motivation?
-
I am not fully convinced by the technical novelty of the proposed approach, as the overall design appears relatively straightforward. Could the authors further clarify the key technical contributions and explain what distinguishes Bolo from existing approaches?
-
I find the experimental evaluation somewhat insufficient to fully support the paper's claims, particularly regarding the effectiveness of the proposed verification mechanism. Could the authors provide additional experiments or more comprehensive evidence to strengthen the empirical validation?
3 comments
Response to Reviewer FnUr (2/2)
2. Technical Novelty
Q1. I am not fully convinced by the technical novelty of the proposed approach, as the overall design appears relatively straightforward. Could the authors further clarify the key technical contributions and explain what distinguishes Bolo from existing approaches?
W1. Limited technical novelty. I find the proposed methodology rather straightforward and not particularly compelling in terms of technical novelty. The framework consists of two main components: (1) Behavioral Verification, which checks whether program outputs satisfy the expected requirements, and (2) Content Verification, which examines whether the implementation contains deviations from the intended behavior. However, executing a program to inspect its outputs and examining its implementation are arguably the most natural approaches to assessing program correctness. The specific technical designs also appear relatively conventional. Behavioral Verification essentially requires all probes to execute successfully and pass their corresponding checks, while Content Verification relies on examining the implementation along five predefined dimensions. I find it difficult to identify a sufficiently novel technical contribution in these designs.
Our core technical contribution is making the content verification work when the intended behavior is only described informally in model cards. Compared with existing approaches:
- Test-based methods judge a program by comparing its outputs with expected outputs. CodeT [1] generates test cases with an LLM and selects the solution that agrees most with the tests and other solutions, and EvalPlus [2] compares outputs with a ground-truth implementation. Inference programs have no ground-truth implementation and often have many valid outputs. Section 3.3 also shows that a program can return a valid output while silently falling back to CPU, no matter which platform is requested.
- LLM-assisted auditors detect known classes of bugs. IRIS [3] uses LLMs to infer taint specifications for detecting security vulnerabilities, and RepoAudit [4] detects data-flow bugs such as null pointer dereference and memory leak. In our case, for example, loading model B’s checkpoint instead of model A’s is not a known bug class. It is wrong only with respect to model A’s model card.
- LLM-as-a-judge methods such as MCTS-Judge [5] return a single correct/incorrect verdict. In our experiments, a binary LLM judge in this style (Section 5.5) flags 657 programs that Bolo finds clean, and none of its stated concerns holds in the 10 programs we inspected (Table 1 in the paper). A binary verdict also has to decide in advance which deviations make a program “incorrect”, but this depends on the user. For example, CPU fallback matters only if latency matters.
Bolo’s content verification differs from these approaches in two ways. (1) It checks the program against model-specific intent taken from the model card, along five dimensions. (2) It returns localized evidence (code lines and a description) instead of a verdict, so users can decide whether a deviation matters to them.
Reference:
[1] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT: Code generation with generated tests. In ICLR, 2023.
[2] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23).
[3] Ziyang Li, Saikat Dutta, and Mayur Naik. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In ICLR, 2025.
[4] Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. REPOAUDIT: an autonomous LLM-agent for repository-level code auditing. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267.
[5] Yutong Wang, Pengliang Ji, Chaoqun Yang, Kaixin Li, Ming Hu, Jiaoyang Li, and Guillaume Sartoretti. MCTS-Judge: Test-time scaling in LLM-as-a-judge for code correctness evaluation. arXiv:2502.12468, 2025.
Thank you for the detailed response. I still think this paper falls short in terms of technical novelty and does not meet the bar of ICLR. I will keep my original rating. Thanks.
Response to Reviewer FnUr (1/2)
We thank the reviewer for the detailed and constructive comments.
1. Motivation and Problem Formulation
The core problem we study is how to support inference programs for public models: for any model on a hub, provide inference program that users and agents can run directly, together with evidence of whether the program does what the model intends. Generation and verification are not two separate problems, but two steps of this one problem.
The two steps differ in difficulty, which is why we spend more effort on verification. Generation is relatively easy, because it has a clear feedback signal. When a program fails, the Python interpreter raises an error as the direct feedback / hints for agents. The agent can therefore repair the program step by step until it runs (Section 4.1).
Verification is harder, because a deviation raises no error. For example, in Section 3.3, the agent writes a program that runs and returns a valid embedding, but no matter which platform is requested, it silently falls back to CPU. So an agent can write a program, but the program is not necessarily correct.