acceptodds
Under review as a conference paper at ICLR 2027

Bolo: A Verified Model Hub with Ready-to-Run Inference Programs for Agents and Humans

Abstract

Both AI agents and human engineers can greatly benefit from the ability to use any public machine learning model. Unfortunately, most public models, including those on Hugging Face, are not usable out of the box, and manually writing inference programs does not scale because failures follow diverse, model-specific patterns. Even worse, existing coding agents often produce faulty or misleading inference programs that run successfully yet silently behave differently than intended. Addressing this requires more sophisticated, model-specific verification strategy that can identify the ways those generated programs deviate from user intentions. To this end, we introduce \bolo, a large-scale model hub for constructing and verifying model inference programs. \bolo verifies each generated program at two levels: behavioral checks that test whether outputs are valid for the model's task, and content-level verification that checks the implementation. Across five model-dependent tasks, agents using \bolo achieve higher-quality outputs than agents with direct Hugging Face access while reducing LLM cost by up to 26%. On 8,570 most widely downloaded Hugging Face models, \bolo covers 65.2%, compared with 41.6% for Hugging Face Transformers \verb|pipeline| API, including 1,195 models outside its support. Finally, among 2,019 programs passing behavioral checks, \bolo's content-level verification judges 88.6% clean, compared with 57.2% for a binary LLM judge. Qualitative case studies further demonstrate that \bolo avoids unsupported rejections while identifying genuine implementation issues that output checks alone cannot detect.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Agents, Tool Use, Model Hubs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FnUrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Overall, I find the paper insufficiently clear in its presentation and limited in technical novelty. First, the motivation and the specific problem being addressed are not clearly articulated in the abstract and introduction, making the paper somewhat difficult to follow. The authors begin by emphasizing that many open-source models are not readily usable due to heterogeneous dependencies, architectures, interfaces, and the lack of ready-to-run inference code. This framing naturally suggests that the primary challenge is how to generate reliable inference programs for these models. However, the discussion then shifts to the potential errors introduced by human developers or coding agents, which raises a different problem: how to verify whether the generated programs faithfully implement the intended inference behavior. Based on my understanding of the proposed method, the paper primarily focuses on the latter problem, namely verification. I believe the authors could substantially improve the presentation by clarifying this distinction and establishing a more coherent motivation. Second, I find the proposed methodology rather straightforward and not particularly compelling in terms of technical novelty. The framework consists of two main components: (1) Behavioral Verification, which checks whether program outputs satisfy the expected requirements, and (2) Content Verification, which examines whether the implementation contains deviations from the intended behavior. However, executing a program to inspect its outputs and examining its implementation are arguably the most natural approaches to assessing program correctness. The specific technical designs also appear relatively conventional. Behavioral Verification essentially requires all probes to execute successfully and pass their corresponding checks, while Content Verification relies on examining the implementation along five predefined dimensions. I find it difficult to identify a sufficiently novel technical contribution in these designs. There are also some limitations in the experiments. Overall, while the problem has practical relevance, I believe the paper has considerable room for improvement, particularly in terms of problem formulation, technical novelty, and experimental validation.

Critical strengths
  1. The paper discusses an interesting and practically relevant problem: how to verify whether agent-generated inference programs faithfully follow their intended design and behavior.

  2. The experimental evaluation is relatively extensive in scale and demonstrates some promising results, highlighting the practical and engineering value of the proposed system.

Critical weaknesses
  1. Unclear motivation and problem formulation. The motivation and the specific problem being addressed are not clearly articulated in the abstract and introduction, making the paper somewhat difficult to follow. The authors initially emphasize that many open-source models are not readily usable due to heterogeneous dependencies, architectures, interfaces, and the lack of ready-to-run inference code. This framing naturally suggests that the primary challenge is how to generate reliable inference programs. However, the discussion then shifts to the potential errors introduced by human developers or coding agents, which raises a different problem: how to verify whether the generated programs faithfully implement the intended inference behavior. Based on my understanding, the paper primarily focuses on the latter problem. I believe the authors should better distinguish these two problems and establish a clearer and more coherent motivation.

  2. Limited technical novelty. I find the proposed methodology rather straightforward and not particularly compelling in terms of technical novelty. The framework consists of two main components: (1) Behavioral Verification, which checks whether program outputs satisfy the expected requirements, and (2) Content Verification, which examines whether the implementation contains deviations from the intended behavior. However, executing a program to inspect its outputs and examining its implementation are arguably the most natural approaches to assessing program correctness. The specific technical designs also appear relatively conventional. Behavioral Verification essentially requires all probes to execute successfully and pass their corresponding checks, while Content Verification relies on examining the implementation along five predefined dimensions. I find it difficult to identify a sufficiently novel technical contribution in these designs.

  3. Insufficient experimental evidence for verification effectiveness. Although the experiments are relatively extensive in scale and demonstrate the practical utility of the system, I do not think they sufficiently establish the effectiveness and reliability of the proposed verification mechanism. In particular, the improvements in model coverage and downstream agent performance do not directly demonstrate the contribution of the verification components, as these improvements may largely result from providing ready-to-run inference programs. Furthermore, the verification evaluation primarily compares the proportion of programs judged clean by Bolo and a binary LLM judge, but a higher clean rate does not necessarily indicate more accurate verification. More importantly, only 20 programs are manually inspected, which seems insufficient to support general conclusions about verification reliability across 2,019 programs. As the authors themselves acknowledge, the proposed verification provides evidence rather than formal guarantees of correctness. While formal verification may not be the intended goal, I believe more comprehensive empirical evidence is needed to establish how reliably the proposed approach identifies genuine implementation deviations and avoids missing errors.

Questions for authors
  1. I recognize that the paper discusses a meaningful and practically relevant problem. However, I found the motivation and the specific research problem somewhat unclear throughout the presentation. Could the authors provide a more concise and explicit statement of the core problem and motivation?

  2. I am not fully convinced by the technical novelty of the proposed approach, as the overall design appears relatively straightforward. Could the authors further clarify the key technical contributions and explain what distinguishes Bolo from existing approaches?

  3. I find the experimental evaluation somewhat insufficient to fully support the paper's claims, particularly regarding the effectiveness of the proposed verification mechanism. Could the authors provide additional experiments or more comprehensive evidence to strengthen the empirical validation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FnUr (1/2)

We thank the reviewer for the detailed and constructive comments.

1. Motivation and Problem Formulation

Q1. I recognize that the paper discusses a meaningful and practically relevant problem. However, I found the motivation and the specific research problem somewhat unclear throughout the presentation. Could the authors provide a more concise and explicit statement of the core problem and motivation?

The core problem we study is how to support inference programs for public models: for any model on a hub, provide inference program that users and agents can run directly, together with evidence of whether the program does what the model intends. Generation and verification are not two separate problems, but two steps of this one problem.

W1. Unclear motivation and problem formulation. The motivation and the specific problem being addressed are not clearly articulated in the abstract and introduction, making the paper somewhat difficult to follow. The authors initially emphasize that many open-source models are not readily usable due to heterogeneous dependencies, architectures, interfaces, and the lack of ready-to-run inference code. This framing naturally suggests that the primary challenge is how to generate reliable inference programs. However, the discussion then shifts to the potential errors introduced by human developers or coding agents, which raises a different problem: how to verify whether the generated programs faithfully implement the intended inference behavior. Based on my understanding, the paper primarily focuses on the latter problem. I believe the authors should better distinguish these two problems and establish a clearer and more coherent motivation.

The two steps differ in difficulty, which is why we spend more effort on verification. Generation is relatively easy, because it has a clear feedback signal. When a program fails, the Python interpreter raises an error as the direct feedback / hints for agents. The agent can therefore repair the program step by step until it runs (Section 4.1).

Verification is harder, because a deviation raises no error. For example, in Section 3.3, the agent writes a program that runs and returns a valid embedding, but no matter which platform is requested, it silently falls back to CPU. So an agent can write a program, but the program is not necessarily correct.

AuthorsOpenReview ↗

Response to Reviewer FnUr (2/2)

2. Technical Novelty

Q1. I am not fully convinced by the technical novelty of the proposed approach, as the overall design appears relatively straightforward. Could the authors further clarify the key technical contributions and explain what distinguishes Bolo from existing approaches?

W1. Limited technical novelty. I find the proposed methodology rather straightforward and not particularly compelling in terms of technical novelty. The framework consists of two main components: (1) Behavioral Verification, which checks whether program outputs satisfy the expected requirements, and (2) Content Verification, which examines whether the implementation contains deviations from the intended behavior. However, executing a program to inspect its outputs and examining its implementation are arguably the most natural approaches to assessing program correctness. The specific technical designs also appear relatively conventional. Behavioral Verification essentially requires all probes to execute successfully and pass their corresponding checks, while Content Verification relies on examining the implementation along five predefined dimensions. I find it difficult to identify a sufficiently novel technical contribution in these designs.

Our core technical contribution is making the content verification work when the intended behavior is only described informally in model cards. Compared with existing approaches:

  • Test-based methods judge a program by comparing its outputs with expected outputs. CodeT [1] generates test cases with an LLM and selects the solution that agrees most with the tests and other solutions, and EvalPlus [2] compares outputs with a ground-truth implementation. Inference programs have no ground-truth implementation and often have many valid outputs. Section 3.3 also shows that a program can return a valid output while silently falling back to CPU, no matter which platform is requested.
  • LLM-assisted auditors detect known classes of bugs. IRIS [3] uses LLMs to infer taint specifications for detecting security vulnerabilities, and RepoAudit [4] detects data-flow bugs such as null pointer dereference and memory leak. In our case, for example, loading model B’s checkpoint instead of model A’s is not a known bug class. It is wrong only with respect to model A’s model card.
  • LLM-as-a-judge methods such as MCTS-Judge [5] return a single correct/incorrect verdict. In our experiments, a binary LLM judge in this style (Section 5.5) flags 657 programs that Bolo finds clean, and none of its stated concerns holds in the 10 programs we inspected (Table 1 in the paper). A binary verdict also has to decide in advance which deviations make a program “incorrect”, but this depends on the user. For example, CPU fallback matters only if latency matters.

Bolo’s content verification differs from these approaches in two ways. (1) It checks the program against model-specific intent taken from the model card, along five dimensions. (2) It returns localized evidence (code lines and a description) instead of a verdict, so users can decide whether a deviation matters to them.

Reference:

[1] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT: Code generation with generated tests. In ICLR, 2023.

[2] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23).

[3] Ziyang Li, Saikat Dutta, and Mayur Naik. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In ICLR, 2025.

[4] Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. REPOAUDIT: an autonomous LLM-agent for repository-level code auditing. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267.

[5] Yutong Wang, Pengliang Ji, Chaoqun Yang, Kaixin Li, Ming Hu, Jiaoyang Li, and Guillaume Sartoretti. MCTS-Judge: Test-time scaling in LLM-as-a-judge for code correctness evaluation. arXiv:2502.12468, 2025.

Thank you for the detailed response. I still think this paper falls short in terms of technical novelty and does not meet the bar of ICLR. I will keep my original rating. Thanks.

Reviewer_1p8xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Bolo, a model hub that automatically constructs reusable inference programs for Hugging Face models using coding agents. To detect errors beyond runtime failures, Bolo combines behavioral checks with dimension-conditioned LLM-based content verification, which identifies deviations from intended inference workflows. Experiments on 8,570 models demonstrate improved executable coverage over the Hugging Face Pipeline API, while downstream evaluations show improvements in task performance and reduced LLM invocation costs in specific scenarios.

Critical strengths
  • Practical and relevant research question: The paper identifies an important limitation of existing model hubs and coding agents: publicly available models are not always directly executable, and successfully running inference code does not necessarily imply faithful implementation. This research question is worth investigating and solving.

  • Large-scale evaluation with meaningful improvements. Bolo increases model coverage from 41.6% to 65.2%, including 1,195 models unsupported by the standard pipeline API. The five downstream tasks also demonstrate improvements in agent performance and LLM costs.

  • Interpretable verification framework. The proposed dimension-conditioned auditing provides localized deviation evidence rather than binary judgments. Concrete examples illustrate errors missed by output-based checks, such as incorrect model loading and CPU fallback.

Critical weaknesses
  • Limited novelty: The core approach largely combines existing coding agents, runtime checks, and LLM-based auditing. The main verification contribution is a five-dimensional prompting framework that asks an LLM to identify deviations from model-specific documentation. The paper does not introduce a fundamentally new program analysis or verification technique, making its methodological contribution relatively incremental.

  • Insufficient baselines and ablation studies to establish the contribution of the verification method. The content verification is primarily compared against a holistic binary LLM judge. It is unclear whether the observed improvements arise from the proposed dimension-conditioned framework or simply from providing more structured prompts and requesting localized evidence. Comparisons against stronger prompting baselines, multi-step LLM auditing, or agent-based code inspection would help isolate the contribution. More ablation would also be needed to show the necessity of each dimension.

  • Incomplete cost-effectiveness evaluation and inconsistent downstream gains: While Bolo reduces downstream LLM costs, the reported comparison excludes the upfront cost of generating and verifying inference programs. No amortized cost analysis or break-even analysis is provided. Moreover, the downstream performance improvements are not consistent across all five tasks, with some Bolo runs failing to produce outputs. Thus, the overall cost-effectiveness of constructing and maintaining the model hub remains unclear.

Questions for authors
  • What is the specific technical contribution of the dimension-conditioned verification framework beyond structured LLM prompting? Could the authors provide ablations comparing the proposed method with a single LLM auditor prompted to inspect all five dimensions, as well as variants without evidence canonicalization and pattern validation? This would help clarify which components are responsible for the reported improvements.

  • Can the authors provide a more systematic evaluation of verification accuracy? The current evaluation mainly reports clean rates and manually examines 20 disagreement cases. Could the authors construct an independently annotated test set containing both clean and faulty inference programs, and report false-positive and false-negative rates?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1p8x

We thank the reviewer for the detailed and constructive comments.

1. Technical Contribution

Q1. What is the specific technical contribution of the dimension-conditioned verification framework beyond structured LLM prompting? Could the authors provide ablations comparing the proposed method with a single LLM auditor prompted to inspect all five dimensions, as well as variants without evidence canonicalization and pattern validation? This would help clarify which components are responsible for the reported improvements.

W1. Limited novelty: The core approach largely combines existing coding agents, runtime checks, and LLM-based auditing. The main verification contribution is a five-dimensional prompting framework that asks an LLM to identify deviations from model-specific documentation. The paper does not introduce a fundamentally new program analysis or verification technique, making its methodological contribution relatively incremental.

W2. Insufficient baselines and ablation studies to establish the contribution of the verification method. The content verification is primarily compared against a holistic binary LLM judge. It is unclear whether the observed improvements arise from the proposed dimension-conditioned framework or simply from providing more structured prompts and requesting localized evidence. Comparisons against stronger prompting baselines, multi-step LLM auditing, or agent-based code inspection would help isolate the contribution. More ablation would also be needed to show the necessity of each dimension.

Our core technical contribution is making the content verification work when the intended behavior is only described informally in model cards. Furthermore, we propose the verification should not produce a simple result (e.g., a binary verdict). Instead, it should be a evidence-based conclusion so users can decide based on the context of running model inference (e.g., which platform is used? CPU / GPU).

2. Cost-Effectiveness Evaluation

W3. Incomplete cost-effectiveness evaluation and inconsistent downstream gains: While Bolo reduces downstream LLM costs, the reported comparison excludes the upfront cost of generating and verifying inference programs. No amortized cost analysis or break-even analysis is provided. Moreover, the downstream performance improvements are not consistent across all five tasks, with some Bolo runs failing to produce outputs. Thus, the overall cost-effectiveness of constructing and maintaining the model hub remains unclear.

We report the cost for running generation and verification of 3,283 Type II models:

Stage (Type II)NMean LLM callsMean LLM cost (USD)Total LLM cost (USD)
Generation3,2835.61$0.01710$56.13
Verification2,3722.29$0.03581$84.95

The generation is cheaper since we use cheaper model (GPT-5.4-mini) while the verification utilizes Claude-Sonnet-5. But the overall cost remains ~140 USD for 3,283 attempts, among them generation succeeds in 2,372 cases.

Reviewer_1p8xOpenReview ↗

Reply to author's rebuttal

Thanks for your rebuttal! I also read your other rebuttals, especially your reply to FnUr about novelty. However, several things remain unclear to me:

  • For the novelty of your problem and methods, yes I do recognize that the model weight-model card misalignment is happening on public models. But do we need to run large scale verification to verify them? That links to the same cost problem below, which is also basically the reason I requested a cost analysis. Only by showing that the total GPU + API consumption is acceptable for single model, and this wouldn't scale up to a unacceptable level on large models, we can believe that the methodology here is a pathway towards solving model weight-model card misalignment.

  • Also, for experiment, I'm curious why only Type II models are reported, since in this way we cannot compare that with other methods. Moreover, did you record wall time or something like time-efficiency? How does GPU consumption cost for different methods? The point here, I think, is comparison, not reporting single number, i.e., why model hub should work this way instead of others, how much it would save under its targeted scope.

Thanks for the reply.

We agree that cost is important for large-scale verification. In our experiments, generating and verifying 3,283 Type II models costs about 453.46 GPU-hours in total. And we run generation in parallel on a single GPU, which costs ~70 hours as wall time. The verification does not require GPU. This cost is paid offline only once per model, and the program is then reused by all users and agents. We believe this cost is acceptable for a model hub. For larger models, the API cost mainly depends on the length of the program (output of LLM) and the model card (input of LLM), not on the model size, because verification reads the code and documentation rather than the weights. The GPU cost grows with model size, mainly from loading and hosting the weights. As stated in Section 6, this paper focuses on models smaller than 16 GB, and we leave larger models to future work.

Type I models are already supported by HuggingFace pipeline API, and Bolo reuses pipeline API and only repairs it when it fails. Type II models are not supported by pipeline, so Bolo must generate the program from scratch. Type II therefore shows the main cost of Bolo. In addition, to our knowledge, there is no existing method that generates and verifies inference programs at scale, so we cannot compare the generation step with a baseline.

Reviewer_Fu3TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on scaling inference program generation and proposes BOLO, a framework to construct and maintain a verified model hub featuring ready-to-run, model-specific inference programs. These inference programs are accessible via unified invocation interfaces. BOLO decouples the program generation from verification and undergo two-level verifications: behavioral verification & content verification to provide deviration check. Experimental results show BOLO is competitive in broader interface support capability and better downstream task completion quality.

Critical strengths
  • The work evaluates and verifies thousands of real-world Hugging Face models, extending ready-to-run execution to more models that lack standard pipeline integration.

  • The work proposes a two-level verification architecture to prevent downstream implementation from malicious or unintended behavior.

  • It structures content verification across five distinct dimensions and provides line-level deviation evidence rather than a coarse binary verdict.

Critical weaknesses
  • Motivation and practical necessity of scaling are under-supported. Although the paper demonstrates that many Hugging Face models are not directly executable through standard pipeline APIs, it does not sufficiently establish the practical demand for generating and maintaining verified inference programs at this scale.

  • A lot of design details are missing; for example, the detailed function implementation design in each equation.

  • The experiment requires more results and details (e.g., token consumption of the BOLO pipeline and an ablation study)

Questions for authors
  1. Could the authors provide stronger evidence demonstrating the practical demand for maintaining thousands of verified model-specific inference programs? In particular, how frequently are the additional models supported by BOLO actually needed in downstream applications, and how does broader model coverage translate into practical benefits?

  2. Could the authors elaborate on the concrete implementation of Equations (1)–(3)? Specifically, how are behavioral test cases generated for heterogeneous models? How is the output check designed in Equation (1)? What mechanisms ensure that the detected deviations are supported by reliable evidence?

  3. What are the average token consumption, number of LLM calls, execution time, and monetary costs of generating and verifying an inference program? How do these costs vary across model categories and successful versus unsuccessful generation attempts? A detailed overhead analysis would help assess BOLO's scalability and feasibility for practical deployment.

  4. How does BOLO ensure that the automatically generated inference programs are concise, well-structured, and human-readable? Compared to manual writing. (I am just thinking about some maintainability and code quality issues)

3 comments
AuthorsOpenReview ↗

Response to Reviewer Fu3T (1/3)

We thank the reviewer for the detailed and constructive comments.

1. Motivation and Practical Demand

Q1. Could the authors provide stronger evidence demonstrating the practical demand for maintaining thousands of verified model-specific inference programs? In particular, how frequently are the additional models supported by BOLO actually needed in downstream applications, and how does broader model coverage translate into practical benefits?

W1. Motivation and practical necessity of scaling are under-supported. Although the paper demonstrates that many Hugging Face models are not directly executable through standard pipeline APIs, it does not sufficiently establish the practical demand for generating and maintaining verified inference programs at this scale.

Table 1. Models evaluated by Bolo.

#ModelsShare of all Hub downloads
Models evaluated by Bolo11,53874.1%
Out of coverage (excluded)3,17512.8%
In coverage8,36361.2%

Table 2. Breakdown of the 8,363 in-coverage models.

#ModelsShare of all Hub downloads
Supported by Hugging Face pipeline API3,56735.1%
Additional models supported by Bolo3,60322.6%
Neither supported1,1933.6%

As shown in Table 1, Bolo considers 8,363 models which counts 61.2% of all Hugging Face Hub downloads (i.e., widely-used models). Among those 8,363 models, Bolo additionally supports 3,603 models which counts 22.6% of all Hugging Face Hub downloads.

To demonstrate that the broader model coverage translate into practical benefits. We additionally design 80 data processing tasks requiring 1 to 4 models for each. We run the Claude agent harness with Hugging Face and Bolo access, the backend LLM is Claude-Haiku-5.5, and each task is repeated 3 times for each agent. And each run is capped by 1 hour. Other setup is the same as Section 5.3.

Table 3. A breakdown of 80 tasks

#Required modelsTask number#TasksPortion
1T001–T0262632.50%
2T027–T0613543.75%
3T062–T0761518.75%
4T077–T08045.00%

We first report the efficiency gain of Bolo-powered agents. As shown in Table 4, Bolo-powered agents finish the task in less time (31.13 vs. 43.28 minutes), because Bolo provides packaged inference programs and dependencies for the agents to use. Bolo-powered agents have a higher LLM cost and more steps. This is because HF-powered agents often get stuck on time-consuming steps (e.g., running or testing models on CPU by mistake), which use up the 1-hour budget, so they stop after fewer steps.

Second, we report task success rate in Table 5: we score the output of each run and compare the score with a threshold. For example, the F1 score should be higher than 0.8, and the WER (for audio tasks) should be lower than 0.2. Bolo-powered agents have a higher success rate (38.3% vs. 29.6%).

Table 4. Efficiency of Bolo-powered and HF-powered agents.

MethodAvg. time (min)Avg. LLM cost ($)Avg. steps
Bolo31.130.0269629.32
HF43.280.0225828.29

Table 5. Success of Bolo-powered and HF-powered agents. A run succeeds if its output score passes the task threshold (e.g., F1 > 0.8, WER < 0.2).

Method#SuccessSuccess rate
Bolo92/24038.3%
HF71/24029.6%
AuthorsOpenReview ↗

Response to Reviewer Fu3T (2/3)

2. Implementation of Verification

Q2. Could the authors elaborate on the concrete implementation of Equations (1)–(3)? Specifically, how are behavioral test cases generated for heterogeneous models? How is the output check designed in Equation (1)? What mechanisms ensure that the detected deviations are supported by reliable evidence?

W2. A lot of design details are missing; for example, the detailed function implementation design in each equation.

  1. Equation (1) is the definition of behavior correctness of an inference program. That is, for each test, the program does not crash (i.e., throw runtime error) and produce the desired output.
  2. Equation (2) describes content verification. AθA_{\theta} is the LLM auditor, which is implemented as a prompting framework. We define five dimensions D\mathcal{D}: Identity, Completeness, Data, Platform, and Safety. For each inference program pp and each dimension dd, we feed the model card ρr\rho_r and the definition of dd to the LLM auditor, which then outputs any evidence that the program violates that definition. Each piece of evidence contains a few lines of code ℓj\ell_j and a description δj\delta_j written by the LLM. For example, the definition of the Identity dimension is “the program should load the desired model checkpoint and preprocessor/tokenizer.” If the desired model is A but the program loads B’s checkpoint, the auditor outputs the checkpoint-loading code and a short description.
  3. Equation (3) states that behavioral correctness does not imply content correctness, and vice versa. For example, a program may produce correct results on tests but loads the checkpoint from other model with same functionality. On the other hand, a program may have no deviations in any of the five dimensions yet still fail the tests, because the model itself is not capable of producing the expected output.

For the output check of Equation 1, we use the LLM-as-judge to by feeding the input & output to an LLM and judge whether the output is desired given the input. We use few-shot prompting framework to ensure the detected deviations are supported reliable evidence. We collect several typical examples of deviations for each dimension as examples. In addition, we have another LLM validator to examine the output of LLM auditor AθA_{\theta}.

AuthorsOpenReview ↗

Response to Reviewer Fu3T (3/3)

3. More Experiments on Bolo to Evaluate Scalability and Feasibility for Practical Deployment

Q3. What are the average token consumption, number of LLM calls, execution time, and monetary costs of generating and verifying an inference program? How do these costs vary across model categories and successful versus unsuccessful generation attempts? A detailed overhead analysis would help assess BOLO's scalability and feasibility for practical deployment.

W3. The experiment requires more results and details (e.g., token consumption of the BOLO pipeline and an ablation study)

We provide additional results on 3,283 Type II models, including both successful and failed generations (we use GPT-5.4-mini as backend LLM). We classify each model into a category according to the metadata in its model card. On average, Bolo takes about 8 minutes and costs less than 0.02 USD to generate an inference program (Table 6). Text / NLP models are the cheapest to generate (0.01447 USD), because they are the largest category in our set and their ecosystem is the best supported. Table 8 reports the content verification cost (we use Claude-Sonnet-5 as backend model), the cost is higher (but still less than 0.045 USD) since we use a more expensive LLM to verify the programs.

Table 6. Total cost for 3,283 generation attempts.

Model categoryMeasured attemptsMean LLM callsMean wall time (min)Mean LLM cost (USD)
Text / NLP16985.228.280.01447
Vision (image, video, 3D)4005.968.390.02061
Audio / speech3575.558.150.01820
Vision-language / multimodal2336.398.640.02320
Tabular / time series686.257.970.02319
Other declared tasks / domains166.197.950.01844
Unclassified metadata5116.198.220.01864
All categories32835.618.290.01710

Table 7. Breakdown of costs of successful and failed generations

Model categorySuccessful NSuccessful mean cost (USD)Unsuccessful measured NUnsuccessful mean cost (USD)
Text / NLP13770.013713210.01771
Vision (image, video, 3D)2820.019611180.02298
Audio / speech2020.019271550.01681
Vision-language / multimodal1220.017771110.02917
Tabular / time series430.02467250.02065
Other declared tasks / domains120.0157340.02658
Unclassified metadata3410.015831700.02429

Table 8. Content verification cost

Model category#Verified programsMean LLM callsMean LLM cost (USD)
Text / NLP13742.280.03656
Vision (image, video, 3D)2792.340.03547
Audio / speech2022.170.03404
Vision-language / multimodal1222.180.03535
Tabular / time series432.470.04360
Other declared tasks / domains122.500.03418
Unclassified metadata3402.370.03337

4. Code Quality & Maintainability

Q4. How does BOLO ensure that the automatically generated inference programs are concise, well-structured, and human-readable? Compared to manual writing. (I am just thinking about some maintainability and code quality issues)

Each template is generated in Jinja2 format, and we only render it during validation; when distributed to users, it remains a Jinja2 template. Important parameters (e.g., the model name, input, or the temperature for a language model) are exposed as Jinja2 variables, so users can immediately see the knobs in the program. In addition, we require the program to return a Python dict whose keys describe each output, so users can know the input and output of a program at a glance. In addition, the program is usually short (less than 50 lines), so the maintainability is not a problem.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.