acceptodds
Under review as a conference paper at ICLR 2027

AI Research Preference Models

Abstract

AI research agents (AIRAs) can drive machine learning experiments from proposal to implementation and evaluation. Yet their progress on frontier tasks is often throttled by the cost of evaluation, which can consume days of GPU time. When an agent can propose far more candidates than it can afford to run, progress depends on its research preference: how it allocates a fixed execution budget across many candidates. We introduce AI Research Preference Models (RPMs) that predict which candidate solution is most promising, without needing to run them all. We build two variants of RPMs from pretrained language models: an inference-only model that reasons over candidate plans, code, and previously executed solutions, and an agentic model that also runs small-scale pilot experiments. Integrated into the AIRA-dojo research agent and evaluated on the machine learning research benchmark AIRS-Bench, the inference-only variant consistently provides gains over the unguided baseline with the agentic variant delivering the best overall performance. Across four LLM backbones (GPT-5, Gemma 4-31B, Qwen3.6-27B and Muse Glimmer), both variants improve the average normalized score over the baseline unguided agent. On average, they match the unguided agent's 24-hour performance in roughly 14 hours, using less than 60% of its execution budget, and they set new state-of-the-art results on three AIRS-Bench tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Automated Research, LLMs, Scientific Discovery, Evolutionary Search

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3xVYOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AI Research Preference Models (RPMs) to select promising candidate solutions before committing to expensive training and evaluation. The authors propose two variants: an inference-only RPM that judges candidates using their plans, code, and search history, and an agentic RPM that additionally conducts small-scale pilot experiments. Both are integrated into the child-creation stage of AIRA-dojo and evaluated on AIRS-BENCH using four LLM backbones. The paper reports improved final performance and reduced execution time to reach the unguided baseline's performance.

Critical strengths

The motivation is clear. AI research agents can generate candidate solutions much faster than they can fully execute and evaluate them. Selecting promising candidates under a limited execution budget therefore addresses a meaningful bottleneck in automated research.

Critical weaknesses
  1. The novelty of preference-based candidate selection appears limited. The general idea of using a judge or preference model to select promising candidates before expensive execution already has precedents. Section 6 explicitly acknowledges Zheng et al. (2026), who predict pairwise preferences between unexecuted ML solutions, and Goldie et al. (2026), who propose using a judge or reward model to select promising leaves in research-agent tree search. The distinctive contributions here appear to be conditioning on the live search tree and incorporating pilot experiments. However, the paper does not sufficiently establish the incremental benefit over these closely related approaches through direct comparisons.

  2. The final performance gains are relatively modest, particularly for the additional agentic component. In Figure 2, Agentic RPM's additional gains over Inference-only RPM are only 0.013, 0.005, 0.018, and 0.026, respectively. Inference-only RPM also slightly underperforms No RPM on Muse Glimmer, scoring 0.694 versus 0.696. These results demonstrate improvements, but their magnitude is limited. Although Figure 2 includes 95% confidence intervals, the paper does not report paired statistical tests establishing the significance of the differences between methods.

  3. It remains unclear how expensive agentic verification is relative to fully executing a candidate. Section 3.2 gives the Agentic RPM access to a single H200 GPU and an environment equivalent to that of the main research agent. Appendix B.1 specifies a nominal budget of 300 seconds per agentic workflow, but also states that the budget is checked only after the current turn's tool calls complete, so actual execution can exceed it. Section 5.2.2 states that pilot experiments share the research agent's time budget, meaning that their execution cost is not simply ignored. Nevertheless, the paper does not provide a clear breakdown of the total pilot cost per candidate-selection step relative to the cost of fully executing the selected candidate. Section 7 also assumes negligible LLM inference cost and provides a latency adjustment only for the inference-only Qwen configuration. Consequently, it is difficult to determine when agentic verification remains substantially cheaper than direct execution after accounting for all comparisons, pilot experiments, and model calls.

Questions for authors
  1. Section 3.3 describes candidate selection as pairwise comparisons in a knockout tournament, while Section 5.1 uses 15 candidates per operator step. Does Agentic RPM follow this same tournament structure, or are all 15 candidates passed to a single agentic workflow? Appendix B contains both two-candidate and multiple-candidate wording, so the exact end-to-end procedure would benefit from clarification.

  2. Is the 300-second budget allocated to each pairwise comparison or to the entire selection among 15 candidates? A standard knockout tournament over 15 candidates requires 14 comparisons. If each comparison receives a separate 300-second budget, this would amount to 70 minutes of nominal aggregate comparison time per selection step before accounting for early termination, overruns, or parallelism. Please clarify the actual allocation and scheduling.

9 comments

TLDR:

  • Novelty with respect to Goldie et al (2026): The authors leave RPM-like approaches as a future research direction, without implementing or evaluating it
  • Novelty with respect to Zheng et al 2026: Our method leverages historical search-tree context and agentic pilot experimentation, and we provide extension online and offline evaluations to show the superiority of this in a novel setting.
  • Statistical significance: We conduct paired task-level statistical tests and show that Agentic RPM significantly outperforms the No-RPM baseline.
  • Inference latency costs: We demonstrate that both RPM approaches continue to outperform the No-RPM baseline on average after accounting for LLM inference latency.
  • Compute overhead analysis: We further analyze how the Agentic RPM allocates its compute budget between pilot experimentation and full candidate execution, quantifying the computational overhead of candidate selection.
  • Tournament structure: We clarify the Agentic RPM's pairwise tournament structure, pilot experimentation budget, and accounting for runtime overruns.

We thank the reviewer for their comments and for recognizing our paper's strong motivation and clear presentation.

Below we address questions and comments raised by the review:

W1: Novelty over Goldie et al. (2026) and Zheng et al. (2026)

A1: We thank the reviewer for highlighting these related works. We believe our work makes several contributions beyond those introduced by Zheng et al. (2026) and Goldie et al. (2026).

Comparison with Goldie et al. (2026)

Goldie et al. suggest using an LLM judge or reward model to select promising leaves within an AI research agent's search tree to reduce the cost of search. However, they present this as a future research direction, without implementing or evaluating the proposed approach. Our work implements and evaluates this direction, demonstrating improvements through extensive offline and end-to-end evaluations, while further extending it through the introduction of Agentic RPMs.

Comparison with Zheng et al. (2026)

Unlike Zheng et al., our Inference-only RPMs leverage previously executed solutions from the search tree and their validation scores to inform the RPM's decision.

Beyond this, we contribute:

  1. Agentic RPMs, which conduct small-scale pilot experiments to inform candidate selection, substantially improving offline selection accuracy and end-to-end performance.
  2. A scaling analysis investigating the effects of historical context, reasoning effort, and suggestion count on the Inference-only RPMs selection accuracy. Our original offline evaluation shows that increasing context from 0 to 100 nodes improves accuracy from 60.7% to 65.0%.
  3. A more extensive end-to-end evaluation across four LLM backbones, 18 AIRS-Bench tasks, and 10 independent seeds per task and method. We also conduct oracle selection evaluations and analyze the correlation between selection quality and final scores. In comparison, Zheng et al. evaluate FOREAGENT with a single LLM backbone on only 5 MLE-bench tasks with 3 seeds each. Our evaluation therefore provides broader empirical evidence for the effectiveness of preference-guided candidate selection.

As part of the rebuttal period, we have conducted additional experiments to further evaluate the benefit of historical search-tree context.

First, we extend our original offline evaluation to two additional backbones:

Backbone0 Context (%)10 Context (%)
Qwen3.6-27B56.8857.19
Muse Glimmer59.5064.12

We further construct an offline evaluation dataset from the end-to-end AIRS-Bench runs presented in Section 5.1, obtaining the following selection accuracy results:

Context NodesLow Effort (%)High Effort (%)
055.0756.10
159.1359.73
1060.7063.17
10062.3063.43

These results further demonstrate the benefit of incorporating historical search-tree context, improving selection accuracy by up to 7.33 percentage points. On the new evaluation dataset, our Agentic RPM outperforms all inference-only configuration with a 74.5% selection accuracy, demonstrating the effectiveness of our novel pilot experimentation method.

W2: Although Figure 2 includes 95% confidence intervals, the paper does not report paired statistical tests establishing the significance of the differences between methods.

A2: To address the reviewer's concerns we conduct paired task-level t-tests across all four LLM backbones and the 18 benchmark tasks shared across methods. For each task, we first compute the difference in mean normalized scores between methods, averaging over seeds, and then average these differences across the four backbones. We perform a two-sided paired t-test on the resulting 18 task-level differences, avoiding treating model–task combinations as independent observations.

We find that the Agentic RPM achieves a statistically significant improvement over No RPM (mean difference +0.034, 95% CI [+0.005, +0.062], p=0.023), improving performance on 13 of the 18 tasks when averaged across models. The Inference-only RPM also improves over No RPM (+0.018, p=0.160) and the Agentic RPM improves over Inference-only RPM (+0.016, p=0.083), although neither differences are statistically significant. All p-values are unadjusted.

These findings provide further evidence that the Agentic RPM improves average E2E performance over the unguided baseline across the evaluated backbones, while accounting for task-level pairing.

Reviewer_CSfhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper develops a new method for AI research agents to select between candidate experiments during research development. They expand upon AIRA-dojo by sampling multiple candidates and selecting between them at each step with a Research Preference Model (RPM). They develop an inference-only RPM that uses an LLM as a judge and an agentic RPM that makes its selection after running pilot experiments on promising candidates. They evaluate their research agents on AIRS-Bench and achieve small improvements over the baseline AIRA-dojo agent, with the agentic RPM performing best across all models tested. They perform additional analysis on a subset of candidate selections from an external dataset to measure selection accuracy across different model configurations.

Critical strengths

The motivation for Research Preference Models is well written and reflects the importance of the problem. The authors select an important and actionable problem, and the value of this work will be very clear to the reader. More broadly, the writing is very clear.

The experimental design is rigorous. The end-to-end experiments are thorough and robustly evaluated. I trust that the prompts are well optimized and that evaluations are run a sufficient number of times to indicate legitimate results. The second section of experiments, which focuses on pairwise comparisons, provides a valuable analysis of research agent performance drivers.

Critical weaknesses

Each RPM has a fairly basic design, and its performance gains over the baseline in the main experiments are quite small. The baseline itself is very simple, so I question whether the already modest improvements reflect that simplicity or just that RPMs use more tokens for candidate selection rather than any strong methodological contribution.

Offline evaluations are run on a private dataset whose similarity to the end-to-end experiments readers cannot confirm.

Questions for authors

Why are offline evaluations on private data and restricted to 1,000 pairs? As I understand it, it should cost no extra training compute to do these evaluations on the end-to-end experiments following their completion. Would it just be expensive to run the evaluations? I understand that the offline evaluations you list were helpful for your own development, but results on the main tasks would be more enlightening.

Can you provide some analysis on the token costs of RPMs if it is available? In practice, both GPUs and tokens are costly, and I would imagine RPMs consume significantly more tokens than the baseline, so some analysis here would be useful to show whether the performance gains are truly worthwhile. One idea is to find the cost of compute at which your RPMs outperform the baseline when accounting for both the token costs of each LLM and the compute time at the selected cost. Identifying whether or not this is representative of a reasonable amount of computational resources would indicate the value of RPMs.

Did RPMs have internet access, and can you perform some checks to see if models are already aware of the research tasks and potential solutions? Analyzing chain-of-thought summaries, if they are saved in your logs, or prompting the agents about their awareness of the tasks up front would be helpful to check for data contamination.

8 comments

TLDR:

  • Private dataset for offline evaluation: We re-evaluate Inference-only and Agentic RPMs on a offline evaluation dataset compiled from public AIRS-Bench tasks, and demonstrate that the trends observed in our private offline evaluation generalize to the public AIRS-Bench tasks.
  • GPU and Token Costs: We analyze the combined GPU compute and LLM token costs required to reach the No-RPM baseline's final performance, demonstrating that RPMs reduce total monetary costs in 7 of 8 configurations, with savings of up to 45%.
  • Contamination: All of our experiments are on well established benchmarks, any contamination concerns would affect the methods we compared equally, and thus issues with evaluation contamination are not fair grounds for critiscism for this work. However, we conduct the contamination check suggested by the reviewer across all four LLM backbones, finding no evidence that the models recognize the specific AIRS-Bench task formulations or their state-of-the-art solutions.

We thank the reviewer for their comments and for recognizing our rigorous experimental design.

Below we address individual questions and comments raised by the reviewer. However, we first provide a direct summary of what we understand to be their main concern:

Gains over the baseline in the main experiments are quite small...

Our gains are +0.034 and +0.018 in normalised score for Agentic and Inference-only RPMs respectively, but that also includes three new SOTA results and savings of ~40% of the compute budget.

...The baseline itself is very simple,...

The baseline we use is the original AIRA-dojo [6], a well-established AI research agent, and one adopted by the original benchmark paper.

... so I question whether the already modest improvements reflect that simplicity or just that RPMs use more tokens for candidate selection rather than any strong methodological contribution

We show below that Inference-only RPMs outperform the baseline even when token cost is accounted for. Agentic RPMs also outperform both in final scores, and do so not just because they spend more on tokens, but because of their ability to leverage small amounts of compute to use that token cost effectively.

Importantly as well, our contributions extend beyond just our E2E evaluations. We provide extensive studies and ablations on improving the selection accuracy of RPMs, through incorporating historical context, pilot experimentation and ensembling.

Q1a: Offline evaluations are run on a private dataset whose similarity to the end-to-end experiments readers cannot confirm. Why are offline evaluations on private data and restricted to 1,000 pairs? ...

A1a: As the reviewer noted, we conducted our offline evaluations to develop and tune our RPMs before undertaking the computationally expensive E2E evaluations. We constructed the offline dataset using a separate set of tasks to avoid validation-test leakage. Specifically, we developed private tasks that follow the same AIRS-Bench task structure but differ in machine learning modalities and objectives.

We understand the reviewer's concerns regarding transparency, and look to release these tasks pending institutional approval. In the meantime, we can disclose that these private set of tasks, comprise of 40 tasks:

  • 25 Image-based tasks (Image Classification, Segmentation, Object Detection, Distillation, Captioning and OCR)
  • 5 Video-based tasks (Video Classification, Summarization, and Captioning
  • 5 Audio-based tasks (Speech Recognition and Classification)
  • 5 Reinforcement Learning tasks

We collected 1000 samples for our offline evaluation dataset to ensure rigorous testing while limiting evaluation costs.


Q1b: ... As I understand it, it should cost no extra training compute to do these evaluations on the end-to-end experiments following their completion. Would it just be expensive to run the evaluations? I understand that the offline evaluations you list were helpful for your own development, but results on the main tasks would be more enlightening.

A1b: Following the reviewer's comments, we construct an offline evaluation dataset using solutions generated during our E2E experiments. We sample 1,000 evaluation pairs following the same criteria introduced in Section 4.2, but using solutions from our E2E evaluations across all three methods (No RPM, Inference-only RPM, and Agentic RPM) and all four backbone models.

Random selection achieves a baseline accuracy of 50% on this dataset.

We evaluate the Inference-only RPM across all combinations of {0, 1, 10, 100} historical context nodes and {low, high} reasoning effort. The results are summarized below.

Context NodesLow Effort (%)High Effort (%)
055.0756.10
159.1359.73
1060.7063.17
10062.3063.43

We also evaluate the Agentic RPM with a 5-minute pilot budget, achieving a selection accuracy of 74.5%.

All offline evaluations use GPT-5 as the backbone and are conducted across three seeds.

These results provide further evidence that the trends observed in our private offline evaluation generalize to the public AIRS-Bench tasks. In particular, both evaluations demonstrate that increasing historical context and reasoning effort improves the Inference-only RPM's selection accuracy, while the Agentic RPM achieves substantially higher accuracy through pilot experimentation.

We thank the reviewer for their suggestion and update the manuscript to include the results above.

Q2: Can you provide some analysis on the token costs of RPMs if it is available?

A2: Following the reviewer's comments, we calculate the average total cost incurred by each method at the point where it reaches the No-RPM baseline's final normalized score. Total cost accounts for GPU hosting costs and LLM inference costs for both candidate generation and selection.

For GPU hosting, we use the publicly listed on-demand price of $6.31 per H200 GPU-hour from CoreWeave (our providers) [1]. For LLM inference, we use OpenAI's API pricing for GPT-5 and OpenRouter's listed pricing for the remaining self-hosted models:

  • GPT-5: 1.25 USD input / 10.00 USD output per million tokens [2].
  • Qwen3.6-27B: 0.25 USD input / 2.00 USD output per million tokens [3].
  • Muse Glimmer 30B: 0.30 USD input / 1.20 USD output per million tokens [4].
  • Gemma 4-31B: 0.10 USD input / 0.34 USD output per million tokens [5].

All prices are in USD, based on publicly listed rates as of November 10, 2026. The resulting costs are summarized below, including the break-even GPU rental price at which each RPM becomes cost-effective relative to No RPM.

ModelMethodGenerationSelectionGPUTotalSavingsBreak-even GPU Price
GPT-5No RPM$9.67—$143.00$152.67——
Inference-only$53.18$21.96$51.05$126.1917.3%$4.49/h
Agentic$20.61$58.12$68.28$147.013.7%$5.83/h
GemmaNo RPM$0.42—$150.31$150.73——
Inference-only$3.34$6.12$73.37$82.8345.0%$0.74/h
Agentic$1.60$10.37$103.69$115.6623.3%$1.56/h
GlimmerNo RPM$1.79—$149.28$151.07——
Inference-only$22.46$16.25$136.81$175.52−16.2%$18.68/h
Agentic$3.17$15.45$84.39$103.0131.8%$1.64/h
QwenNo RPM$2.40—$138.13$140.53——
Inference-only$18.53$13.70$97.18$129.417.9%$4.60/h
Agentic$15.55$22.78$98.50$136.832.6%$5.72/h

These results demonstrate that RPMs remain cost-effective even after accounting for additional inference costs, with 7 of 8 configurations reducing total cost relative to No RPM at equivalent performance. By reaching the baseline's final score in fewer GPU hours, RPMs save expensive experimental compute, with these savings outweighing the additional LLM inference costs. Overall cost savings reach 45.0%, with the Agentic RPM reducing costs across all four backbones. We amend Section 5.1 in the manuscript to detail the above analysis.

We thank the reviewer for suggesting this analysis and include it in the updated manuscript.

Reviewer_fCHtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AI Research Preference Models to rank unexecuted candidate solutions in research agents. The agent generates multiple candidate children per step, and an RPM selects one for full execution. The inference-only variant uses search history and code, while the agentic variant runs short pilot experiments. Evaluated across multiple benchmarks and backbones, both variants improve average scores and search efficiency over random selection.

Critical strengths
  1. Practical problem framing and strong validation. Full model training is much more expensive than code generation. Framing compute allocation as preference ranking is effective. RPMs successfully match the unguided baseline using a fraction of the compute budget.
  2. Clear scaling and scope analysis. Offline experiments systematically test context size, candidate pool size, and pilot budgets. The paper also provides an honest negative result showing that RPMs do not help after candidates are executed.
Critical weaknesses
  1. Missing comparisons against static judges and non-adaptive baselines. The only end-to-end baseline is random choice, and oracles are unbudgeted upper bounds. The paper does not compare live search history against static context, nor does it compare the agentic RPM against a fixed pilot script.
  2. Incomplete compute accounting and statistical analysis. LLM inference latency and candidate generation overhead are excluded from the main budget. There are also discrepancies between the pairwise tournament text and the multi-candidate algorithm, alongside overlapping confidence intervals without task-level breakdowns.
Questions for authors
  1. Does the pilot budget apply to each individual match or the entire tournament? How are runtime overruns and LLM inference latencies counted?
  2. How does the RPM compare against a static history-free judge and a fixed non-adaptive pilot policy under equal compute?
  3. What is the correlation within each method rather than pooled across methods? Please also provide task-level scores and clarify minor text inconsistencies.
5 comments

TLDR:

  • Static history-free judges and non-adaptive pilot policies: We run further experiments to show that Agentic RPMs are leveraging history, and are not 'just' naively applying basic simplifications to each solution.
  • Inference latency: We demonstrate that RPM approaches continue to beat the no-RPM baseline with inference latency considered.
  • Statistical significance: We conduct a paired t-test and show that the Agentic RPM significantly outperforms the no RPM baseline.
  • Pilot budget and tournament structure: We provide task-level breakdowns of scores and clarify questions related to the Agentic RPM's pilot budget and tournament.

We thank the reviewer for their comments. We also thank them for recognizing the practical problem framing, the strong empirical validation, and systematic analysis of RPM's scaling trends which we've conducted.

Below we address questions and comments raised by the review.

Q1: Missing comparisons against history-free static judges and non-adaptive baselines

A1: We interpreted the reviewers request of "history-free baselines" to mean "an RPM without historical context". We already provided this for GPT-5 in Section 5.2.1 and Figure 3, where we show that increasing the number of historical context nodes improves selection accuracy. At the reviewers request, we further extend this ablation to two additional backbones, Qwen3.6-27B and Muse Glimmer, comparing 10 provided context nodes vs none, and summarize results below:

LLM Backbone0 Context1 Context10 Context100 Context
GPT-560.70%60.90%62.70%65.00%
Qwen3.6-27B (new)56.88%—57.19%—
Muse Glimmer (new)59.50%—64.12%—

Across all three backbones, the history-free judge performs worst, demonstrating the benefit of incorporating previously executed solutions into candidate selection.

We interpreted the reviewers request of "non-adaptive baselines" to mean "an RPM that uses a fixed strategy to simplify the candidates before running them". This is in contrast to our agentic RPM that "adapts" the simplification strategy to the solution, ie it does not change the number of cross-fold validation steps if they are not prohibitive.

We conduct additional offline evaluations comparing the Agentic RPM against two fixed validation strategies that simplify the 5-fold cross-validation used in the original AIRA-dojo experiments. We instruct the agent to reduce this to either 3-fold CV, training on two folds and evaluating on the third, or a single 80/20 split, training on 80% of the data and evaluating on the remaining 20%. All configurations are given the same five-minute pilot budget.

LLM Backbone3-Fold CV80/20 SplitUnconstrained Agentic RPM
Qwen3.6-27B69.85%73.37%73.87%
Muse Glimmer68.64%71.16%72.16%

These results show that the Agentic RPM is doing more than just a simple fixed simplification strategy - it is chosing its validation strategy to get the most information gain from its pilot experiments.

Both the history-free judge and fixed pilot strategy evaluations use the same 1,000-example offline evaluation dataset presented in Section 4.2.

We thank the reviewer for their suggestion and update the manuscript to include both experiments.

Q2: LLM inference latency and candidate generation overhead are excluded from the main budget

A2: Our work presents a preliminary investigation into incorporating Research Preference Models (RPMs) within AI Research Agents (AIRAs), focusing on whether more informed candidate selection improves the allocation of expensive experimental compute. In real-world, industrial-scale ML research, where we see AIRAs ultimately being deployed, the computational cost of training and evaluating models (which can exceed thousands of GPU hours) can vastly outweigh the LLM inference required to generate the code. Consequently, our primary evaluation focused on the effectiveness of allocating a fixed experimental compute budget, rather than modeling inference latency as part of the main evaluation.

However, at the reviewer's request, we include a retrospective account of the RPM inference latency across our current E2E evaluations. For each run, we add the accumulated RPM selection inference latency to each node's original compute time, enforce a strict 24-hour cutoff, and select the best-by-validation solution among the remaining nodes. The candidate generation latency was already accounted for.

The table below presents the original and latency-adjusted average normalized scores across all four LLM backbones.

LLM BackboneMethodRPM Inference TimeOriginal ScoreLatency-Adjusted Score
GPT-5No RPM—0.7130.713
-Inference-only RPM1.91 h0.7380.738
-Agentic RPM9.38 h0.7510.735
Gemma 4-31BNo RPM—0.7330.733
-Inference-only RPM1.92 h0.7550.756
-Agentic RPM6.65 h0.7600.749
Qwen3.6-27BNo RPM—0.6840.684
-Inference-only RPM0.66h0.7110.708
-Agentic RPM9.93 h0.7290.687
Muse GlimmerNo RPM—0.6960.696
-Inference-only RPM1.72 h0.6940.696
-Agentic RPM10.09 h0.7200.701

Both RPM variants retain an improvement over No RPM after accounting for inference latency. Across the four backbones, Inference-only RPM improves average normalized score by +0.018, while Agentic RPM (which incurs greater inference overhead) improves it by +0.012.

We thank the reviewer for their comment and acknowledge that for the scale of AIRS-Bench evaluations, including inference latency reduces the advantage of the Agentic RPM. Nevertheless, our motivation is to build a method that can scale to the industrial setting, where LLM inference becomes increasingly negligible next to the cost of running and fully evaluating candidate solutions. The benefits of Agentic RPMs vs Inference-only naturally become most apparent in that paradigm.

We have updated Section 7 to acknowledge the current inference overhead and reference the latency-adjusted analysis in Appendix C.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.