acceptodds
Under review as a conference paper at ICLR 2027

Does Better Estimation Imply Better Selection? A Decision-Theoretic Audit of LLM Agent Updates

Abstract

Evaluating LLM-agent updates with low-cost proxies aims to reduce benchmarking expense, yet lower surrogate estimation error does not ensure better adoption decisions. We present a decision-theoretic audit framework that separates finite-pool mean squared error (MSE), same-pool decision regret, observed held-out gain, and acquisition cost. Our empirical benchmark evaluates three language models and twelve fixed update pairs on LiveCodeBench, using exact combinatorial integration across all target-sampling configurations to eliminate Monte Carlo acquisition noise over 12,384 recorded runs. In the post hoc subgroup of eight statically changed updates (Changed8), four-pair cross-fitting (CF4; two folds of two target task pairs) exhibits severe small-sample degeneracy at a 512-token proxy ceiling: both fitted slopes are zero in 95.71% of joint sampling configurations, yielding exactly the target-only mean on the same sample. Across all configurations in this regime, CF4 alters the corresponding target-only decision in only 0.076% of cases while requiring, on average, 2.40x as many cold-start acquisition completion tokens. At a 2,048-token proxy ceiling in the same subgroup, CF4 reduces mean MSE relative to unit-coefficient uniform correction but increases same-pool regret; externally calibrated correction reduces regret relative to CF4 but achieves lower observed held-out gain. Across all twelve updates, simultaneous 95% confidence bounds contain zero for both prespecified pooled utility contrasts. These findings reveal a low-budget regime where proxy adjustment adds substantial token overhead with little effect on decisions, demonstrating that surrogate evaluators must be benchmarked against matched target-only baselines through decision regret, held-out utility, and total acquisition cost rather than surrogate MSE alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent evaluation, update selection, proxy-assisted evaluation, decision making, cost-aware evaluation, multi-fidelity evaluation, LiveCodeBench

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UeuWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether using low-cost surrogate tasks for estimation when evaluating the updates of large language model agents can truly lead to better adoption decisions. The authors propose a decision-theoretic auditing framework that separates the finite pool mean squared error, pool decision regret, observed retained set gain, and acquisition cost. Through empirical evaluation on a real code benchmark of three language models and twelve fixed update pairs, and by using precise combination integration to eliminate Monte Carlo acquisition noise, a low-budget mechanism was discovered. Under this mechanism, although the agent adjustment may reduce the estimation error, it incurs a large amount of token cost and has little impact on the final decision. The paper convincingly demonstrates that the surrogate evaluator must be benchmarked against the matching direct target baseline by decision regret, retained set utility, and total acquisition cost, and cannot rely solely on the surrogate mean squared error.

Critical strengths
  1. The article did not merely focus on the traditional analysis of prediction errors. Instead, it introduced a decision theory perspective, clearly distinguishing between estimation accuracy and downstream decision utility. This has significant theoretical guidance for resource-constrained large language model intelligent agent search scenarios.

  2. The author employed an accurate combined integration method to traverse all target sampling configurations, thereby eliminating the noise brought by Monte Carlo sampling. This exhaustive verification over more than ten thousand run records significantly enhanced the statistical reliability of the experimental conclusions.

  3. The research clearly demonstrated that under a low agent token limit, cross-validation and other adaptive correction methods would exhibit severe small-sample degradation phenomena. The slope estimation frequently returned to zero, resulting in a very low rate of decision changes but consuming multiple times the acquisition cost. This discovery timely warns the current engineering practice that blindly pursues complex agent evaluation.

Critical weaknesses
  1. The experiment was conducted only on twelve fixed update pairs and specific code generation benchmarks, failing to cover a wider range of agent update types, such as modification of tool invocation logic or adjustment of long-context reasoning strategies. This limits the generalizability of the conclusions to more complex agent workflows.

  2. The article mainly focuses on the consumption of tokens, but does not fully consider the combined computational cost resulting from concurrent execution, cache hit rate, or different model call delays in the actual system. This makes the assessment dimension of acquisition cost somewhat thin.

  3. The research focused on the static selection decisions made after generation, while the actual large language model agent search is an iterative process. If it could further demonstrate how this decision theoretical framework affects the final search efficiency and result quality in the complete multi-round automated search cycle, the persuasiveness of the paper would be significantly enhanced.

Questions for authors

see weakness

Reviewer_7bPYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines whether improved surrogate estimation leads to better selection decisions for LLM-agent updates. Through extensive empirical analysis, the authors show that lower estimation error does not necessarily improve downstream utility, while proxy-based evaluation may incur substantial additional costs.

Critical strengths

The paper provides a relatively thorough empirical analysis, including extensive comparisons, exact enumeration of sampling configurations, and careful accounting of evaluation costs.

Critical weaknesses
  1. Poor presentation and readability. The paper is difficult to follow due to its convoluted organization and excessive use of experiment-specific abbreviations and labels (e.g., Changed8, CF4, C1/C2). The writing often feels overly mechanical and formulaic, and the logical flow is not sufficiently clear. Overall, the presentation substantially obscures the main ideas and findings.

  2. Limited novelty. While the investigated problem is meaningful, the central observation that improved estimation accuracy does not necessarily translate into better downstream decisions is already well established. The paper mainly provides an empirical illustration of this phenomenon in LLM-agent evaluation, without introducing substantial new theoretical insights or methodological contributions. Moreover, the empirical conclusions are limited to a relatively narrow experimental setting. Despite the careful experimental analysis, I find the overall contribution insufficient for ICLR.

Questions for authors
  1. Could the authors clarify what fundamentally new insights this work provides beyond existing studies on the gap between estimation accuracy and downstream decision quality?

  2. Could the authors substantially improve the organization and presentation, particularly by simplifying the terminology and making the main contributions and conclusions easier to follow?

Reviewer_uBQBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether more accurate proxy-based estimators actually lead to better decisions about adopting updates to LLM-agent workflows. It evaluates multiple sampling, correction, and selection strategies on LiveCodeBench.

The authors distinguish four quantities: finite-pool estimation MSE, decision regret, held-out utility, and acquisition cost. They employ exact combinatorial integration to eliminate Monte Carlo sampling noise from the evaluation of fixed selection policies.

The central finding is that small-sample cross-fitted proxy corrections can collapse to direct target estimation, incurring considerable proxy acquisition costs with almost no change in the resulting selection decisions.

Critical strengths
  1. The work addresses a meaningful problem with an appropriate decision-oriented perspective. Agent workflow optimization often involves repeated decisions about whether candidate mutations should be retained. Evaluating the quality of those decisions, rather than merely the accuracy of intermediate estimates, is practically important.

  2. The empirical methodology is careful for a negative-results paper. Exact combinatorial integration is an important strength. It removes sampling noise associated with repeated hypothetical target-subset acquisition, while appropriately conditioning the conclusions on recorded model executions.

    The authors also introduce relevant controls, distinguish statically changed from inert mutations, acknowledge service nondeterminism, and avoid attaching unjustified significance claims to descriptive comparisons.

  3. The cross-fitting degeneracy is quantitatively compelling. On Changed8/L512, both fitted slopes vanish in 95.71% of enumerated configurations, while the cross-fitted estimator changes the matched target-only decision in only 0.0759% of configurations. Yet CF4 consumes approximately 2.40 times the acquisition completion tokens of TARGET_RANDOM4.

Critical weaknesses
  1. Limited novelty relative to established decision-focused learning. The distinction between estimation accuracy and downstream decision quality is already central to decision-focused learning. For example, Smart “Predict, then Optimize” explicitly motivates optimization based on decision losses rather than prediction losses.
  2. The principal degeneracy may be driven by the specific adaptive estimator design. CF4 estimates each regression slope using only two target pairs and clips it to [0, 1]. Under sparse discrete outcomes, such estimates are particularly prone to collapsing to zero. Although the CF8 experiments partially address this concern, it remains unclear whether the findings extend to more stable adaptive estimators.
  3. Generalization and decision opportunities are limited. The evaluation covers one code-generation benchmark, three models, and twelve fixed updates, including four statically inert pairs. Most changed model–update combinations also have nonpositive held-out margins, making conservative rejection unusually competitive. The results may not generalize to broader agent-search settings with more beneficial candidate updates.
  4. The cost comparisons need a more equal-total-budget frontier. CF4 is primarily compared with TARGET_RANDOM4, which uses the same number of target evaluations but substantially fewer total tokens. Comparisons with stronger target-only baselines at similar total acquisition costs, particularly TARGET_RANDOM8, would better establish the practical value of proxy-assisted evaluation.
Questions for authors
  1. How does CF4 compare with target-only selection at similar total acquisition costs, particularly TARGET_RANDOM8?
  2. Does the observed collapse persist with more stable coefficient estimation, such as pooled or regularized slopes, or when restricting the analysis to cells with informative proxy variation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.