acceptodds
Under review as a conference paper at ICLR 2027

RAISE: Diagnosing Acquisition Collapse In Costly LLM Signals

Abstract

Large language models (LLMs) are increasingly used as costly, on-demand components in real systems, but calling them indiscriminately can waste substantial compute, latency, and serving budget. The key deployment question is therefore not only whether an LLM helps on average, but when it is worth calling. We identify a common failure mode, which we call acquisition collapse: an LLM signal can appear useful in aggregate or post hoc, yet still provide too little before-call information to support reliable selective use. We introduce RAISE (Reward-SNR Actionability in Signal Evaluation), a pre-routing diagnostic framework for testing whether a costly signal is truly actionable before committing to a routing strategy. We instantiate RAISE with Structured Hypothesis Embeddings (SHE), a frozen-LLM intent signal for recommendation using one LLM call per user, and evaluate it through controlled, retrospective, and fresh-cohort studies and a prospective pilot whose audit decisions are frozen before independent outcomes are revealed. Across these settings, predictable incremental benefit, not average lift alone, is what determines whether selective LLM calling becomes deployable, while seemingly strong oracle or subgroup gains can disappear under independent evaluation. More broadly, RAISE reframes costly inference as an information-acquisition problem: before paying for an expensive model, tool, sensor, or measurement, first test whether its value is predictable at decision time. This principle motivates cost-aware acquisition in settings ranging from agent tool use and stronger-model consultation to robotic sensing and clinical decision pipelines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Cost-Aware Routing, Active Feature Acquisition, Policy Learning, Heterogeneous Treatment Effects, Selective Acquisition, Recommender Systems, Selective Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XtvTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Problem. The paper studies LLM routing: when is it worth calling an expensive AI component? The quantity of interest is how much calling the LLM for a given user improves the outcome relative to not calling it. The authors define acquisition collapse as the situation where the average effect is positive, or post hoc selection appears to yield gains, yet a router trained only on before-call information fails to establish a selection gain over budget-matched random calling on independent data.

Proposed method. RAISE is an audit procedure run before deploying a router. It checks whether the routing problem is well posed at all, and whether the gain obtained by a given router is credible. Concretely, it first tests whether the LLM helps on average, then tests whether the differences in benefit across users are real and whether they can be predicted from before-call information, and evaluates the router on fully independent data. Finally, it incorporates the call cost to decide whether selective calling, calling for everyone, or calling for no one is the most cost-effective option.

Experiments. The experiments include synthetic data, retrospective analyses on the original MIND and Amazon cohorts, Fresh MIND with 4,954 new users, and a prospective pilot with decisions frozen before outcomes (MIND and Amazon, each with GRU and SASRec backbones). On Fresh MIND, most routers achieve positive held-out selection gains: calling the LLM for 30% of users retains roughly 70% of the gain from calling it for everyone. In the prospective pilot, the gain on the primary endpoint (MIND GRU) is above zero but does not reach the target ε = 0.005, while both Amazon conditions reach the target.

Conclusion. The paper identifies and demonstrates that average lift does not imply routability, but it cannot yet show that RAISE judges better than mean-only, HTE-only, a direct cross-validation pilot, or even always-approve.

Critical strengths
  1. The problem is practically relevant, and distinguishing average benefit from predictable incremental benefit is reasonable.

  2. The paper explicitly connects LLM routing with causal inference methods.

Critical weaknesses
  1. The audit framework itself is not validated.

The evaluation covers only a recommendation task and a single LLM signal, and relies mainly on offline metrics. To compare RAISE against baselines (other audit methods), one would need many independent routing scenarios, establish the ground truth for each on larger data, and then compare metrics such as FPR and cost. The paper lacks audit-level metrics: it mainly reports each router's G_0.30, which is a router outcome metric rather than a metric of how good the audit framework is overall.

Baselines are present, but the comparison has no discriminative power. The paper compares mean-only, HTE-only, a CV pilot, and always-approve / always-defer. However, on the two clearly successful Amazon conditions, RAISE-v1, the simple rules, and always-approve all made the right call, so the experiments cannot show that RAISE is better.

Overall, the current experiments show neither that RAISE-v1 is more accurate than simple audit rules nor that RAISE can reliably predict whether a router is worth deploying.

  1. Limited novelty. RAISE reads more as an organization or report of existing practice than as a new algorithm or statistical method. It introduces no new methodological component, and the literature it borrows from already offers more complete inference tools. The evaluation metric G_b is the existing PAPE, the argument that an average effect is not the same as targetable heterogeneity comes from prior work, and the routing literature has already reported the same phenomenon.
Questions for authors

N/A

Reviewer_pmuVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether an expensive LLM-derived signal can be selectively acquired using information available before a call. It distinguishes average incremental benefit from budget-matched selection gain, expressing the latter as the covariance between acquisition decisions and conditional benefit. RAISE combines reward-precision checks, predictable heterogeneity, held-out policy evaluation and cost analysis. The experiments include a synthetic controlled feature-alignment intervention and, using the SHE recommendation signal, retrospective MIND/Amazon analyses, a fresh MIND cohort and a prospectively frozen four-condition pilot. The intended contribution is an acquisition-auditing framework and diagnostic evidence, rather than a new routing algorithm.

Critical strengths
  1. The controlled experiment isolates the central distinction effectively. Section 4 and Figure 2 hold rewards fixed while disrupting their relationship with before-call features. This demonstrates why positive average benefit need not imply selective value, and why predictable heterogeneous benefit can support selection even with zero mean. The budget-matched endpoint in Section 2.1 makes the distinction operational and connects it explicitly to existing treatment-rule evaluation.

  2. The evaluation separates important data roles and reports uncertainty carefully. In the prospective pilot, audit, router-training and evaluation users are disjoint, and selections are locked before outcomes are scored (Section 5.3; Appendix E). The manuscript reports negative results, distinguishes marginal policy intervals from stronger family-level claims, and acknowledges the limitations of historical perturbation references. These choices improve confidence in the narrowly stated empirical findings.

Critical weaknesses
  1. The incremental decision value of RAISE remains insufficiently demonstrated. Its endpoint and core separation build on acknowledged PAPE/RATE and heterogeneous-treatment-effect methodology. The four-condition prospective pilot does not establish that the proposed audit guides decisions better than simpler screens: mean-only, HTE-only, RAISE-v1 and always-approve make the same approvals on the two correlated Amazon conditions exceeding the practical target, while both MIND outcomes remain unresolved against that target (Section 5.3; Appendix E, especially Table 11). The MIND abstentions therefore cannot be counted as errors, but neither can the pilot establish discrimination against genuinely below-target settings. The authors acknowledge this (Section 5.3; Section 7), but it leaves the central contribution claim unsupported: a pre-specified comparison spanning independently resolved useful and below-target cases, with false approvals, missed opportunities and audit costs, would be needed to establish the framework's practical contribution. The present results support careful evaluation practice more strongly than an additional validated diagnostic method.

  2. The concrete v1 approval rule is not adequately justified by the stated selection objective. Section 2.1 shows that selection gain is invariant to a uniform benefit shift, yet RAISE-v1 requires both positive mean benefit and predictable heterogeneity (Appendix E.3). The paper's own Section 4 screen, which conjoins the same two tests, withholds approval in 19 of 20 validation draws of the zero-mean heterogeneous case despite recoverable selective value (Figure 2B; Appendix B), and the authors note this miss is structural. A positive mean is relevant to acquire-all versus skip, but is not necessary for a useful selective policy. The rule is intentionally conservative rather than mathematically invalid; the unresolved issue is what utility or risk criterion justifies this conservatism and whether its reduced false approvals compensate for the missed opportunities. The current pilot does not establish that tradeoff.

Questions for authors
  1. What decision-relevant evidence distinguishes RAISE from an HTE-only screen or a small direct policy pilot, particularly on conditions conclusively below the practical target?
  2. What explicit utility or risk objective motivates the positive-mean conjunction? How do its false approvals and missed opportunities compare with a rule that evaluates selective utility directly?
Reviewer_6JXDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAISE, a pre-routing audit for assessing whether the benefit of acquiring a costly signal is predictable from before-call information and whether selective acquisition is worthwhile. The authors instantiate the audit using Structured Hypothesis Embeddings (SHE), a frozen LLM-generated intent signal for recommendation, and evaluate it in controlled studies.

Critical strengths

The proposed framework addresses a practical problem: a cost-aware planner needs to determine not only whether an expensive inference improves decision quality, but also whether it can predict when that inference is worth acquiring.

Critical weaknesses
  1. The prospective pilot does not yet establish that RAISE provides decision value beyond simpler rules. RAISE-v1 records insufficient evidence for the primary MIND endpoint. In the two Amazon settings that exceed the target, simpler rules make the same approvals.

  2. Research in this area is moving fast. Similar idea on "pre-routing" analysis to decide the cost-benefit has been discussed (see Question for details).

  3. The methodological contribution appears to lie mainly in combining existing ideas from heterogeneous treatment-effect and policy evaluation into a pre-routing audit, rather than introducing a new estimator.

Questions for authors
  1. Since the proposed method matches the simpler rules on the two confirmed-useful secondary endpoints, it would be more convincing to provide more evidences that RAISE can add decision value beyond those rules.

  2. The following recent works seem to be related but not cited. What are the distinctions of the proposed method from them?

[1] Q. Wu et al. To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling, arXiv 2026. [2] H. Xu et al. Alignment for Efficient Tool Calling of Large Language Models, EMNLP 2025. [3] B. Liu et al. Utility-Guided Agent Orchestration for Efficient LLM Tool Use. arXiv 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.