- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Problem. The paper studies LLM routing: when is it worth calling an expensive AI component? The quantity of interest is how much calling the LLM for a given user improves the outcome relative to not calling it. The authors define acquisition collapse as the situation where the average effect is positive, or post hoc selection appears to yield gains, yet a router trained only on before-call information fails to establish a selection gain over budget-matched random calling on independent data.
Proposed method. RAISE is an audit procedure run before deploying a router. It checks whether the routing problem is well posed at all, and whether the gain obtained by a given router is credible. Concretely, it first tests whether the LLM helps on average, then tests whether the differences in benefit across users are real and whether they can be predicted from before-call information, and evaluates the router on fully independent data. Finally, it incorporates the call cost to decide whether selective calling, calling for everyone, or calling for no one is the most cost-effective option.
Experiments. The experiments include synthetic data, retrospective analyses on the original MIND and Amazon cohorts, Fresh MIND with 4,954 new users, and a prospective pilot with decisions frozen before outcomes (MIND and Amazon, each with GRU and SASRec backbones). On Fresh MIND, most routers achieve positive held-out selection gains: calling the LLM for 30% of users retains roughly 70% of the gain from calling it for everyone. In the prospective pilot, the gain on the primary endpoint (MIND GRU) is above zero but does not reach the target ε = 0.005, while both Amazon conditions reach the target.
Conclusion. The paper identifies and demonstrates that average lift does not imply routability, but it cannot yet show that RAISE judges better than mean-only, HTE-only, a direct cross-validation pilot, or even always-approve.
Critical strengths
-
The problem is practically relevant, and distinguishing average benefit from predictable incremental benefit is reasonable.
-
The paper explicitly connects LLM routing with causal inference methods.
Critical weaknesses
- The audit framework itself is not validated.
The evaluation covers only a recommendation task and a single LLM signal, and relies mainly on offline metrics. To compare RAISE against baselines (other audit methods), one would need many independent routing scenarios, establish the ground truth for each on larger data, and then compare metrics such as FPR and cost. The paper lacks audit-level metrics: it mainly reports each router's G_0.30, which is a router outcome metric rather than a metric of how good the audit framework is overall.
Baselines are present, but the comparison has no discriminative power. The paper compares mean-only, HTE-only, a CV pilot, and always-approve / always-defer. However, on the two clearly successful Amazon conditions, RAISE-v1, the simple rules, and always-approve all made the right call, so the experiments cannot show that RAISE is better.
Overall, the current experiments show neither that RAISE-v1 is more accurate than simple audit rules nor that RAISE can reliably predict whether a router is worth deploying.
- Limited novelty. RAISE reads more as an organization or report of existing practice than as a new algorithm or statistical method. It introduces no new methodological component, and the literature it borrows from already offers more complete inference tools. The evaluation metric G_b is the existing PAPE, the argument that an average effect is not the same as targetable heterogeneity comes from prior work, and the routing literature has already reported the same phenomenon.
Questions for authors
N/A