acceptodds
Withdrawn from ICLR 2027

A Regime Theory of Controller Class Selection for LLM Action Decisions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language and vision-language models must decide whether to answer directly, retrieve evidence, defer to a stronger model, or abstain. Existing routing methods refine routers of a preselected type but offer little guidance on choosing the type itself. We address this gap through controller class selection, where a controller maps inputs to actions. We organize controllers into four classes: fixed actions, partition routers, instance-level learned controllers, and prior-gated controllers. With finite data, greater flexibility is not always better: different statistical regimes favor different classes. Our framework guides class selection using three measurable properties: how much improvement is possible beyond the best fixed action, whether there are enough samples to reliably detect gains from instance-level signals, and how much improvement a coarse partition can recover. We derive a finite-sample threshold for certifying selective prediction gains, with a matching-order lower bound, alongside bounds on residual improvement and partition gains. These results motivate a selection procedure that computes its diagnostics from training data alone. Across ten benchmark settings evaluated under strict nested cross-validation, the predicted class matches the empirical winner in nine. Replacing the judge on the same dataset shifts the diagnostics and reverses the winning class, as predicted. On TextVQA, an OCR experiment further separates the benefit of additional information from that of a deterministic gate, which outperforms the tested learners given the same information. Code is available at https://github.com/Anonymous-Awesome-Submissions/Regime-Theory.

Primary area:
learning theory
Keywords:
large language models, statistical learning theory, model selection, selective classification, sample complexity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nvBTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to choose a controller for LLM action decisions, such as answering, using more context, calling a stronger model, or abstaining. It considers four controller classes, from fixed actions to learned routers and prior-based rules. The method uses the possible gain, feature quality, and sample size to guide this choice. The experiments show that simpler controllers can perform better on some tasks. Although the authors have made good efforts, some issues still need to be addressed.

Critical strengths
  1. The paper studies a useful problem. More complex routers do not always work better when training data are limited. The proposed framework helps readers think about whether a task needs a learned router or whether a simpler rule is enough.

  2. The experiments cover several tasks and include tests with different judges and loss weights. The OCR experiment also gives the same information to different methods for comparison. These tests, together with the reported failure case, help show the benefits and limits of the method.

Critical weaknesses

1.The class definitions and the comparisons do not fully match. Section 2.1 defines Π₂ to include Π₁. Thus, the best policy in Π₂ cannot be worse than the best policy in Π₁. However, Theorem 4 compares their gain estimates to choose a winning class, while the experiments compare separate model families. The paper needs to explain this difference.

  1. The binary analysis does not fully support the four-action selection rule. The main threshold studies whether to answer or use a fallback, but the tested controllers choose among four actions. The SST-2 failure shows that a weak rejection signal does not mean other action choices are useless. It is unclear when the rule can safely remove a whole controller class.
Questions for authors
  1. In Section 2.1, Theorem 4, and Appendix B.3, the authors should clarify whether they compare policy sets or specific learning methods. They should make the definitions, theory, and experiments consistent.

  2. In Section 2.2 and Appendix E.6, the authors should check the full condition required by Theorem 2. The theorem needs more than agreement in sign, but Table 10 mainly checks the sign. The authors should explain whether the stronger condition holds and what happens when it fails.

  3. The authors should better explain how the binary result applies to four-action decisions. A sample size below the stated bound only means that the bound gives no guarantee. It does not show that the whole class cannot improve performance. The selection rule should reflect this limit.

  4. In Section 3.1, the authors should test the rule on new tasks after fixing all selection steps. They should report when it chooses the wrong class and how much loss this causes. This would help readers understand when the method works and when it may fail.

Reviewer_kp9WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Motivation: Before optimizing a specific router model, one should first determine which decision structure (i.e., how complex a routing design) the available data can actually support. This is especially useful for small-sample routing: when a complex model performs poorly, the cause is not necessarily an inadequate model structure; the data may simply be insufficient to reliably estimate per-instance decisions.

Proposed approach: The paper first defines a unified decision objective, then identifies the best fixed action and checks whether there is room for improvement beyond it, determines whether the data are sufficient to train a per-instance router, compares group-level decisions with per-instance decisions, and finally optimizes within the selected class.

Experimental design: The authors first generate the outcomes of four actions for each sample and construct a loss matrix from correctness, semantic risk, and operational cost. They then use diagnostics computed on training data to select a controller class, choose a specific model within that class via nested cross-validation, and test on held-out data whether the procedure selects the routing structure with the lowest loss.

Conclusion: The most complex router is not always the best; the controller structure should be chosen according to the task's room for improvement, the strength of the feature signal, and the sample size.

Critical strengths
  1. It offers a valuable research perspective: instead of assuming by default that a complex router is optimal, it asks whether the data support that level of complexity.

  2. Unified framework: answering directly, retrieving, deferring, and abstaining are all compared under a single loss function.

Critical weaknesses
  1. Experimental design: The paper proposes a complexity-selection prior that may be more robust than directly comparing validation scores, but it has not sufficiently shown that this prior generally outperforms standard model selection.

The most critical comparison is missing: how much worse would the results be if the diagnostics were skipped and all classes were simply placed into nested CV for selection? A more convincing experiment would directly compare against baselines such as methods that favor simpler models, complexity penalties, and plain cross-validation, and then evaluate, across many new tasks and repeated data resampling, which approach more often selects the right class and which yields the lowest deployed loss.

In addition, external validation on independent tasks is lacking: the diagnostic rules and the experimental datasets were designed and validated within the same research process, which the authors acknowledge in the Limitations section.

  1. The experiments do not yet sufficiently demonstrate the generality of the method.

The core experimental result is that the method predicts which class to use correctly in 9 of 10 test settings, but 3 of these are numerical ties that do not exceed random noise.

The results depend on the loss weights and the judge: changing the weights on errors, semantic risk, and cost may change the best class, and replacing the judge on HallusionBench also flips the best class.

The retrieve action is not real retrieval. Π1 is instantiated only with KMeans and CART, and Π2 only with HGBC and logistic regression; on the core benchmarks, the inputs are all 12 to 41 dimensional handcrafted scalar features (only 12 dimensions on FOLIO).

Questions for authors

N/A

Reviewer_82rxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies which type of controller to use when an LLM can answer, retrieve, defer, or abstain. It organizes policies into fixed actions, partition routers, instance-level controllers, and controllers with a prior gate. Residual improvement, an AUC-based sample-size diagnostic for a binary rejector, and partition gains are combined into a class-selection rule. The experiments compare controller families using nested cross-validation. The paper reports agreement with the empirical winning class on nine of ten benchmark settings, with prospective selection evaluated on eight. Additional experiments change the evaluation judge and compare OCR gating with learners given the same OCR information. The question is useful, but the claimed theoretical basis for selecting a class is not established by the current analysis.

Critical strengths
  1. The paper puts a useful practical decision at the center of the study: whether the available data justify learning a more flexible router at all. Comparing fixed policies and partition routers with instance-level learners is valuable, particularly when the labeled routing dataset is small. The FOLIO comparison gives a concrete example of why tuning only an instance-level family can miss a better deployed solution.

  2. The judge and OCR controls help interpret the empirical results. Changing the HallusionBench judge while holding the items, policies, and features fixed shows how the preferred controller depends on the loss being optimized. The equal-information OCR experiment is especially useful: Table 12 separates the benefit of providing OCR features from the remaining 0.0060 loss advantage of the gate over the best tested learner with those features. This is stronger evidence than a comparison in which only the gate receives OCR.

Critical weaknesses
  1. The central class-selection claim conflicts with the policy-class definition. Definition 1 explicitly sets Π2=Π1∪F\Pi_2=\Pi_1\cup\mathcal F, so inf⁡π∈Π2R(π)≤inf⁡π∈Π1R(π)\inf_{\pi\in\Pi_2}R(\pi)\leq\inf_{\pi\in\Pi_1}R(\pi). Theorem 4(iii), however, says that the population-optimal winner between these classes is determined by comparing CΠ1C_{\Pi_1} and CΠ2C_{\Pi_2} (p. 5, lines 255–258). A smaller nested class cannot have strictly better population-optimal risk. There is a second issue in that comparison: q∗β(Lw−Lr)q^*\beta(L_w-L_r) is derived as a gain scale for a particular binary rejector, not as an upper bound on all policies in Π2\Pi_2. Calling it a ceiling does not make it one. The SST-2 result, where this quantity is negative but the deployed controller improves, makes the distinction concrete. Comparing the finite-sample risks of specified learning algorithms could support the intended argument, but that is a different statement and needs its own justification.

  2. The reported diagnostics do not support the assumption needed for the sample-size certificates. Theorem 2 requires μw(q)−αmin⁡≥β\mu_w(q)-\alpha_{\min}\geq\beta, not merely agreement of signs. At the operating q=0.3q=0.3, Table 10 reports tail margins of 0.265, −0.162, and 0.033 for HallusionBench, A-OKVQA, and FOLIO, against β\beta values of 0.469, 0.246, and 0.053. None of these reported estimates satisfies the stronger condition; A-OKVQA also fails the sign condition. The paper acknowledges the latter and explains that its deployed controller has four actions, but this does not justify applying the binary theorem to certify that controller. These are estimated quantities, so they do not by themselves disprove a population assumption. They do show that the paper has not supplied evidence for the assumption on which its numerical certificates depend. The nine-of-ten agreement can support an empirical heuristic; it does not resolve this gap in the theory.

Questions for authors
  1. In Theorem 4(iii), is the intended comparison between population optima over the nested sets, or between the expected risks of particular fitted algorithms? Please give the corrected statement and explain which result makes CΠ2C_{\Pi_2} an upper bound for the object being compared.

  2. How are the Theorem 2 certificates justified given the measurements in Table 10? Please distinguish sign agreement from the quantitative condition, and identify which benchmark conclusions remain theoretically supported after accounting for estimation uncertainty. If the four-action results require a different assumption or analysis, please state it explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.