- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to choose a controller for LLM action decisions, such as answering, using more context, calling a stronger model, or abstaining. It considers four controller classes, from fixed actions to learned routers and prior-based rules. The method uses the possible gain, feature quality, and sample size to guide this choice. The experiments show that simpler controllers can perform better on some tasks. Although the authors have made good efforts, some issues still need to be addressed.
Critical strengths
-
The paper studies a useful problem. More complex routers do not always work better when training data are limited. The proposed framework helps readers think about whether a task needs a learned router or whether a simpler rule is enough.
-
The experiments cover several tasks and include tests with different judges and loss weights. The OCR experiment also gives the same information to different methods for comparison. These tests, together with the reported failure case, help show the benefits and limits of the method.
Critical weaknesses
1.The class definitions and the comparisons do not fully match. Section 2.1 defines Π₂ to include Π₁. Thus, the best policy in Π₂ cannot be worse than the best policy in Π₁. However, Theorem 4 compares their gain estimates to choose a winning class, while the experiments compare separate model families. The paper needs to explain this difference.
- The binary analysis does not fully support the four-action selection rule. The main threshold studies whether to answer or use a fallback, but the tested controllers choose among four actions. The SST-2 failure shows that a weak rejection signal does not mean other action choices are useless. It is unclear when the rule can safely remove a whole controller class.
Questions for authors
-
In Section 2.1, Theorem 4, and Appendix B.3, the authors should clarify whether they compare policy sets or specific learning methods. They should make the definitions, theory, and experiments consistent.
-
In Section 2.2 and Appendix E.6, the authors should check the full condition required by Theorem 2. The theorem needs more than agreement in sign, but Table 10 mainly checks the sign. The authors should explain whether the stronger condition holds and what happens when it fails.
-
The authors should better explain how the binary result applies to four-action decisions. A sample size below the stated bound only means that the bound gives no guarantee. It does not show that the whole class cannot improve performance. The selection rule should reflect this limit.
-
In Section 3.1, the authors should test the rule on new tasks after fixing all selection steps. They should report when it chooses the wrong class and how much loss this causes. This would help readers understand when the method works and when it may fail.