- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes HARISSA, a framework for efficient and reliable local language model inference. It leverages hidden-state-based correctness estimation at two stages: a prefill probe predicts whether an inference action will succeed before generation, while an answer probe evaluates correctness after generation. These signals guide a confidence-based cascade that adaptively allocates computation and defers unreliable answers to humans. Experiments on three multiple-choice benchmarks evaluate two settings: adaptive reasoning with a single on-device model and cascading across multiple model sizes on a local server. Results demonstrate improved accuracy–latency trade-offs over FrugalGPT and Self-REF, along with more effective identification of incorrect answers for selective deferral.
Critical strengths
-
Strong Accuracy–Efficiency Trade-off. HARISSA achieves competitive accuracy with reduced inference latency. In the device setting, it approaches CoT accuracy with 2.7× lower latency.
-
Effective Integration of Self-Assessment Mechanisms. The framework combines pre-generation and post-generation correctness estimation to support adaptive inference and selective deferral.
Critical weaknesses
- Model routing and model cascading have been extensively studied in prior work, including FrugalGPT, RouteLLM, Hybrid LLM, ThinkSwitcher, and DiffAdapt. HARISSA primarily integrates established techniques, such as hidden-state correctness estimation, LoRA fine-tuning, linear probes, and confidence-based cascading, without introducing substantial algorithmic or technical novelty.
- Although the paper emphasizes the unique challenges of local-only deployment, HARISSA is essentially a generic adaptive inference framework. The device and server settings mainly differ in their action spaces, without deployment-specific designs addressing resource limitations, privacy requirements, or hardware constraints.
- The experiments primarily compare against FrugalGPT and Self-REF, omitting several relevant methods for model routing and cascading (e.g., Cascade Routing [1], RLCascadeRouter [2]), adaptive reasoning (e.g., ThinkSwitcher [3], Sonata [4], DiffAdapt [5], and RPO [6]), and hidden-state-based correctness estimation (e.g., IntroLM [7]). Moreover, in the server setting, a fixed 4B model outperforms both baselines at their selected operating points across all three benchmarks, further weakening the empirical comparisons.
- The evaluation covers only three multiple-choice benchmarks, two of which are medical QA tasks, without assessing open-ended generation or real-world applications. The server experiments are restricted to Qwen 3.5 models ranging from 2B to 27B, lacking evaluations with larger or heterogeneous model configurations. Additionally, some results rely on a single random seed, with insufficient statistical significance and cross-domain robustness analyses.
- The paper emphasizes safe deployment but evaluates reliability primarily through answer correctness estimation and selective deferral, without addressing broader safety concerns such as harmful content generation, privacy leakage, or adversarial robustness. Consequently, the use of "Safe" in the title may overstate the actual scope of the contributions.
[1] A Unified Approach to Routing and Cascading for LLMs [2] RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning [3] ThinkSwitcher: When to Think Hard, When to Think Fast [4] AdaptThink: Reasoning Models Can Learn When to Think [5] DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference [6] Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking [7] IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation
Questions for authors
See weaknesses.
4 comments
The experiments primarily compare against FrugalGPT and Self-REF, omitting several relevant methods for model routing and cascading (e.g., Cascade Routing [1], RLCascadeRouter [2]), adaptive reasoning (e.g., ThinkSwitcher [3], Sonata [4], DiffAdapt [5], and RPO [6]), and hidden-state-based correctness estimation (e.g., IntroLM [7]).
We ran an experiment for each group of methods in your list, adapting them to our setting. First, the adaptive reasoning methods (ThinkSwitcher, Sonata, DiffAdapt, RPO, and AdaptThink) decide before the answer exists whether or how the model reasons. On the device, we compared HARISSA with a policy that decides from the prefill probe alone whether the model reasons, and delivers the answer without checking it. Second, Cascade Routing and RLCascadeRouter decide at each step which model to run next, or to stop. On the server, we ran a decision rule of Cascade Routing's kind over the four model sizes, with our two probes as its quality estimates. Third, IntroLM decides before generation which of two models answers. On the server, we compared HARISSA with a policy that chooses among the four model sizes from the prefill probe alone. The table reports the cost that HARISSA saves against each policy (the other policy's cost minus HARISSA's), where a policy's cost is its wrong answers per 100 queries plus its mean latency and with paired 95% bootstrap intervals over test queries.
| Cost saved by HARISSA (95% CI) | MedQA | MedMCQA | BBH |
|---|---|---|---|
| vs reasoning decided before the answer, device (same kind of decision as ThinkSwitcher, Sonata, AdaptThink, DiffAdapt, RPO) | 3.68 [2.51, 4.82] | 1.09 [−0.47, 2.67], tie | 2.66 [1.30, 4.04] |
| vs Cascade Routing rule, server (same kind of decision as Cascade Routing, RLCascadeRouter) | 1.30 [0.13, 2.45] | 0.64 [−0.75, 2.04], tie | 1.23 [0.16, 2.30] |
| vs model chosen before generation, server (same kind of decision as IntroLM) | 0.75 [−0.28, 1.78], tie | 0.28 [−0.09, 0.66], tie | 2.02 [1.13, 2.95] |
HARISSA's cost is significantly lower in five of the nine comparisons and tied in the other four. The policies in the first and third rows deliver the answer without the answer probe's check, so where HARISSA's cost is lower in those rows, the saving comes from the prefill check. We will upload the revised paper with these additional adapted baselines.
Moreover, in the server setting, a fixed 4B model outperforms both baselines at their selected operating points across all three benchmarks, further weakening the empirical comparisons.
The paper started FrugalGPT and Self-REF at the 2B size, which nearly half of MedQA questions correctly. For this response we reran both starting at 4B. On MedQA and MedMCQA, both then have the accuracy and latency of 4B alone, so the table compares HARISSA with 4B alone directly, with the cost defined as above.
| Cost saved by HARISSA (95% CI) | MedQA | MedMCQA | BBH |
|---|---|---|---|
| vs 4B alone | −3.38 [−5.35, −1.63], 4B better | −1.89 [−3.09, −0.71], 4B better | 1.88 [0.07, 3.73] |
4B alone has the lower cost on the two medical tasks, and HARISSA has the lower cost on BBH. In the rerun, HARISSA is also more than 5 accuracy points ahead of both baselines on BBH, where each baseline is taken at its most accurate setting that is no slower than HARISSA on the development set. The revision will add 4B alone, and both baselines started at 4B, to the server results.
**The evaluation covers only three multiple-choice benchmarks, two of which are medical QA tasks, without assessing open-ended generation or real-world applications. **
HARISSA does not depend on multiple choice or on a single model family. The probes are trained on labels of whether an answer was right or wrong, so any task with a grader (exact match, unit tests, a judge model) can supply them, and we chose multiple-choice benchmarks because their answers give unambiguous labels. The three benchmarks cover medicine in two exams and general reasoning in the sixteen BBH subtasks. We are also running open-ended generation versions of the benchmarks and will report the results in this thread shortly.
The server experiments are restricted to Qwen 3.5 models ranging from 2B to 27B, lacking evaluations with larger or heterogeneous model configurations.
We explicitly focus on the local setting which is why we do not run large model settings.
Additionally, some results rely on a single random seed, with insufficient statistical significance and cross-domain robustness analyses.
We agree that the paper's main results came from one seed and were reported without intervals. For this response we computed intervals for the paper's comparisons, trained a second seed, and tested transfer between the two medical datasets. On the device, HARISSA's cost is lower than or tied with every baseline's.
The table gives the cost that HARISSA saves against each device baseline, with the cost and the intervals defined as in the first table.
| Cost saved by HARISSA (95% CI) | MedQA | MedMCQA | BBH |
|---|---|---|---|
| vs always direct | 3.93 [2.15, 5.72] | 1.09 [−0.48, 2.66], tie | 5.42 [3.92, 7.01] |
| vs always reasoning | 3.67 [2.50, 4.83] | 1.97 [0.98, 2.97] | 0.96 [−0.46, 2.35], tie |
| vs self-consistency (3 direct samples) | 4.72 [2.93, 6.58] | 1.62 [−0.01, 3.22], tie | 3.52 [1.81, 5.27] |
| vs FrugalGPT | 3.42 [2.27, 4.56] | 0.97 [−0.58, 2.56], tie | 4.83 [3.39, 6.34] |
| vs Self-REF | 2.58 [1.38, 3.84] | 1.34 [0.09, 2.60] | 3.21 [2.01, 4.51] |
HARISSA's cost is significantly lower in 11 of the 15 comparisons and tied in the other 4.
We also computed intervals for the paper's comparison of the answer probe with three other confidence signals for choosing which answers to hand to a person: sequence likelihood, the margin between the top two answer options, and Self-REF's confidence token. When 10% of queries are handed to a person, the answer probe leaves significantly fewer wrong answers than the other signal in every comparison on the device and in five of nine on the server, where the other four are ties.
For the second seed, we retrained the MedQA device model. The policy answers 86% of MedQA queries correctly with either seed, the two costs are not significantly different, and the answer probe's AUROC changes by about 0.01.
For transfer, we applied the MedQA model, probes and thresholds to MedMCQA with nothing refit. The policy answers 68% of MedMCQA queries correctly, as does the policy fit on MedMCQA, and the answer probe's AUROC stays within 0.01 of the probe fit on MedMCQA.
Finally, we correct one statement in the paper. It says latency is computed from each query's token counts, but our code used the mean length of each way of answering on the task. Recomputed from each query's own length (estimated from the response's character count), HARISSA is about 2.1 times faster than always reasoning on average over the three tasks, where the paper reported 2.7 times. Wall-clock timing of test queries per task on an RTX A6000, run for this response, gives the same average. The tables in this response use each query's own length.
The paper emphasizes safe deployment but evaluates reliability primarily through answer correctness estimation and selective deferral, without addressing broader safety concerns such as harmful content generation, privacy leakage, or adversarial robustness. Consequently, the use of "Safe" in the title may overstate the actual scope of the contributions.
The title uses "safe" in the one sense the abstract defines: a deployment must decide which answers are safe to deliver, because delivering a wrong answer is worse than handing the query to a person. The paper evaluates this by the wrong answers that remain at a given share of queries handed off. Harmful content and adversarial robustness are different problems, and the paper does not claim to address them. Privacy is handled by the local-only setting, which never sends a query off the machine.
Please tell us which further evaluation would be most useful to addressing your concerns.
Thank you for the detailed rebuttal and additional experiments. I appreciate the improved baseline comparisons and statistical analyses. However, my main concerns remain unresolved:
-
W1. Limited Methodological Novelty. The authors' explanation primarily highlights technical differences rather than substantive methodological innovation. Jointly optimizing two established correctness prediction tasks still appears to be an integration of existing mechanisms. More importantly, the rebuttal does not directly establish the necessity of joint training. The original paper also reports a prefill skip threshold of zero across all three device benchmarks, indicating that the prefill probe does not contribute to action skipping in this setting.
-
W2. Mismatch Between Motivation and Methodology. Although the paper emphasizes the unique challenges of local-only deployment, HARISSA remains a generic adaptive inference framework without specific designs addressing deployment constraints, such as limited computational resources or restricted information sharing. Consequently, the connection between the stated motivation and the proposed methodology remains insufficiently justified.
-
W3. Baseline Competitiveness and Attribution. I acknowledge the additional baseline comparisons. However, the new results show that a simple fixed 4B model achieves significantly lower cost than HARISSA on both medical benchmarks, raising concerns about the practical benefits of the proposed adaptive framework. Furthermore, the authors attribute certain improvements to the prefill check, although the corresponding baselines already use the prefill probe. Since HARISSA additionally employs the answer check, this attribution is not adequately supported.
-
W4. Limited Experimental Generalization. Although the authors promise additional open-ended generation experiments, these results have not yet been provided. Thus, the concern regarding generalization beyond multiple-choice benchmarks remains unresolved.
Overall Assessment: The rebuttal strengthens the empirical evaluation, particularly through additional comparisons and statistical analyses. However, it does not sufficiently resolve the central concerns regarding methodological novelty, motivation–methodology alignment, baseline competitiveness, and generalization. Therefore, I maintain my original rating of rejection.
Thank you for the review. We answer each point in the order of the review.
We agree that the techniques you list (hidden-state correctness estimation, LoRA fine-tuning, linear probes and confidence-based cascading) are established. They are also active areas of research, and methods in this area, including those named in your review, contribute through what they build on them.
HARISSA introduces a new fine-tuning objective and a new policy that uses it. The objective trains a model so that its hidden state at the last prompt token predicts whether its answer will be correct, and its hidden state at the last answer token predicts whether the generated answer is correct. No tokens are added to the prompt or the answer. On the device model, a linear probe on each state predicts correctness better after this fine-tuning than before it. The policy uses both predictions to decide how much computation a query receives and whether its answer is delivered or handed to a person. It is built for a deployment whose largest model is still local and may fail.
Methods in your review relate to established techniques in the same way: they either combine them or add a training objective. FrugalGPT scores the answers of a model cascade with a separate DistilBERT scorer, and Cascade Routing (ICML 2025) unifies routing and cascading in one decision rule. Self-REF (ICML 2025) fine-tunes with LoRA to generate a confidence token after the answer, and IntroLM (Findings of ACL 2026) trains a LoRA adapter on introspection tokens added to the prompt. Each of these four answers a hard query by calling another model, and Self-REF also studies abstaining when no larger model is available . ThinkSwitcher and DiffAdapt choose before generation, from a model that is not fine-tuned, whether or how the model reasons, and deliver whatever that produces. HARISSA trains the estimate before generation and the estimate after it in one model, so its policy can have the same model reason more, check the answer, and hand the query to a person. In short, we use existing techniques but introduce a new training objective and policy and show it's benefits for our setting.
HARISSA is an adaptive inference framework targeting a local-only deployment. The action space is defined by the specific deployment: the device only runs a single SLM while the server can have a cascade of local models. Can you clarify why this is considered a weakness?