acceptodds
Under review as a conference paper at ICLR 2027

HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment

Abstract

Running a language model locally offers advantages in privacy, latency, and cost, but local hardware fits only small models, which are less capable than frontier models. The usual remedy for a hard query, escalating it to a cloud model, gives up the privacy and cost advantages of running locally. A deployment that stays local faces two decisions for hard queries instead. First, it can spend more computation on a query, e.g., reasoning before answering, which raises accuracy at a cost in latency, so it must decide which queries are worth the extra computation (efficiency). Second, some queries are beyond the local model, and delivering a wrong answer is worse than deferring the query to a human in the loop, so it must decide which answers are safe to deliver (safety). We show that both decisions can be made from the model's own hidden states. The prefill state, computed before any token is generated, predicts whether the model will answer correctly, and the answer state, at the end of the generated answer, predicts whether that answer is correct. HARISSA fine-tunes the model so that both states predict correctness, then makes both decisions with one policy that cascades through the ways of answering from cheapest to most expensive, skipping a way the prefill state predicts will fail and deferring the query when the answer it stops with is predicted wrong. On a device running a single model, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency. On a server holding four sizes of one model, HARISSA is more accurate than the FrugalGPT and Self-REF cascades at the same latency, and at the same deferral rate the answer state leaves fewer wrong answers than the standard confidence signals in five of six task and setting pairs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Local language models, LLM cascades, hidden-state probing, selective prediction, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2xhKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes HARISSA, a framework for efficient and reliable local language model inference. It leverages hidden-state-based correctness estimation at two stages: a prefill probe predicts whether an inference action will succeed before generation, while an answer probe evaluates correctness after generation. These signals guide a confidence-based cascade that adaptively allocates computation and defers unreliable answers to humans. Experiments on three multiple-choice benchmarks evaluate two settings: adaptive reasoning with a single on-device model and cascading across multiple model sizes on a local server. Results demonstrate improved accuracy–latency trade-offs over FrugalGPT and Self-REF, along with more effective identification of incorrect answers for selective deferral.

Critical strengths
  1. Strong Accuracy–Efficiency Trade-off. HARISSA achieves competitive accuracy with reduced inference latency. In the device setting, it approaches CoT accuracy with 2.7× lower latency.

  2. Effective Integration of Self-Assessment Mechanisms. The framework combines pre-generation and post-generation correctness estimation to support adaptive inference and selective deferral.

Critical weaknesses
  1. Model routing and model cascading have been extensively studied in prior work, including FrugalGPT, RouteLLM, Hybrid LLM, ThinkSwitcher, and DiffAdapt. HARISSA primarily integrates established techniques, such as hidden-state correctness estimation, LoRA fine-tuning, linear probes, and confidence-based cascading, without introducing substantial algorithmic or technical novelty.
  2. Although the paper emphasizes the unique challenges of local-only deployment, HARISSA is essentially a generic adaptive inference framework. The device and server settings mainly differ in their action spaces, without deployment-specific designs addressing resource limitations, privacy requirements, or hardware constraints.
  3. The experiments primarily compare against FrugalGPT and Self-REF, omitting several relevant methods for model routing and cascading (e.g., Cascade Routing [1], RLCascadeRouter [2]), adaptive reasoning (e.g., ThinkSwitcher [3], Sonata [4], DiffAdapt [5], and RPO [6]), and hidden-state-based correctness estimation (e.g., IntroLM [7]). Moreover, in the server setting, a fixed 4B model outperforms both baselines at their selected operating points across all three benchmarks, further weakening the empirical comparisons.
  4. The evaluation covers only three multiple-choice benchmarks, two of which are medical QA tasks, without assessing open-ended generation or real-world applications. The server experiments are restricted to Qwen 3.5 models ranging from 2B to 27B, lacking evaluations with larger or heterogeneous model configurations. Additionally, some results rely on a single random seed, with insufficient statistical significance and cross-domain robustness analyses.
  5. The paper emphasizes safe deployment but evaluates reliability primarily through answer correctness estimation and selective deferral, without addressing broader safety concerns such as harmful content generation, privacy leakage, or adversarial robustness. Consequently, the use of "Safe" in the title may overstate the actual scope of the contributions.

[1] A Unified Approach to Routing and Cascading for LLMs [2] RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning [3] ThinkSwitcher: When to Think Hard, When to Think Fast [4] AdaptThink: Reasoning Models Can Learn When to Think [5] DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference [6] Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking [7] IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation

Questions for authors

See weaknesses.

4 comments

Thank you for the review. We answer each point in the order of the review.

Model routing and model cascading have been extensively studied in prior work, including FrugalGPT, RouteLLM, Hybrid LLM, ThinkSwitcher, and DiffAdapt. HARISSA primarily integrates established techniques, such as hidden-state correctness estimation, LoRA fine-tuning, linear probes, and confidence-based cascading, without introducing substantial algorithmic or technical novelty.

We agree that the techniques you list (hidden-state correctness estimation, LoRA fine-tuning, linear probes and confidence-based cascading) are established. They are also active areas of research, and methods in this area, including those named in your review, contribute through what they build on them.

HARISSA introduces a new fine-tuning objective and a new policy that uses it. The objective trains a model so that its hidden state at the last prompt token predicts whether its answer will be correct, and its hidden state at the last answer token predicts whether the generated answer is correct. No tokens are added to the prompt or the answer. On the device model, a linear probe on each state predicts correctness better after this fine-tuning than before it. The policy uses both predictions to decide how much computation a query receives and whether its answer is delivered or handed to a person. It is built for a deployment whose largest model is still local and may fail.

Methods in your review relate to established techniques in the same way: they either combine them or add a training objective. FrugalGPT scores the answers of a model cascade with a separate DistilBERT scorer, and Cascade Routing (ICML 2025) unifies routing and cascading in one decision rule. Self-REF (ICML 2025) fine-tunes with LoRA to generate a confidence token after the answer, and IntroLM (Findings of ACL 2026) trains a LoRA adapter on introspection tokens added to the prompt. Each of these four answers a hard query by calling another model, and Self-REF also studies abstaining when no larger model is available . ThinkSwitcher and DiffAdapt choose before generation, from a model that is not fine-tuned, whether or how the model reasons, and deliver whatever that produces. HARISSA trains the estimate before generation and the estimate after it in one model, so its policy can have the same model reason more, check the answer, and hand the query to a person. In short, we use existing techniques but introduce a new training objective and policy and show it's benefits for our setting.

Although the paper emphasizes the unique challenges of local-only deployment, HARISSA is essentially a generic adaptive inference framework. The device and server settings mainly differ in their action spaces, without deployment-specific designs addressing resource limitations, privacy requirements, or hardware constraints.

HARISSA is an adaptive inference framework targeting a local-only deployment. The action space is defined by the specific deployment: the device only runs a single SLM while the server can have a cascade of local models. Can you clarify why this is considered a weakness?

The experiments primarily compare against FrugalGPT and Self-REF, omitting several relevant methods for model routing and cascading (e.g., Cascade Routing [1], RLCascadeRouter [2]), adaptive reasoning (e.g., ThinkSwitcher [3], Sonata [4], DiffAdapt [5], and RPO [6]), and hidden-state-based correctness estimation (e.g., IntroLM [7]).

We ran an experiment for each group of methods in your list, adapting them to our setting. First, the adaptive reasoning methods (ThinkSwitcher, Sonata, DiffAdapt, RPO, and AdaptThink) decide before the answer exists whether or how the model reasons. On the device, we compared HARISSA with a policy that decides from the prefill probe alone whether the model reasons, and delivers the answer without checking it. Second, Cascade Routing and RLCascadeRouter decide at each step which model to run next, or to stop. On the server, we ran a decision rule of Cascade Routing's kind over the four model sizes, with our two probes as its quality estimates. Third, IntroLM decides before generation which of two models answers. On the server, we compared HARISSA with a policy that chooses among the four model sizes from the prefill probe alone. The table reports the cost that HARISSA saves against each policy (the other policy's cost minus HARISSA's), where a policy's cost is its wrong answers per 100 queries plus its mean latency and with paired 95% bootstrap intervals over test queries.

Cost saved by HARISSA (95% CI)MedQAMedMCQABBH
vs reasoning decided before the answer, device (same kind of decision as ThinkSwitcher, Sonata, AdaptThink, DiffAdapt, RPO)3.68 [2.51, 4.82]1.09 [−0.47, 2.67], tie2.66 [1.30, 4.04]
vs Cascade Routing rule, server (same kind of decision as Cascade Routing, RLCascadeRouter)1.30 [0.13, 2.45]0.64 [−0.75, 2.04], tie1.23 [0.16, 2.30]
vs model chosen before generation, server (same kind of decision as IntroLM)0.75 [−0.28, 1.78], tie0.28 [−0.09, 0.66], tie2.02 [1.13, 2.95]

HARISSA's cost is significantly lower in five of the nine comparisons and tied in the other four. The policies in the first and third rows deliver the answer without the answer probe's check, so where HARISSA's cost is lower in those rows, the saving comes from the prefill check. We will upload the revised paper with these additional adapted baselines.

Moreover, in the server setting, a fixed 4B model outperforms both baselines at their selected operating points across all three benchmarks, further weakening the empirical comparisons.

The paper started FrugalGPT and Self-REF at the 2B size, which nearly half of MedQA questions correctly. For this response we reran both starting at 4B. On MedQA and MedMCQA, both then have the accuracy and latency of 4B alone, so the table compares HARISSA with 4B alone directly, with the cost defined as above.

Cost saved by HARISSA (95% CI)MedQAMedMCQABBH
vs 4B alone−3.38 [−5.35, −1.63], 4B better−1.89 [−3.09, −0.71], 4B better1.88 [0.07, 3.73]

4B alone has the lower cost on the two medical tasks, and HARISSA has the lower cost on BBH. In the rerun, HARISSA is also more than 5 accuracy points ahead of both baselines on BBH, where each baseline is taken at its most accurate setting that is no slower than HARISSA on the development set. The revision will add 4B alone, and both baselines started at 4B, to the server results.

**The evaluation covers only three multiple-choice benchmarks, two of which are medical QA tasks, without assessing open-ended generation or real-world applications. **

HARISSA does not depend on multiple choice or on a single model family. The probes are trained on labels of whether an answer was right or wrong, so any task with a grader (exact match, unit tests, a judge model) can supply them, and we chose multiple-choice benchmarks because their answers give unambiguous labels. The three benchmarks cover medicine in two exams and general reasoning in the sixteen BBH subtasks. We are also running open-ended generation versions of the benchmarks and will report the results in this thread shortly.

The server experiments are restricted to Qwen 3.5 models ranging from 2B to 27B, lacking evaluations with larger or heterogeneous model configurations.

We explicitly focus on the local setting which is why we do not run large model settings.

Additionally, some results rely on a single random seed, with insufficient statistical significance and cross-domain robustness analyses.

We agree that the paper's main results came from one seed and were reported without intervals. For this response we computed intervals for the paper's comparisons, trained a second seed, and tested transfer between the two medical datasets. On the device, HARISSA's cost is lower than or tied with every baseline's.

The table gives the cost that HARISSA saves against each device baseline, with the cost and the intervals defined as in the first table.

Cost saved by HARISSA (95% CI)MedQAMedMCQABBH
vs always direct3.93 [2.15, 5.72]1.09 [−0.48, 2.66], tie5.42 [3.92, 7.01]
vs always reasoning3.67 [2.50, 4.83]1.97 [0.98, 2.97]0.96 [−0.46, 2.35], tie
vs self-consistency (3 direct samples)4.72 [2.93, 6.58]1.62 [−0.01, 3.22], tie3.52 [1.81, 5.27]
vs FrugalGPT3.42 [2.27, 4.56]0.97 [−0.58, 2.56], tie4.83 [3.39, 6.34]
vs Self-REF2.58 [1.38, 3.84]1.34 [0.09, 2.60]3.21 [2.01, 4.51]

HARISSA's cost is significantly lower in 11 of the 15 comparisons and tied in the other 4.

We also computed intervals for the paper's comparison of the answer probe with three other confidence signals for choosing which answers to hand to a person: sequence likelihood, the margin between the top two answer options, and Self-REF's confidence token. When 10% of queries are handed to a person, the answer probe leaves significantly fewer wrong answers than the other signal in every comparison on the device and in five of nine on the server, where the other four are ties.

For the second seed, we retrained the MedQA device model. The policy answers 86% of MedQA queries correctly with either seed, the two costs are not significantly different, and the answer probe's AUROC changes by about 0.01.

For transfer, we applied the MedQA model, probes and thresholds to MedMCQA with nothing refit. The policy answers 68% of MedMCQA queries correctly, as does the policy fit on MedMCQA, and the answer probe's AUROC stays within 0.01 of the probe fit on MedMCQA.

Finally, we correct one statement in the paper. It says latency is computed from each query's token counts, but our code used the mean length of each way of answering on the task. Recomputed from each query's own length (estimated from the response's character count), HARISSA is about 2.1 times faster than always reasoning on average over the three tasks, where the paper reported 2.7 times. Wall-clock timing of test queries per task on an RTX A6000, run for this response, gives the same average. The tables in this response use each query's own length.

The paper emphasizes safe deployment but evaluates reliability primarily through answer correctness estimation and selective deferral, without addressing broader safety concerns such as harmful content generation, privacy leakage, or adversarial robustness. Consequently, the use of "Safe" in the title may overstate the actual scope of the contributions.

The title uses "safe" in the one sense the abstract defines: a deployment must decide which answers are safe to deliver, because delivering a wrong answer is worse than handing the query to a person. The paper evaluates this by the wrong answers that remain at a given share of queries handed off. Harmful content and adversarial robustness are different problems, and the paper does not claim to address them. Privacy is handled by the local-only setting, which never sends a query off the machine.

Please tell us which further evaluation would be most useful to addressing your concerns.

Thank you for the detailed rebuttal and additional experiments. I appreciate the improved baseline comparisons and statistical analyses. However, my main concerns remain unresolved:

  • W1. Limited Methodological Novelty. The authors' explanation primarily highlights technical differences rather than substantive methodological innovation. Jointly optimizing two established correctness prediction tasks still appears to be an integration of existing mechanisms. More importantly, the rebuttal does not directly establish the necessity of joint training. The original paper also reports a prefill skip threshold of zero across all three device benchmarks, indicating that the prefill probe does not contribute to action skipping in this setting.

  • W2. Mismatch Between Motivation and Methodology. Although the paper emphasizes the unique challenges of local-only deployment, HARISSA remains a generic adaptive inference framework without specific designs addressing deployment constraints, such as limited computational resources or restricted information sharing. Consequently, the connection between the stated motivation and the proposed methodology remains insufficiently justified.

  • W3. Baseline Competitiveness and Attribution. I acknowledge the additional baseline comparisons. However, the new results show that a simple fixed 4B model achieves significantly lower cost than HARISSA on both medical benchmarks, raising concerns about the practical benefits of the proposed adaptive framework. Furthermore, the authors attribute certain improvements to the prefill check, although the corresponding baselines already use the prefill probe. Since HARISSA additionally employs the answer check, this attribution is not adequately supported.

  • W4. Limited Experimental Generalization. Although the authors promise additional open-ended generation experiments, these results have not yet been provided. Thus, the concern regarding generalization beyond multiple-choice benchmarks remains unresolved.

Overall Assessment: The rebuttal strengthens the empirical evaluation, particularly through additional comparisons and statistical analyses. However, it does not sufficiently resolve the central concerns regarding methodological novelty, motivation–methodology alignment, baseline competitiveness, and generalization. Therefore, I maintain my original rating of rejection.

Reviewer_6PKcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HARISSA addresses two decisions in local language model deployment: how much computation to use and whether to deliver an answer or defer the query to a person. The method jointly trains correctness signals at the prefill and answer hidden states. A threshold-based policy then uses these signals to skip actions, continue to a more expensive action, or withhold an answer.

The paper evaluates direct versus thinking modes for one model and cascades across four model sizes. Results on MedQA, MedMCQA, and BBH show promising accuracy–latency tradeoffs. However, the experiments do not fully isolate the contribution of the combined design. Several protocol details also require clarification.

Critical strengths

S1. HARISSA addresses a practical challenge: allocating computation and deciding when to defer in local deployments. The prefill and answer probes provide a coherent framework for decisions before and after generation.

S2. On MedQA, the device policy achieves 85.7% accuracy at 33.0 seconds, compared with 86.2% at 81.0 seconds for always thinking (Table 1). Adaptation also improves the direct-answer probe’s AUROC from 0.716 to 0.840 (Table 11).

S3. The paper examines probe removal, frozen versus adapted models, and action frequencies. These analyses help explain the performance differences, while additional model families and reported negative results provide a more balanced evaluation.

Critical weaknesses

W1. Correctness probing and adaptive cascading have substantial prior work, so the novelty mainly lies in their integration. However, the selected device policy uses a zero skip threshold on all three tasks, leaving the prefill decision inactive. This does not exclude a training benefit, but a control trained with only the answer-probe objective is needed to establish the value of joint training.

W2. The fixed 4B model is both more accurate and faster than the reported FrugalGPT and Self-REF operating points on all three tasks (Tables 1 and 6). This suggests that some gains may reflect the baselines’ fixed 2B-first configuration. Comparisons with 4B-first cascades and stage configurations selected on the development set would make the evidence more convincing.

W3. Despite the zero device skip threshold, the full policy and no-prefill ablation produce different results (Tables 6 and 8). The authors should explain whether checkpoints, samples, or tuning procedures differ. The hardware description also needs clarification: 27B parameters in BF16 require approximately 54 GB for weights alone, exceeding the stated 48 GB A40. End-to-end cascade measurements would strengthen the latency claims.

W4. The deferral cost μ\mu is unspecified, and deferral is disabled in most experiments. Tying the deferral and fall-through thresholds also restricts human deferral to the final stage. Separately, the main accuracy comparisons lack confidence intervals, and evaluation covers only in-distribution multiple-choice tasks. These limitations weaken the evidence for the joint objective and broader safety claims.

Questions for authors
  1. What causes the difference between full device HARISSA at a zero skip threshold and the ablation without prefill skipping?
  2. How much of the server gain remains when baseline cascades start at 4B?
  3. Does joint supervision improve over answer-only training under matched data and computation?
4 comments

Thank you for engaging with the paper and offering a thoughtful review. We address your concerns below.

W1. Correctness probing and adaptive cascading have substantial prior work, so the novelty mainly lies in their integration.

HARISSA contributes a new fine-tuning objective and a policy that uses it. Correctness probing and adaptive cascading are established and active areas of research, and new methods contribute by what they build with these techniques. HARISSA's objective trains one model so that its hidden states predict, before generation, whether its answer will be correct and, after generation, whether the generated answer is correct, with no tokens added to the prompt or the answer. The policy uses both predictions to decide how much computation a query receives and whether its answer is delivered or handed to a person, in a deployment whose largest model is still local and may fail.

However, the selected device policy uses a zero skip threshold on all three tasks, leaving the prefill decision inactive.

The prefill decision is active on the server on all three tasks, and the skip threshold selected on the development set disables the prefill decision only on the device. On the server, skipping a model size saves a full generation, whereas on the device, skipping the direct answer saves about an eighth of the time of the reasoned answer that follows. The selected device policy therefore generates every direct answer and lets the answer probe decide whether to reason. We do not view this as a weakness because the policy finds what works for each setting regardless of which hidden state signal it relies on.

This does not exclude a training benefit, but a control trained with only the answer-probe objective is needed to establish the value of joint training.

Does joint supervision improve over answer-only training under matched data and computation?

Joint supervision improves the prefill probe at no measurable cost to the answer probe or to the device policy. For this response we trained the control on the MedQA device model, using the paper's recipe with the same data and the same seed and with only the prefill loss removed, and we trained a second seed of the joint model.

MedQA, devicePrefill AUROCGain over control (95% CI)Answer AUROCAccuracy
Joint, paper's seed0.690.05 [0.01, 0.10]0.8286%
Joint, second seed0.700.06 [0.02, 0.11]0.8386%
Control, answer loss only0.64reference0.8286%

Joint training significantly raises the prefill probe's AUROC with either seed. The answer probe's AUROC does not differ significantly between the control and either joint model, and the device policy answers 86% of MedQA queries correctly with all three models.

W4. The deferral cost (\mu) is unspecified, and deferral is disabled in most experiments.

No result in the paper depends on a value of the deferral cost. The accuracy and latency comparisons turn deferral off so that every policy answers every query, and the deferral comparisons give every confidence signal the same share of queries to hand to a person. Otherwise measuring accuracy and latency is no longer a 1:1 comparison with the other methods that cannot defer.

Tying the deferral and fall-through thresholds also restricts human deferral to the final stage.

HARISSA's model and probes support a hand-off from any stage. With the two thresholds tied as in the paper, an answer the answer probe does not trust first receives more computation and reaches a person only when the last answer is also not trusted. For this response we also tested a hand-off from any stage, with a deferral cost of 0.25, so that a wrong answer costs as much as four hand-offs.

Total cost (lower is better)MedQAMedMCQABBH
Device, tied thresholds15.226.79.5
Device, any stage15.224.18.4
Server, tied thresholds41.653.922.2
Server, any stage36.234.722.2

Counting wrong answers, latency and hand-offs, HARISSA's total cost with a hand-off from any stage is equal to or lower than its total cost under the paper's rule on all three tasks, on the device and on the server. The revision will report both rules.

W3. Despite the zero device skip threshold, the full policy and no-prefill ablation produce different results (Tables 6 and 8). The authors should explain whether checkpoints, samples, or tuning procedures differ.

What causes the difference between full device HARISSA at a zero skip threshold and the ablation without prefill skipping?

The two rows are the same policy, run on the same checkpoint and the same generated answers, and the reported rows differ because the ablation's fall-through threshold was selected from a different grid than the main policy's. For this response we compared the decisions of the two rows query by query, and at the same fall-through threshold they make the same decision on every query on all three tasks. We also selected the ablation's threshold from the main policy's grid.

Accuracy at mean latencyMedQAMedMCQABBH
HARISSA (paper)85.7% at 33.0 s68.5% at 38.4 s88.7% at 11.7 s
No prefill skip (paper)86.6% at 39.7 s67.5% at 26.5 s88.6% at 11.6 s
No prefill skip, main grid (new)85.8% at 33.2 s68.5% at 38.3 s88.7% at 11.7 s

With its threshold selected from the main policy's grid, the ablation matches the main row to within 0.1 accuracy points and 0.2 s on all three tasks. The revision will select both rows from that grid.

The hardware description also needs clarification: 27B parameters in BF16 require approximately 54 GB for weights alone, exceeding the stated 48 GB A40.

Yes this requires a correction. The 27B size ran on two GPUs. We generated its answers on two RTX A6000 cards and measured its prefill and per-token times with the model split across two A40 cards. We measured the 2B, 4B and 9B sizes on one A40 each, as the paper states, and the revision will correct the description of the 27B size.

End-to-end cascade measurements would strengthen the latency claims.

For this response we measured latency on the device by wall clock, and HARISSA is about 2.1 times faster than always reasoning (chain-of-thought in the paper) on average over the three tasks. We timed the device model's direct answers and its reasoned answers on 300 test queries per task on an RTX A6000 and computed each policy's latency on each query from these measured times. We also ran the full device cascade end to end on 100 MedQA queries, where its mean time per query matched the sum of its separately timed steps to within 0.01 s.

Measured speed-up (95% CI)MedQAMedMCQABBH
HARISSA vs always reasoning2.06 [1.82, 2.38]1.48 [1.35, 1.65]2.69 [2.29, 3.26]

HARISSA is significantly faster than always reasoning on every task.

W2. The fixed 4B model is both more accurate and faster than the reported FrugalGPT and Self-REF operating points on all three tasks (Tables 1 and 6). This suggests that some gains may reflect the baselines' fixed 2B-first configuration. Comparisons with 4B-first cascades and stage configurations selected on the development set would make the evidence more convincing.

How much of the server gain remains when baseline cascades start at 4B?

The paper started FrugalGPT and Self-REF at the 2B size, which answers roughly half of MedQA questions correctly. For this response we reran both baselines starting at 4B, and again with their sizes selected on the development set. On MedQA and MedMCQA, each baseline at its selected thresholds has the accuracy and latency of 4B alone in both reruns, so the first table compares HARISSA with 4B alone directly. A policy's cost is its wrong answers per 100 queries plus its mean latency, with 0.5 s on the server counted as one wrong answer per 100 queries and each query's latency computed from its own answer length. The second table compares accuracy on BBH with each baseline started at 4B, taken at its most accurate setting that is no slower than HARISSA on the development set. Both tables give paired 95% bootstrap intervals over test queries.

Cost saved by HARISSA (95% CI)MedQAMedMCQABBH
vs 4B alone−3.38 [−5.35, −1.63], 4B better−1.89 [−3.09, −0.71], 4B better1.88 [0.07, 3.73]
Accuracy gained by HARISSA (95% CI)BBH
vs FrugalGPT started at 4B5.2 [3.6, 6.8]
vs Self-REF started at 4B5.4 [3.8, 7.2]

4B alone has the lower cost on the two medical tasks, and HARISSA has the lower cost on BBH, where it is also significantly more accurate than both baselines started at 4B. The revision will add 4B alone, and both baselines started at 4B, to the main server results.

Separately, the main accuracy comparisons lack confidence intervals

With intervals, HARISSA on the device is significantly more accurate than FrugalGPT or Self-REF in four of six comparisons and tied in the other two, and it is tied with always reasoning on the two medical tasks. For this response we computed paired 95% bootstrap intervals over test queries for the paper's device comparisons.

Accuracy gained by HARISSA (95% CI)MedQAMedMCQABBH
vs always reasoning−0.39 [−1.57, 0.79], tie−0.10 [−1.05, 0.86], tie−1.88 [−3.31, −0.55], always reasoning better
vs FrugalGPT0.16 [−0.94, 1.26], tie4.64 [3.06, 6.21]5.52 [3.97, 7.06]
vs Self-REF1.26 [0.00, 2.51], tie1.77 [0.53, 3.01]3.09 [1.88, 4.42]

and evaluation covers only in-distribution multiple-choice tasks.

HARISSA does not depend on multiple choice, and its device policy carries over from MedQA to MedMCQA with nothing refit. The probes are trained on labels of whether an answer was right or wrong, which any graded task can supply. For this response we applied the MedQA device model, probes and thresholds to MedMCQA unchanged.

MedMCQAAccuracyAnswer AUROC
Policy fit on MedMCQA68.5%0.77
MedQA policy, unchanged (new)68.4%0.77

The MedQA policy is within 0.1 accuracy points of the policy fit on MedMCQA, and its answer probe's AUROC is within 0.01. We are also running open-ended generation versions of the benchmarks and will report the results in this thread shortly.

These limitations weaken the evidence for the joint objective and broader safety claims.

The paper's safety claim is the one its abstract defines, deciding which answers are safe to deliver, and the paper evaluates it by the wrong answers that remain when a given share of queries is handed to a person. For this response we computed intervals for the paper's comparison of the answer probe with three other confidence signals, which are sequence likelihood, the margin between the top two answer options, and Self-REF's confidence token. When 10% of queries are handed to a person, the answer probe leaves significantly fewer wrong answers than the other signal in every comparison on the device and in five of nine on the server, where the other four are ties. We can will narrow our definition of safety or use a different term for clarity's sake in the revision.

Thank you for your response and the additional experiments. After considering these additions, I still find the conceptual contribution beyond prior work insufficient to warrant a higher rating. I therefore stand by my original assessment.

Reviewer_63RjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HARISSA targets a local-only LLM deployment — laptop, phone, or on-premises server — where escalating a hard query to a cloud model is ruled out by privacy or budget constraints. Two decisions follow per query: how much local computation to spend, and whether the answer is safe to deliver or should go to a person. The claim is that both can be read from the model's own hidden states — the prefill state, available before generation, predicts whether a way of answering will succeed, and the answer state predicts whether the answer just produced is correct. The model is LoRA fine-tuned so both states carry this signal, calibrated probes are fit on each, and one policy cascades cheapest-first: skipping an action the prefill probe predicts will fail, stopping at the first answer the answer probe trusts, deferring to a person otherwise. Evaluated on MedQA, MedMCQA and BBH, on a device running Qwen 3.5 4B with {direct, think} and a server holding four Qwen 3.5 sizes.

Critical strengths

1. The direction and the underlying insight are interesting and timely. Using a model's own internal states to decide, before it generates anything, whether a way of answering is worth running is an appealing idea, and the local-only framing — no stronger model to escalate to, a human as the fallback — is a real constraint the routing and cascading literature mostly assumes away. §6.1 is the most informative part: the answer state separates correct from incorrect answers far better than the prefill state (0.83 vs 0.63 AUROC on MedQA), and both beat a classifier fit on the question text alone — a concrete, reusable finding about where this signal lives.

2. The headline result is a strong one. 2.7× lower mean latency while staying within one accuracy point of always reasoning, averaged over three tasks, is a meaningful operating point for an interactive deployment — reached by thinking on 29% of MedQA queries where FrugalGPT thinks on 75%. The numbers are internally consistent, and the supporting material (per-task sweeps, ablations, two further model families, released graded outputs) is unusually complete.

Critical weaknesses

1. Latency is the paper's central axis and it is never measured. Appendix C concedes that every latency figure is an analytic model — per-size prefill plus tokens times a per-token rate — calibrated on eight prompts on one NVIDIA A40 at concurrency one, and that "the device numbers are those of a 4B model on that GPU rather than on a phone." The motivation is laptops and phones; no result comes from either. In our own throughput measurements, edge inference above roughly 4B parameters is slower than a strong frontier cloud model that is also more capable, so the trade being asked of a deployment is steeper than the modeled numbers suggest, and the paper never quantifies what the local-only constraint costs against the cloud baseline it rules out. Only mean end-to-end latency is reported, yet for the interactive scenarios the introduction invokes, time to first token and sustained throughput matter as much — and a cascade that generates a direct answer, discards it, then regenerates with reasoning has a time-to-first-token profile that mean latency conceals.

2. It is unclear whether the phenomenon is intrinsic or manufactured by the recipe, and the two-state claim is not exercised in the headline setting. If hidden states reveal whether a model knows an answer, that is a notable property of language models. If it must be installed by LoRA fine-tuning on ~10,000 graded queries per task plus per-task probe fitting, it is a far narrower engineering result — and Table 11 points to the latter for the prefill state, which reaches 0.594 AUROC on MedQA frozen and 0.639 fine-tuned, barely above chance. The signal is also strongly task-dependent (0.63 on MedQA against 0.81 on BBH). This surfaces in the policy: Table 8 gives a selected skip threshold of 0.00 on all three device tasks, so the prefill probe never fires there and the device configuration — the source of the 2.7× headline — reduces to the answer probe plus a cascade. §6.1 is honest about this, but the claim that both states drive the policy then holds only on the server.

Questions for authors
  1. Are there wall-clock measurements on a laptop or phone rather than token counts scaled by A40 rates, and can you report time to first token and throughput alongside mean latency — especially for the fall-through case where a direct answer is generated and discarded?
  2. How much prefill and answer signal exists before fine-tuning — is this capability intrinsic to the backbone or induced by your recipe? What is the smallest training set that reproduces the device results?
  3. Three numbers I could not reconcile: §6.1's answer-probe gains of 0.12/0.06/0.09 AUROC (averaging Table 11 over both actions gives 0.065/0.055/0.119); the BBH question-only AUROC quoted as 0.74 against Table 11's 0.76 mean; and Table 6's "without the prefill skip" differing from Table 1's HARISSA on device (86.6% @ 39.7 s vs 85.7% @ 33.0 s on MedQA) although Table 8 gives that skip threshold as 0.00.
  4. Were FrugalGPT's and Self-REF's thresholds selected under the same objective and latency price as HARISSA's? FrugalGPT delivering 98.7% direct answers on device MedMCQA suggests otherwise.
  5. How many seeds underlie Table 1, and can you give intervals? At n = 1273 the binomial standard error is about one accuracy point.
4 comments

Thank you for the review.

Latency is the paper's central axis and it is never measured. Appendix C concedes that every latency figure is an analytic model — per-size prefill plus tokens times a per-token rate — calibrated on eight prompts on one NVIDIA A40 at concurrency one, and that "the device numbers are those of a 4B model on that GPU rather than on a phone."

For this response we measured latency by wall clock, and on average over the three tasks HARISSA is about 2.1 times faster than always reasoning (chain-of-thought in the paper), where the paper reported 2.7 times. We timed the device model's direct answers and its reasoned answers on the test queries per task on an RTX A6000 and computed each policy's latency on each query from these measured times. We also recomputed the paper's modeled latency from each query's own answer length.

The motivation is laptops and phones; no result comes from either.

HARISSA saves latency by generating fewer reasoning tokens, and for a given model a query's token count does not depend on the hardware. All of our timings of the 4B device model, in the paper and for this response, are on a GPU. At the thresholds selected for the GPU's latencies, HARISSA reasons on 29% of MedQA queries, and a deployment on other hardware selects the thresholds for its own latencies.

In our own throughput measurements, edge inference above roughly 4B parameters is slower than a strong frontier cloud model that is also more capable, so the trade being asked of a deployment is steeper than the modeled numbers suggest, and the paper never quantifies what the local-only constraint costs against the cloud baseline it rules out.

The paper studies deployments for which a cloud model is not an option. Such deployments hold clinical records that European data-protection law restricts from transfer outside the European Union, or lack the legal review, IT staff and per-query budget that a cloud contract requires, etc.. A deployment of this kind chooses among local policies, so the paper compares the accuracy and latency of those policies.

Only mean end-to-end latency is reported, yet for the interactive scenarios the introduction invokes, time to first token and sustained throughput matter as much — and a cascade that generates a direct answer, discards it, then regenerates with reasoning has a time-to-first-token profile that mean latency conceals.

Are there wall-clock measurements on a laptop or phone rather than token counts scaled by A40 rates, and can you report time to first token and throughput alongside mean latency — especially for the fall-through case where a direct answer is generated and discarded?

HARISSA shows an answer once the answer probe has checked it, and in our wall-clock timing the median query on every task receives its answer in under 10 s. For this response we timed HARISSA on the test queries per task on an RTX A6000, where the device model generates its first token after about 0.14 s and then about 54 tokens per second.

Measured latencyMedQAMedMCQABBH
Mean (s)42.748.119.9
Median (s)7.39.95.3
Sent to reasoning29%46%15%
Their mean (s)129.697.496.9

A query sent to reasoning waits for the same reasoning as under always reasoning, plus a discarded direct answer that in the paper's results takes about an eighth of the time of a reasoned answer.

Speed-up over always reasoningMedQAMedMCQABBH
Modeled, mean lengths (paper)2.451.733.84
Modeled, each query's length (new)2.001.452.71
Measured, 95% CI (new)2.06 [1.82, 2.38]1.48 [1.35, 1.65]2.69 [2.29, 3.26]

The paper states that latency is computed from each query's token counts, but our code used the mean length of each way of answering on the task. The revision will report the corrected and the measured latencies.

It is unclear whether the phenomenon is intrinsic or manufactured by the recipe, and the two-state claim is not exercised in the headline setting. If hidden states reveal whether a model knows an answer, that is a notable property of language models. If it must be installed by LoRA fine-tuning on ~10,000 graded queries per task plus per-task probe fitting, it is a far narrower engineering result — and Table 11 points to the latter for the prefill state, which reaches 0.594 AUROC on MedQA frozen and 0.639 fine-tuned, barely above chance. The signal is also strongly task-dependent (0.63 on MedQA against 0.81 on BBH).

How much prefill and answer signal exists before fine-tuning — is this capability intrinsic to the backbone or induced by your recipe?

The signal is intrinsic, because the hidden states of the frozen model already predict correctness, and fine-tuning raises the probes' AUROC further. On the frozen device model, the paper reports an answer probe AUROC between 0.68 and 0.85 across the three tasks and both ways of answering. The frozen prefill probe's AUROC is above the AUROC of a classifier trained on the question text in all six cases, e.g., 0.59 against 0.54 on MedQA's direct answers. The prefill probe's AUROC is higher on tasks where more of a question's difficulty is visible in the question's text, and the question-text classifier alone reaches an average AUROC of 0.76 on BBH against 0.54 on MedQA.

What is the smallest training set that reproduces the device results?

With the fine-tuned model fixed, 250 graded training queries are enough to fit the probes. For this response we refit the probes on random subsets of 250 and of 1,000 training queries, five subsets of each size, and ran the device policy with each fit. A policy's cost is its wrong answers per 100 queries plus its mean latency, with 10 s counted as one wrong answer per 100 queries. The table gives the mean cost over the five subsets and the cost with probes fit on the full training set, all computed for this response.

Cost (lower is better)MedQAMedMCQABBH
Probes fit on 250 queries17.035.312.8
Probes fit on 1,000 queries17.135.312.2
Probes fit on the full set17.635.312.4

Fine-tuning also works with fewer than 10,000 queries, since on BBH fine-tuning uses 2,188 training queries and raises the answer probe's AUROC more than on either medical task.

This surfaces in the policy: Table 8 gives a selected skip threshold of 0.00 on all three device tasks, so the prefill probe never fires there and the device configuration — the source of the 2.7× headline — reduces to the answer probe plus a cascade. §6.1 is honest about this, but the claim that both states drive the policy then holds only on the server.

The prefill decision is active on the server on all three tasks, and the skip threshold selected on the development set disables the prefill decision only on the device. On the server, skipping a model size saves a full generation, whereas on the device, skipping the direct answer saves about an eighth of the time of the reasoned answer that follows. The selected device policy therefore generates every direct answer and lets the answer probe decide whether to reason. The answer probe is what separates HARISSA from FrugalGPT and Self-REF, which run the same two-step cascade on the device, and on MedQA HARISSA reaches FrugalGPT's accuracy while reasoning on 29% of queries where FrugalGPT reasons on 75%. We do not view this as a weakness because the policy finds what works for each setting regardless of which hidden state signal it relies on.

Three numbers I could not reconcile: §6.1's answer-probe gains of 0.12/0.06/0.09 AUROC (averaging Table 11 over both actions gives 0.065/0.055/0.119); the BBH question-only AUROC quoted as 0.74 against Table 11's 0.76 mean; and Table 6's "without the prefill skip" differing from Table 1's HARISSA on device (86.6% @ 39.7 s vs 85.7% @ 33.0 s on MedQA) although Table 8 gives that skip threshold as 0.00.

You are right that the first two numbers in the paper's text are wrong. The answer probe's gain is about 0.06 on the medical tasks and 0.12 on BBH, the question-text classifier's AUROC on BBH is 0.76, and the revision will report these values. The two rows in your third point are the same policy, run on the same checkpoint and the same generated answers, and the reported rows differ because the ablation's fall-through threshold was selected from a different grid than the main policy's. For this response we compared the decisions of the two rows query by query, and at the same fall-through threshold they make the same decision on every query on all three tasks. We also selected the ablation's threshold from the main policy's grid.

Accuracy at mean latencyMedQAMedMCQABBH
HARISSA (paper)85.7% at 33.0 s68.5% at 38.4 s88.7% at 11.7 s
No prefill skip (paper)86.6% at 39.7 s67.5% at 26.5 s88.6% at 11.6 s
No prefill skip, main grid (new)85.8% at 33.2 s68.5% at 38.3 s88.7% at 11.7 s

With its threshold selected from the main policy's grid, the ablation matches the main row to within 0.1 accuracy points and 0.2 s on all three tasks. The revision will select both rows from that grid.

Were FrugalGPT's and Self-REF's thresholds selected under the same objective and latency price as HARISSA's? FrugalGPT delivering 98.7% direct answers on device MedMCQA suggests otherwise.

Yes, the thresholds of both baselines were selected on the same development split as HARISSA's, under the same objective and the same latency price, and FrugalGPT's 98.7% of direct answers follows from that objective. In the paper's results on MedMCQA, reasoning adds 5 accuracy points for 59 s of latency. At the paper's price of 10 s of latency per accuracy point, the added latency outweighs the gain, so reasoning is worth the latency only on the queries whose direct answer is likely to be wrong. With the answer probe, HARISSA reasons on 46% of MedMCQA queries and gains 4.6 accuracy points over FrugalGPT.

How many seeds underlie Table 1, and can you give intervals? At n = 1273 the binomial standard error is about one accuracy point.

The paper's main results use one fine-tuning seed and one generation seed, and with intervals HARISSA's cost on the device is lower than or tied with every baseline's. For this response we computed paired 95% bootstrap intervals over test queries for the paper's device comparisons, for accuracy and for cost. A policy's cost is its wrong answers per 100 queries plus its mean latency, with 10 s counted as one wrong answer per 100 queries and each query's latency computed from its own answer length.

Accuracy gained by HARISSA (95% CI)MedQAMedMCQABBH
vs always reasoning−0.39 [−1.57, 0.79], tie−0.10 [−1.05, 0.86], tie−1.88 [−3.31, −0.55], always reasoning better
vs FrugalGPT0.16 [−0.94, 1.26], tie4.64 [3.06, 6.21]5.52 [3.97, 7.06]
vs Self-REF1.26 [0.00, 2.51], tie1.77 [0.53, 3.01]3.09 [1.88, 4.42]
Cost saved by HARISSA (95% CI)MedQAMedMCQABBH
vs always direct3.93 [2.15, 5.72]1.09 [−0.48, 2.66], tie5.42 [3.92, 7.01]
vs always reasoning3.67 [2.50, 4.83]1.97 [0.98, 2.97]0.96 [−0.46, 2.35], tie
vs self-consistency4.72 [2.93, 6.58]1.62 [−0.01, 3.22], tie3.52 [1.81, 5.27]
vs FrugalGPT3.42 [2.27, 4.56]0.97 [−0.58, 2.56], tie4.83 [3.39, 6.34]
vs Self-REF2.58 [1.38, 3.84]1.34 [0.09, 2.60]3.21 [2.01, 4.51]

Thank you for the clarification and additional experiments. I also remain unconvinced that the additional findings establish a sufficiently substantial conceptual advance over prior work. I therefore maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.