acceptodds
Under review as a conference paper at ICLR 2027

Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference

Abstract

Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We find that the LLM need not be used this way: even a truncated prefix of its response raises SLM accuracy, with gains that broadly grow with prefix length on all four benchmarks we study, also it hold across model families, tokenizers, and SLM sizes. We therefore introduce LLM Shepherding, which requests only a short prefix (a *hint*) from the LLM and lets the SLM produce its own complete response. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On GSM8K, CNK12, HumanEval, and MBPP, we compare shepherding with routing and cascading. Across full cost–accuracy sweeps, Reactive Shepherding attains the largest area under the cost–accuracy curve on all four benchmarks; at a fixed target of 90% of LLM accuracy, it is the cheapest method on all four; and at default operating points it reduces cost by 33–65% relative to LLM-only inference and delivers up to 4x cost reduction compared to state-of-the-art routing and cascading baselines. To our knowledge, shepherding is the first to formalize and learn a single hand-off prefix hint, controlled only through `max_new_tokens`, under a black-box per-token cost model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, LLM cascading, SLM-LLM collaboration, Cost-efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ah96OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors address the cost-accuracy tradeoff in deploying large language models (LLMs), where existing routing and cascading frameworks treat LLMs as all-or-nothing resources, incurring full LLM inference cost whenever a query exceeds small language model (SLM) capability. The authors propose LLM Shepherding, a novel framework that requests only a short prefix (a "hint") from the LLM and then lets the SLM generate its own complete response conditioned on the hint. Two instantiations are introduced: Proactive Shepherding, which directly predicts whether a hint is needed and how many tokens to request, and Reactive Shepherding, which cascades from SLM consensus checks into hint-guided re-generation. A two-stage predictor based on DeBERTa-v3-large jointly determines hint necessity and hint size, addressing key challenges including zero-inflated hint distributions and non-smooth accuracy responses. Theoretically, the authors prove that under oracle decision-making, shepherding costs no more than routing and cascading under mild token-price assumptions. Extensive experiments on four benchmarks spanning mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) demonstrate that Reactive Shepherding achieves the highest cost-accuracy area-under-curve (AIQ) across all benchmarks, is the cheapest method at a 90% LLM-accuracy target, and reduces cost by 33–65% relative to LLM-only inference, with up to 4× cost reduction over state-of-the-art routing and cascading baselines.

Critical strengths
  1. The paper challenges the prevailing all-or-nothing paradigm of routing and cascading by proposing LLM Shepherding, a novel framework that leverages only a short LLM prefix to guide SLM generation. Theoretically, it proves that shepherding incurs no higher cost than routing and cascading under oracle decision-making, and empirically validates that hint size consistently correlates with SLM accuracy across diverse model families, tokenizers, and SLM sizes.

  2. Extensive experiments across four benchmarks spanning mathematical reasoning and code generation demonstrate that Reactive Shepherding achieves state-of-the-art cost-accuracy tradeoffs, reducing inference cost by 33–65% relative to LLM-only deployment and up to 4× over existing routing and cascading baselines, while exhibiting strong cross-domain generalization from math-trained predictors to unseen code generation tasks.

Critical weaknesses
  1. The biggest omission is the lack of comparison with speculative decoding. Both shepherding and speculative sampling use a small plus large model pair to reduce LLM cost, and they occupy nearly the same design space. Yet the paper does not cite or evaluate any speculative decoding baseline. Without this, it is impossible to tell whether shepherding actually delivers a better cost-accuracy frontier, or just beats up on the weaker routing and cascading baselines. This directly undercuts the paper's core claim.

  2. The empirical foundation is narrow. All main results use a single Llama-3.2-3B and Llama-3.3-70B pair, coding evaluation is on saturated 2021 benchmarks (HumanEval, MBPP) with a math-trained predictor transferred zero-shot, and cost and latency numbers come from a bare local setup rather than real serving engines like vLLM. The 33% to 65% cost-reduction claim may not hold across other model pairs, harder tasks, or production deployments.

Questions for authors
  1. Do the cost-accuracy results still hold when the SLM is served through vLLM or SGLang, where continuous batching and prefix caching change effective throughput?

  2. How sensitive are the findings to the model pair? Have you tested on non-Llama architectures or different SLM sizes?

4 comments

We thank the reviewer for the comments. We address each point below, grouping Q2 with W2.

W1: Comparison with speculative decoding

Speculative decoding returns the LLM's own output at the LLM's own token count. On a cost-accuracy plane it is therefore the LLM-only point, the reference point of Figure 3 and Tables 2 to 5. The submission discusses it in Appendix K.3 and K.5, and we agree that this discussion belongs in the main text. In the revised paper, we will move it.

The two methods change different quantities. Speculative decoding is lossless and reduces latency. It does not reduce the LLM's computation: the total number of arithmetic operations is unchanged when a draft is accepted and higher when one is rejected (Leviathan et al., 2023, Section 3.4). Shepherding reduces the number of LLM tokens requested, and accuracy becomes a function of the hint length.

They also need different access. Speculative decoding needs the target model's next-token distribution at every drafted position, so only the party hosting the LLM can run it. Shepherding needs one request parameter, max_new_tokens. Under per-token billing, a provider that serves with speculative decoding bills the same tokens at the same price, so evaluating it would add no point to Figure 3. Routing and cascading are the existing methods that trade accuracy for cost, which is why they are our baselines.

When the LLM is self-hosted, speculative decoding can lower the cost of each LLM token for every method alike, and shepherding still requests fewer of them.

W2 & Q2: Model pairs and SLM sizes

The submission tests both in Appendix A. The hint-size sweep of Figure 2 is repeated with a Gemma-3-4B-IT / Gemma-3-27B-IT pair and with a cross-family pair, Llama-3.2-3B with Gemma-3-27B-IT (Table 6, Figure 5). Accuracy rises approximately monotonically with hint size in each. Table 7 varies the SLM size on GSM8K: from no hint to a 90% hint, accuracy rises by 33.2 points for Llama 1B, 16.6 for 3B, and 10.4 for 8B. Table 8 runs the full pipeline and all four baselines on the Gemma pair.

Q1: Serving the SLM through vLLM

The SLM cost in the paper already comes from vLLM. The SLM price in Appendix G was measured by serving Llama-3.2-3B-Instruct with vLLM and continuous batching on the RTX 5090 used in the experiments. The effective throughput the reviewer mentions is therefore the one that sets the price. Cost is counted per query, from the tokens each query sends and receives, so batching changes the price per SLM token and not the token counts. Prefix caching would only lower that price further, and Appendix G shows that the ranking on GSM8K is unchanged even when the SLM price is ten times higher. The serving engine does not change the model or its sampling distribution, so accuracy does not depend on it beyond sampling noise.

Reviewer_Ah96OpenReview ↗

Thanks for the follow-up comments.

Thanks for the follow-up comments. They address some of my concerns, but my main concern remains the lack of comparison with speculative decoding. Both Shepherding and speculative decoding aim to improve inference efficiency through small–large model collaboration. Differences in implementation alone do not justify excluding speculative decoding from the comparison. If they are alternative approaches, their quality–efficiency trade-offs should be compared directly. If they are compatible, at least the authors should evaluate whether Shepherding provides additional gains on top of a state-of-the-art speculative decoding method. Thus I will maintain my score.

AuthorsOpenReview ↗

Thanks for following question

We thank the reviewer for the following question. Speculative Decoding (SD) and Shepherding are two different techniques solving different problems. SD is a latency-reducing technique used in inference engines in the cloud; k LLMs are run in parallel to ideally generate k tokens in the time required for one token generation by the LLM. A draft model is only used to facilitate this parallelization. The final output will always be 'equivalent' to the LLM response. In contrast, LLM Shepherding is an SLM accuracy-improving technique, not an LLM latency-reducing technique. It aims to improve the SLM response accuracy on a device with minimal help from the cloud LLM. Thus, SD and LLM Shepherding are not alternative techniques.

Also, from a systems perspective, Shepherding is deploying the SLM on the device and seeking help from the cloud LLM. If the LLM is generating tokens using SD (using a draft model in the cloud), the turnaround time for the SLM on the device to receive these tokens will be reduced. Beyond this, SD does not have any effect on the Shepherding system. Likewise, the Shepherding system does not have any control over how LLM generates the tokens beyond specifying the maximum number of tokens.

Reviewer_Ah96OpenReview ↗

Thanks for the follow-up comments.

Thank you for the response. However, I do not think the authors have adequately addressed my main concern, so I will maintain my original score.

  1. The response mainly explains implementation differences, which do not by themselves justify excluding SD from evaluation. Given its widespread adoption, the practical benefit of Shepherding remains unclear without either a direct quality–efficiency comparison or evidence of additional gains when combined with a strong SD method. Moreover, the description of SD is inaccurate: standard SD uses a single target LLM to verify multiple draft tokens in parallel, rather than running kk LLMs.

  2. If black-box API access is the main justification for excluding SD, experiments with proprietary models such as GPT, Claude, or Gemini would strengthen the empirical support for this setting. The current experiments exclusively use open-weight models accessed through APIs.

Reviewer_s5cROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLM Shepherding, a way to reduce the cost of using a large model by asking it only for the first few tokens of its answer (a hint) and letting a small model finish the response. The authors show that the small model gets more accurate as the hint gets longer, and prove that with oracle decisions, and when LLM output tokens are the main cost, this costs no more than routing or cascading. They then train a predictor that decides whether a hint is needed and how long it should be, in a routing-style and a cascading-style version. On GSM8K, CNK12, HumanEval and MBPP, the cascading-style version gives the best cost-accuracy trade-off.

Critical strengths

(1) I like the idea. It is simple, it only needs the max_new_tokens option of a normal LLM API, so it also works with closed models, and routing and cascading are just its two extreme cases. Fig. 2 and Corollary 2.2 make it easy to see why and when a short hint should save money.

(2) The experiments look at the problem from three sides (the full cost-accuracy curve, the cheapest way to reach 90% of LLM accuracy, and the default settings), and the cascading-style version wins on the first two on all four benchmarks. It is also nice that the predictor trained only on GSM8K works on HumanEval and MBPP without retraining.

Critical weaknesses

(1) My main concern is the positioning of the paper. It says that existing methods treat the LLM as all-or-nothing, but I don’t think this is true. Co-LLM (Shen et al., ACL 2024) and R2R (Fu et al., NeurIPS 2025) already let a small model hand single tokens to a larger one, and AdaSwitch (Sun et al., EMNLP 2024) lets a local model ask a cloud LLM for help only on the hard reasoning steps. None of them is cited, so it is hard to tell what is new here beyond using one prefix through max_new_tokens. I would like the authors to discuss these works and, if possible with the same model pair, compare with at least one of them.

(2) I also think the cost analysis leaves out costs that matter in practice. The paper motivates the method with edge devices and latency-sensitive applications, but Sec. 4.5 only reports the small-model and decision-module times, not the end-to-end latency. In the cascading-style version, a hinted query runs K small-model samples, then an LLM call, then another small-model generation, so it may well be slower than just calling the LLM. Building the training labels also needs about ten LLM calls and ten small-model runs per training query (Sec. 3.1), and this cost is not counted anywhere. I would like to see the end-to-end latency per query for each method, and the labeling cost together with how many queries it takes before shepherding pays it back.

(3) In Sec. 4.1, the GSM8K test set is split into 776 questions for evaluation and 543 for validation, while CNK12 gets its own validation split. Why is the GSM8K validation set taken from the test questions rather than from the 7,473 training questions, and was this split random? Since the results cover only part of the test set, they cannot be compared directly with other work on GSM8K. It would help to take the validation set from the training data and report the results on the full 1,319 test questions.

(4) I also find the evaluation too narrow to support a general claim. Also, GSM8K is nearly solved by the LLM (98.0%), with 80% of its queries needing no hint at all. It is therefore not clear whether shepherding still helps on harder reasoning, where the gap between the two models is larger and a hint has more to add, or on open-ended tasks. I would like to see results on harder math and science benchmarks such as MATH-500, AIME and GPQA, and on MMLU, ARC-Challenge and MT-Bench, which are standard in the routing literature (e.g. RouterBench, RouteLLM ... etc). Finally, HumanEval+ and MBPP+ have 80 and 35 times more test cases than the originals and would make the code results more reliable.

Questions for authors

(1) In Fig. 2, the small model without a hint gets about 81% on GSM8K, 47% on CNK12 and 54% on MBPP, but Tables 2, 3 and 5 report 73.0%, 53.8% and 44.2%, while the LLM numbers match. Is Fig. 2 computed on a different split, prompt or decoding setup? Since this figure is the main motivation of the paper, the numbers should agree.

(2) In Tables 2 to 5, the baselines use their released default settings, while the shepherding parameters are chosen by the authors (Sec. 4.4). How were they chosen, and would tuning the baselines change the ACE ranking? At these points ABC is more accurate than Reactive Shepherding on all four benchmarks, so I am not sure the ACE comparison is fair. Confidence intervals would also help here, since HumanEval has only 164 problems.

(3) The abstract says shepherding "delivers up to 4× cost reduction", but Sec. 4.4 shows that this compares cost reductions relative to the LLM (33% vs. 5 to 8% for routing on CNK12). Compared directly with the routing baselines on CNK12, Reactive Shepherding is only about 27 to 30% cheaper ($0.373 vs. $0.513 and $0.533). Could you rephrase this?

I am willing to raise my score if my concerns are addressed.

3 comments

We thank the reviewer for the constructive feedback. We address the reviewer’s questions below.

W1: Positioning relative to Co-LLM, R2R, and AdaSwitch We thank the reviewer for these pointers. The three works, and the later CITER, switch between the two models many times within one response.

  • Co-LLM (Shen et al., ACL 2024): the base model learns, as a latent variable, whether to generate the next token itself or defer it to the assistant model.
  • R2R (Fu et al., NeurIPS 2025): a router sends to the LLM only the tokens at which the SLM would diverge from the LLM's reasoning path.
  • AdaSwitch (Sun et al., EMNLP 2024): the local agent checks its own reasoning steps and asks the cloud agent for help on the steps it judges wrong.
  • CITER (Zheng et al., 2025): a router trained by policy optimization decides for every token which model generates it. Shepherding makes one decision per query: how many tokens the LLM writes at the start of the response. The LLM is called once with max_new_tokens set to this number, and the SLM completes the response. The LLM therefore reads the question once, whereas every switch in the methods above re-reads the growing context as billed input tokens. Shepherding also needs only a black-box API and leaves both models unchanged. We ran CITER with the same model pair and prices, and charged each LLM invocation for the context it re-reads. The table below gives the AIQ (%). The GSM8K column uses all 1,319 test questions (W3). | Method | GSM8K | CNK12 | HumanEval | MBPP | |---|---:|---:|---:|---:| | Reactive Shepherding | 85.7 | 69.3 | 69.3 | 63.3 | | Proactive Shepherding | 84.0 | 66.1 | 63.0 | 59.1 | | CITER | 72.3 | 55.8 | 54.2 | 44.7 |

In the revised paper, we will add CITER to Table 1 and Figure 3, discuss the four works in the related work, and clarify that the "all-or-nothing" statement applies to query-level routing and cascading.

W2: End-to-end latency and labeling cost Latency. The estimated end-to-end latency per query on the GSM8K test set is 472 ms for Proactive Shepherding and 1,375 ms for Reactive Shepherding, against 446 ms for LLM-only and 385 ms for SLM-only. Reactive Shepherding takes 596 ms if its three consensus samples are drawn concurrently. It is slower than LLM-only because it runs three SLM generations before any LLM call. We estimate these values from the component times of Section 4.5 and Groq's published generation speed for Llama-3.3-70B, because Groq no longer serves the model. We will add them to Section 4.5. Labeling cost. Each training query needs ten LLM calls (nine partial hints and one full response) and eleven SLM runs. At the prices of Appendix G, the LLM calls cost USD 6.44 for the 7,473 GSM8K training questions and USD 28.52 for the 18,250 CNK12 training questions. These figures follow the paper's cost model, which bills the question as the only input. With the full prompt billed, the costs are USD 15.66 and USD 51.02. The SLM runs add less than USD 1. Pay-back. Reactive Shepherding recovers the labeling cost after about 80,000 GSM8K queries and about 330,000 CNK12 queries. These counts follow from the savings of Reactive Shepherding over LLM-only: USD 0.1768 against USD 0.0709 per 1,319 GSM8K queries (W3), and USD 0.559 against USD 0.373 per 2,147 CNK12 queries (Section 4.3).

W3: GSM8K validation split and results on all 1,319 test questions The reviewer's reading is correct. The GSM8K validation set was taken from the test set, and 27 test questions that neither model answered at any hint size were removed. The submitted GSM8K numbers are therefore not comparable with published results. This is our mistake. We have redone GSM8K as the reviewer asks. The validation set (543 questions) is now drawn from the training set, the predictor is retrained on the remaining 6,873 training questions, and every method is evaluated on all 1,319 test questions. New hints and LLM answers come from a self-hosted Llama-3.3-70B, because Groq no longer serves the model. Each method is swept on the validation set, and its non-dominated configurations are evaluated once on the test set.

MethodAIQ, all 1,319 questions (validation-selected)AIQ, submitted (776 questions)
RouteLLM79.181.6
GraphRouter79.081.5
Proactive Shepherding84.087.9
FrugalGPT78.080.5
ABC85.587.9
CITER72.3n/a
Reactive Shepherding85.791.5

Reactive Shepherding has the highest AIQ, with ABC 0.2 points behind, and Proactive Shepherding remains the best routing method. We will revise the paper and update GSM8K numbers.

W4: Harder benchmarks We ran MATH-500 with the same Llama pair. The two models are 34 points apart on MATH-500 (SLM 39.4%, LLM 73.6%), against 25 on the full GSM8K test set and 31 on CNK12. No MATH data is used for training or tuning. The predictor is the one trained on CNK12. Every method uses the configurations selected on the CNK12 validation set, which are applied unchanged to the 500 MATH-500 problems. As the hint grows from 10% to 90% of the LLM’s answer, SLM accuracy rises monotonically from 42.6% to 60.6%, the same trend as in Figure 2. The table gives the AIQ (%) of each method.

MethodAIQ on MATH-500
RouteLLM53.1
GraphRouter50.8
Proactive Shepherding50.9
FrugalGPT52.6
ABC44.9
CITER40.9
Reactive Shepherding53.5
Among the other benchmarks the reviewer lists, AIME and GPQA are too hard for this pair. Llama-3.3-70B solves a few AIME problems and about half of GPQA's 198 questions, so a 90%-of-LLM target says little. MT-Bench is open-ended and needs a judge model in place of the binary quality function, which Section 5 lists as a limitation. On multiple-choice benchmarks such as MMLU and ARC-Challenge the response is a single letter, so the hint is the full response and shepherding reduces to routing. We will scope the claims to tasks with multi-step responses. For code, we agree about test coverage and will re-score the stored HumanEval and MBPP generations with HumanEval+ and MBPP+ in the revision.

Q1: SLM accuracy in Figure 2 versus Tables 2, 3 and 5 Figure 2 is computed on the full dataset of each benchmark, whereas Tables 2 to 5 report the test split, which is why the SLM accuracies differ.

Q2: How the shepherding parameters were chosen, tuning the baselines, and confidence intervals Parameter choice. The predictor has two heads, one for whether a query needs a hint and one for the hint size. Both are trained once on the training split by minimizing L=λ⋅BCE+(1−λ)⋅SmoothL1\mathcal{L} = \lambda \cdot \mathrm{BCE} + (1-\lambda) \cdot \mathrm{SmoothL1}, where BCE is the binary cross-entropy on the hint indicator and SmoothL1 is the Huber loss on the log hint size. The weight λ\lambda is tuned on the validation split. The three shepherding parameters are then chosen on the validation split as well. They are the probability threshold α\alpha above which a query is given a hint, the hint-size threshold ηhint\eta_{\text{hint}} above which a query is sent to the LLM, and the number KK of SLM samples in the consensus check. We sweep the three parameters and take the cheapest configuration that reaches 90% of the LLM's accuracy on the validation split. Tables 2 to 5 report the shepherding methods at this configuration. Tuning the baselines. Tables 2 to 5 use the released defaults of each baseline. At these points, ABC is more accurate than Reactive Shepherding because it sends more queries to the LLM, and it costs more. The comparison with tuned baselines is in Figures 3 and 4, where each baseline is swept over its own hyperparameters and all methods are compared at the same accuracy. Confidence intervals. On HumanEval's 164 problems, the 95% confidence interval of an accuracy near 80% is about ±6 points. We will add confidence intervals to Tables 2-5.

Q3: "Up to 4x cost reduction" We agree that the phrase is ambiguous. The factor compares cost reductions relative to LLM-only: on CNK12, Reactive Shepherding reduces cost by 33%, against 5 to 8% for the routing baselines. In the revised paper, we will replace the "4x" and "1.5x" factors with these reductions and add the direct comparison, in which Reactive Shepherding costs 27 to 30% less than the routing baselines on CNK12.

Thank you for the detailed response and the new experiments. I think the new clean GSM8K re-evaluation and MATH-500 show that the gain over the baselines mostly disappears (85.7 vs 85.5 and 53.5 vs 53.1). Also, reactive shepherding is still about 3x slower than calling the LLM directly. I've decided to keep my score.

Reviewer_5h9oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

his paper proposes LLM Shepherding, which requests a limited prefix from a large model and provides it to a small model, using a learned policy to decide whether assistance is needed and how many hint tokens to request. The authors study proactive and reactive variants, derive oracle cost conditions, and compare cost–accuracy trade-offs against routing and cascading on mathematical reasoning and code-generation benchmarks.

Critical strengths
  1. The framework offers a practical intermediate option between small-model-only execution and full large-model generation, implemented through an accessible output-token limit.

  2. The evaluation considers full cost–accuracy curves and fixed accuracy targets, and the appendices additionally test cross-model transfer and sensitivity to small-model pricing.

Critical weaknesses
  1. The fixed-target cost comparisons need stronger separation between configuration selection and final testing. Appendix J.1 states that trade-off parameters are swept on each benchmark’s test set, while the minimum-cost metric selects a configuration satisfying the accuracy target; this establishes a retrospective test-set envelope rather than the cost of a policy selected without access to test outcomes.

  2. The oracle formulation and practical fallback do not use an entirely consistent definition of success. Equation 2 defines the minimum hint by whether the hint-conditioned SLM succeeds, whereas Appendix D.2 assigns a full-response-length label and directly returns the LLM output when no partial hint succeeds, which is a different action and need not satisfy the same success condition.

  3. The reported accuracy and cost require a clearer account of the seven-trial evaluation protocol. The paper reports majority-vote outcomes over seven trials, but for code generation it describes executing programs and checking whether all tests pass, leaving unclear whether voting uses program identity or correctness outcomes and whether the cost includes all seven trials.

Questions for authors
  1. Could the authors select all thresholds and hint budgets on validation data and report their held-out test performance, including whether the fixed accuracy requirement is actually met?(

  2. How is majority voting implemented for code outputs, and are the reported monetary costs totals over seven trials or averages per trial?

  3. Could the authors report complete request latency, including remote LLM calls, hint generation, and the final SLM response, rather than only local decision-stage latency?

2 comments

We thank the reviewer for the comments. We address the questions below.

W1 & Q1: Selection on validation data The reviewer is right about the curves of Section 4.2. These curves, and the AIQ values of Table 1, were traced over the grid of Appendix J.1 (α\alpha, ηhint\eta_{\text{hint}}, kk) on each test set, so their points are test-set optima. We thank the reviewer for raising this, and we have repeated the selection on validation data for CNK12. We swept the same grid on the CNK12 validation split (1,073 questions), kept the configurations on the validation curve, and evaluated each of them once on the test set. The table gives the resulting AIQ next to the submitted values. Under validation selection the AIQ is 66.0 for Proactive and 67.8 for Reactive Shepherding, against 61.2 to 62.6 for the baselines, so both shepherding methods remain above every baseline. We will run the same experiment on the other three benchmarks and update the paper with the new results.

MethodAIQ, validation-selectedAIQ, submitted (test-selected)
RouteLLM61.462.0
GraphRouter62.663.3
Proactive Shepherding66.066.1
FrugalGPT61.562.4
ABC61.262.0
Reactive Shepherding67.869.3

The operating points of Sections 4.3 and 4.4 do not come from these test-set sweeps. Section 4.3 targets the 90% LLM accuracy requirement. Eq. (4) is solved on the validation split, and the cheapest validation configuration that reaches the requirement is the one listed in Appendix I. Figure 4 reports its cost and accuracy on the test set, and the accuracies above the bars show that the requirement is met on all four test sets. Section 4.4 (Tables 2–5) reports each method at one fixed default setting, which is the predictor trained by minimizing the loss of Section 3.2 for the shepherding methods and the defaults released by their authors for the baselines. We will state the protocol of each section in Sections 2.3 and 4.3 and in Appendix J.1.

W2

We thank the reviewer for pointing this out. We will make the definition in (2) consistent by capping the minimum hint size at the LLM response length. Further, if the minimum hint size equals the LLM response length, we use the LLM response directly in place of the SLM response.

W3 & Q2: Voting and cost over the seven trials

For code generation, the vote is over execution outcomes, not over program text. Each of the seven sampled programs is run against the benchmark's unit tests; a trial counts as correct only when all tests pass (Appendix H.2), and a query counts as solved when a majority of the seven trials are correct. The reported cost is the average over the seven trials, not their sum. The LLM cost of a query is incurred once and is identical across all trials. We will add both points in Section 4.1 and Appendix H.2.

Q3: End-to-end latency

The reviewer is right that Section 4.5 reports the decision stage only. Llama-3.3-70B is no longer served on Groq, so we estimated the end-to-end latency on the GSM8K test set from the per-query component times of Section 4.5 and Groq's published generation speed for this model. The estimated average per query is 472 ms for Proactive Shepherding and 1,375 ms for Reactive Shepherding (596 ms if the three SLM samples are drawn concurrently), against 446 ms for LLM-only and 385 ms for SLM-only. We will add this estimate to Section 4.5.

Thanks for the authors' responses. It is suggested the authors to include these new content in to the revision and clarfiy the details. I keep my original rating by considering the novelty and the soundness of the experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.