acceptodds
Under review as a conference paper at ICLR 2027

The Right Kind of Disagreement: Dynamic Collaborator Pruning for Speculative Token-Level Collaboration

Abstract

Speculative token-level collaboration reduces collaboration overhead by having a main model generate candidate tokens while auxiliary models verify them in parallel. However, existing methods typically preselect collaborator models and keep them active throughout the generation process, so verification costs accumulate across decoding rounds. We propose CoStay, a disagreement-driven selective-exit framework for token-level collaboration systems. Guided by the principle that collaboration requires a balance between consensus and informative disagreement, CoStay uses disagreements between the main and collaborator models during token-level decisions as an online signal. When multiple collaborators participate, aggregate disagreement cannot be attributed to any single collaborator. CoStay therefore applies leave-one-out analysis to quantify each collaborator’s marginal impact on the aggregated decision and uses a two-sided confidence-bound rule to determine when a collaborator should stop participating in subsequent rounds. Across six benchmarks and eight model configurations, CoStay achieves 99.12% of the mean accuracy of SAFE, CoSD, and UniTE, with a gap of only 0.64 percentage points, while reducing the average collaborator token cost by 44.43%. In end-to-end runtime experiments, this reduced collaboration overhead translates into a 40.8%–57.3% throughput improvement over full-participation collaboration methods. The anonymized source code is available at https://anonymous.4open.science/r/CoStay-0732/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Token-level Collaboration, Speculative Decoding, Large Language Models, Selective Exit, Collaborative Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MwzdOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CoStay a method for dynamic collaborator pruning for speculative token-level collaboration. Existing methods today keep every collaborator active for the whole generation, which is costly. The author's propose a training-free method that decides when a collaborator should stop. It measures how much each collaborator changes draft rejection, tracks this over a sliding window with confidence bounds, and removes collaborators whose influence stays very low or very high.

Critical strengths

[S1] The efficiency gains are substantial. Collaborator token cost drops by about 46% compared with SAFE for about 1% decrease in accuracy, and throughput improves on all three tested configurations.

[S2] The analysis by collaborator strength is good. With identical models, CoStay cuts cost by over 80% with no accuracy loss, and with a weaker collaborator it limits the damage better than other methods. This comes at the trade-off of some accuracy with stronger collaborators, which the authors attribute to removing some useful participation

[S3] The bounded distinctiveness stopping idea is intuitive and well grounded. It removes collaborators that add almost nothing and those that intervene too much, and it builds on the ensemble view that diversity helps only when it is balanced

[S4] The comparison of CoStay with the fixed-cutoff version CoStay-W is good. The fixed cutoff is cheapest but loses clearly more accuracy with stronger collaborators. The authors attribute this to stopping useful collaboration without regard to what it contributes.

Critical weaknesses

[W1] The leave-one-out comparison is presented as a key part of the method, but all main experiments use only one collaborator. The cases where leave-one-out actually matters, with several collaborators, appear only in Table 8. It could be useful to move this into the main text.

[W2] The comparison of CoStay with CoStay-W, the fixed cutoff uses far fewer collaborator calls than CoStay (Table 2). A comparison at a similar cost would better show how much of the saving comes from the disagreement signal itself.

[W3] Once a collaborator is stopped, it never rejoins later rounds. A collaborator that was unhelpful early in an answer might still be useful later, for example in a later reasoning step, and it could be useful to mention or explore this as future works.

[W4] Because the stopping rule waits for a full window of 10 rounds, a collaborator can never be stopped before round 10. The window size itself is fixed rather than studied. It could be useful to discuss this design choice.

Questions for authors

Please address weaknesses.

1 comment
AuthorsOpenReview ↗

We summarized the multi-collaborator results in the main text and added a fixed-cutoff comparison at a similar token cost. We also explained how the 10-round window affects the earliest stopping decision and discussed collaborator reactivation as future work. These changes have significantly improved the quality of the manuscript. The revised manuscript added the relevant content and marked it in blue. We hope they address the reviewers' concerns.

A1 We added multi-collaborator results to the main text.

Thank you for this valuable suggestion. Leave-one-out comparison is applicable to both single- and multiple-collaborator settings. Due to the page limit, we retain the detailed results in Table 8 in the appendix, while clarifying its role in the main text to better highlight its effectiveness in multi-collaborator scenarios. For instance, we added the following description to the main text:

We further evaluate CoStay in multi-collaborator settings to examine its scalability and the effectiveness of leave-one-out-based collaborator selection. The results show that adding more collaborators does not necessarily yield proportional accuracy gains, despite introducing additional inference overhead. For the three two-collaborator configurations, CoStay achieves an average accuracy of 71.94%, improving upon Base-only by 3.55 percentage points, while reducing collaborator token costs by 65.32% and 56.74% compared with SAFE and UniTE, respectively. Moreover, the benefit of additional collaborators is non-monotonic: adding HY4 to Q25$\rightarrow$Q34 reduces accuracy from 73.83% to 72.17%, while introducing another collaborator further increases token cost without improving accuracy. These results suggest that increasing the number of collaborators incurs additional inference costs, but does not guarantee commensurate accuracy improvements. Thus, using a single collaborator may offer a more favorable trade-off when its contribution is sufficient.

A2 We added cost-matched fixed-cutoff experiments.

Thank you for this valuable suggestion. We added the method to the revised manuscript: CoStay-W(43), which uses a fixed stopping point. Since CoStay requires an average of 42.74 collaborator calls, we set the fixed cutoff of CoStay-W to 43 to ensure a comparable collaboration budget. Its actual average number of calls is slightly lower because some tasks terminate before reaching the cutoff.

Unlike CoStay, a fixed-cutoff strategy requires the stopping budget to be manually specified in advance. A smaller cutoff may reduce computational cost but prematurely terminate useful collaboration (such as CoStay-W(10)), whereas a larger cutoff incurs more cost without necessarily improving accuracy. In contrast, CoStay adapts the stopping time to the observed disagreement. Experimental results show that CoStay achieves higher accuracy than CoStay-W(43) under a comparable collaboration budget, demonstrating the advantage of disagreement-guided adaptive deactivation over a manually predefined stopping point.

Table 1: Overall performance across all single-collaborator configurations

MethodAccuracy↑Calls↓Active Ratio↓Token Cost↓
CoStay-W(10)69.83%10.0018.25%49.91
CoStay-W(43)71.21%41.9065.94%208.88
CoStay71.58%42.7462.10%212.70

A3 We included collaborator reactivation in future work.

Thank you for this helpful suggestion. We have acknowledged this limitation in the revised manuscript and identified adaptive collaboration as a promising direction for future work. The revised text is as follows:

Future work will explore adaptive collaboration mechanisms to address these limitations. In particular, dynamically assigning non-uniform weights according to the observed reliability of each LLM and enabling the reactivation of previously stopped collaborators could better exploit complementary capabilities across different reasoning stages.

A4 We added a discussion on the window-size selection.

Thank you for this insightful comment. The full observation window is required to avoid premature stopping decisions based on insufficient observations or transient fluctuations. A smaller window enables faster responses but is more sensitive to short-term fluctuations, whereas a larger window provides more historical information at the cost of delayed stopping decisions and potentially unnecessary collaboration overhead.

In this work, we set the window size to W=10 as a reasonable trade-off between decision stability and responsiveness. Since the window size and the upper and lower thresholds are interdependent, jointly optimizing these parameters would substantially expand the search space and increase the experimental cost. More importantly, our primary focus is to evaluate the effectiveness of the proposed mechanism rather than extensively optimize its window size.

Reviewer_FziVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoStay, a training-free method for dynamically pruning collaborators in speculative token-level LLM collaboration. Unlike existing methods that retain a predefined set of collaborators throughout generation, CoStay estimates each collaborator's marginal influence on the collaborative decision using a leave-one-out comparison of chunk-level disagreement intensity. It then employs a sliding observation window and two-sided Student's t confidence bounds to remove collaborators with persistently low or high measured influence.The method is evaluated on six benchmarks (ARC, MMLU, HellaSwag, TruthfulQA, WinoGrande, and GSM8K) using four models and eight directed single-collaborator configurations, with additional multi-collaborator experiments. The paper reports that CoStay achieves 71.58% average accuracy, retains 99.12% of the mean accuracy of SAFE, CoSD, and UniTE, and reduces their mean collaborator token cost by 44.43%. Runtime experiments on three configurations also show throughput improvements over the full-participation collaboration baselines.

Critical strengths
  1. CoStay uses signals available during existing verification, requires no additional model forward passes, and does not require retraining. Its focus on controlling collaborator participation complements existing approaches that focus on token-level routing or prediction fusion.
  2. The evaluation covers six benchmarks, several model-capability relationships, and both single- and multi-collaborator settings. The inclusion of a fixed-cutoff baseline, parameter sensitivity analysis, collaborator-exit statistics, and qualitative examples helps characterize the behavior of the proposed method. The paper also provides an anonymized source-code link, which is helpful for reproducibility.
Critical weaknesses
  1. Eqs. (2)–(3) measure change in the first disagreement position after removing a collaborator, not the collaborator’s full effect on the aggregated distribution or replacement token. If that position stays fixed while the replacement token changes, a materially influential collaborator may receive near-zero influence; conversely, large influence does not imply helpfulness or harm. This weakens the two-sided stopping rule, which equates low influence with redundancy and high influence with harmful intervention, although the statistic establishes neither. The paper acknowledges that disagreement signals cannot directly determine benefit, but the empirical relationship between the statistic and collaborator utility remains insufficiently established. Since it grounds the pruning mechanism, the issue affects the method’s justification, not just presentation.

  2. The aggregate results are promising, but the overall accuracy-retention figure obscures meaningful differences across collaborator configurations. For example, in the stronger-collaborator setting, CoStay achieves 69.42% accuracy, compared with 73.08% for SAFE and 72.75% for UniTE. In the HY4→Q34 configuration, CoStay achieves 69.50%, which is 5.67 percentage points below SAFE. These results illustrate that the pruning mechanism can discard useful collaboration as well as reduce harmful or redundant participation.The paper explicitly describes the accuracy differences on the deterministic tinyBenchmarks subsets as descriptive rather than statistically significant. Nevertheless, the central claim is that CoStay preserves accuracy while substantially reducing cost.

Questions for authors
  1. Does the proposed marginal-influence statistic capture changes that matter to generation quality?
  2. Since the magnitude of the leave-one-out statistic does not distinguish helpful from harmful intervention, what empirical evidence supports interpreting persistently high influence as a reason to terminate collaboration?
  3. Can the authors provide paired uncertainty estimates for question-level accuracy differences and further analyze the effects of overlapping, temporally dependent observation windows on the stopping rule?
  4. can the end-to-end runtime evaluation be extended to more model configurations and include Base-only throughput?
4 comments
AuthorsOpenReview ↗

We clarified how marginal influence measures changes in draft acceptance and used the threshold sensitivity analysis to explain the upper stopping rule. We also added paired confidence intervals for accuracy and token cost, discussed dependence between overlapping windows, and extended the throughput evaluation to more configurations, including Base-only. The revised manuscript added the relevant content and marked it in blue. We hope these efforts address the reviewers' concerns.

A1/W1: We clarified the interpretation and role of marginal influence.

Thank you for this comment. The marginal-influence statistic measures a collaborator's impact on generation decisions by comparing outcomes with and without it, rather than assessing whether its influence is beneficial or harmful.

CoStay is motivated by the observation that a collaborator's intervention may correct an error or alter a decision that is already correct. Since the magnitude of influence alone cannot distinguish between these cases, CoStay tracks each collaborator's influence over a sliding window and uses it as a signal for managing continued participation, rather than treating a single high-influence event as evidence of a useful contribution. The results show that CoStay achieves 71.58% accuracy, compared with 69.25% for Base-only. Compared with SAFE/Unite/CoSD, it substantially reduces token cost with only a small accuracy trade-off.

A2/W1: We explained and empirically supported the upper exit condition.

Thanks. Regarding the concern in W1 that LOO may overlook distributional changes, it focuses on changes in the accepted sequence rather than the underlying probabilities. For example, given the same draft "ABC", if ABC remains accepted after removing a collaborator, the acceptance decision is unchanged; if only AB is accepted, the collaborator's influence is captured.

Next, our motivation for the upper exit condition is to control a regime in which a collaborator repeatedly exerts a strong influence on the base model's decisions. We hypothesize that sustained strong intervention may make collaboration less stable and may not justify its continuing computational cost. CoStay consequently applies the upper threshold to a window-level statistic, rather than terminating a collaborator because of a single high-influence event.

Table 1 shows that continued collaboration can reduce accuracy in weaker collaborator settings: Base-only achieves the highest accuracy, while CoStay and its early-exit variants outperform full-time collaboration methods such as SAFE. Moreover, Figure 4 (Appendix C, page 15) shows that higher thresholds increase cost but do not necessarily improve accuracy. These results support the need for an upper exit condition to prevent prolonged, potentially harmful collaboration, without assuming that strong influence is inherently harmful.

Table 1 The key results in weaker collaborator settings

MethodAccuracy↑Token Cost↓
Base-only71.25%—
SAFE68.67%361.42
CoSD67.50%354.37
UniTE69.83%289.19
CoStay-W(10)70.08%49.76
CoStay-W(43)69.83%206.31
CoStay-Mean70.58%149.75
CoStay70.25%254.0

A3/W2: We added paired uncertainty analysis and discussed temporal dependence.

Thank you for this insightful comment. We added paired question-level analysis to assess the accuracy-cost trade-off. Table 2 shows that CoStay reduces collaborator token cost by 40.06%-46.91% across all baselines, while the accuracy differences are not statistically significant for SAFE and CoSD. This indicates substantial cost savings without significant accuracy differences.

Regarding the impact of windows on stopping rules, overlapping windows make successive stopping checks dependent. Serial correlation within a window can affect the coverage of the nominal 95% t-based bounds, so CoStay uses them as heuristic stopping margins and evaluates the stopping policy empirically.

Table 2 Paired question-level comparisons

BaselineAccuracy change(pp)Token cost reduction(%)
SAFE−1.06[−2.29,+0.15]45.85[44.29,47.40]
UniTE−1.54[−3.02,−0.08]46.91[44.95,48.80]
CoSD+0.71[−0.83,+2.27]40.06[38.24,41.86]

A4: We extended inference efficiency experiments.

Thank you for this suggestion. We extended the end-to-end inference efficiency evaluation (Table 3). The results show that collaborative inference introduces additional overhead compared with standalone inference. Nevertheless, CoStay consistently achieves the highest throughput among the collaborative methods. Moreover, its throughput advantage over SAFE becomes more pronounced as the number of collaborators increases. For example, with Q25 as the base model, CoStay improves throughput over SAFE by 33.8%, 52.0%, and 63.3% with one, two, and three collaborators, respectively. These results demonstrate that CoStay effectively mitigates the inference overhead of multi-collaborator generation through dynamic collaborator deactivation.

Table 3 Comparison of Inference Efficiency

ConfigurationBase-onlyUniTESAFECoSDCoStay
HY4→{Q34}21.0410.5515.3813.5218.71
LM8→{HY4}30.0612.4117.9317.3621.14
Q25→{Q34}27.7911.3916.6016.6422.21
HY4→{LM8,Q34}21.047.2312.64—16.88
LM8→{Q25,HY4}30.068.2313.89—18.52
Q25→{HY4,Q34}27.797.6112.30—18.68
HY4→{Q25,LM8,Q34}21.045.289.26—15.14
Q25→{HY4,LM8,Q34}27.795.779.97—16.28

Thank you for your reply. I still feel that the reply about W1 doesn't address my concerns. I therefore keep my score.

AuthorsOpenReview ↗

Thank you. To clarify CoStay's design rationale and its empirical basis, we organize our response around three questions. We hope our explanations address your concerns. (Note: Q1 and Q2 are addressed in one comment, while Q3 is addressed in a separate comment.)

Thank you for your feedback. We address your concerns by discussing three questions concerning CoStay's design rationale and empirical justification.

Q1: Why does CoStay measure changes in the first disagreement position, rather than considering the aggregated distribution?

Q2: What motivates the lower- and upper-threshold stopping rules, and how should their underlying assumptions be interpreted?

Q3: What empirical evidence supports the effectiveness of the lower- and upper-threshold exits, respectively?

Q1: Why does CoStay measure changes in the first disagreement position, rather than considering the aggregated distribution?

CoStay measures changes in the first disagreement position because this position directly determines the accepted draft prefix and, consequently, the portion of the draft that needs to be regenerated. Thus, tracking its changes provides a direct indication of how collaborator removal affects draft acceptance.

There are two considerations behind this choice. First, unlike a general measure of prediction differences, the first disagreement position is directly related to the speculative decoding process: a shift in this position changes how much of the current draft can be retained. Second, changes in the aggregated distribution indicate prediction differences but cannot determine which alternative will lead to a better final answer, whose quality depends on subsequent generation. For example, after removing a collaborator, the aggregated distribution may shift from b4b_4 to c4c_4, but this shift alone cannot establish which token will produce a better final response.

We acknowledge that this positional metric does not capture all effects of collaborator removal on aggregated predictions. In particular, the replacement token may change even when the first disagreement position remains unchanged. Therefore, a small positional change only indicates a limited impact on this specific statistic, rather than proving that the collaborator makes no contribution. We adopt this metric as a lightweight indicator of the impact on draft acceptance, while leaving the integration of distributional information into a more comprehensive influence metric for future work.

Q2: What motivates the lower- and upper-threshold stopping rules, and how should their underlying assumptions be interpreted?

Due to the inherent uncertainty of LLM reasoning, the quality of an individual token-generation decision cannot be reliably determined at the time it is made. Moreover, a collaborator's relative capability is unknown at inference time, making it difficult to predict whether continued collaboration will improve or impair answer quality. Therefore, due to the inherent limitations of LLM reasoning process, disagreement intensity can only reflect the extent to which a collaborator influences the base LLM, rather than whether such influence is beneficial or detrimental.

Nevertheless, inspired by the role of disagreement in human collaboration, we adopt a heuristic strategy: persistently low disagreement indicates limited collaborative contribution, whereas persistently high disagreement suggests substantial influence that warrants controlled termination to balance costs and benefits. Accordingly, CoStay employs two thresholds to balance avoiding unnecessary collaboration overhead against the risk of prematurely terminating valuable guidance.

The lower-threshold rule is motivated by computational efficiency. When the upper confidence bound of positional influence within the sliding window falls below the lower threshold, it indicates that removing the collaborator has had consistently limited impact on the measured disagreement positions. CoStay uses this as a heuristic basis for terminating collaboration to reduce potentially unnecessary overhead.

The upper-threshold rule is motivated by limiting prolonged collaboration, rather than identifying harmful collaborators. High influence alone does not establish that a collaborator is harmful: persistent influence may result from either useful guidance or potentially disruptive interference. Table 1 illustrates that collaboration can yield substantially different accuracy effects depending on the relative capabilities of the base and collaborator LLMs. For example, collaboration with a stronger LLM improves accuracy by 8.33 percentage points, whereas collaboration with a weaker LLM reduces it by 2.17 percentage points. These results motivate caution when terminating high-influence collaborators, as doing so may interrupt useful guidance.

Table 1. Accuracy changes under different settings in SAFE (ICLR 2026).

CategoryBase LLM → Collaborator LLMGain (pp)
A stronger collaboratorHunyuan-4B → Llama-3.1-8B+8.33
A weaker collaboratorLlama-3.1-8B → Hunyuan-4B-2.17
AuthorsOpenReview ↗

The Response of Q3

Q3: What empirical evidence supports the effectiveness of the lower- and upper-threshold exits, respectively?

To examine how the disagreement signal behaves in CoStay, we paired its outputs with SAFE across eight base-model–collaborator configurations. The results show that the lower-threshold exit is effective in reducing potentially unnecessary collaboration while largely preserving answer correctness, whereas the upper-threshold exit provides an additional means of limiting prolonged collaboration, albeit with a more pronounced and configuration-dependent accuracy impact. The corresponding evidence is presented below.

Lower-threshold exits generally preserved answer correctness. Among the 1,683 questions that triggered a lower-threshold exit, 1,586 (94.23%) retained the same correctness status, 44 changed from incorrect to correct, and 53 changed from correct to incorrect. The net accuracy difference within this group was −0.53 percentage points. Across the seven heterogeneous model pairs, 986 of 1,083 instances (91.04%) retained their correctness status. These results support using the lower threshold as a heuristic for stopping collaboration with a small aggregate accuracy loss in the tested configurations.

Upper-threshold exits represent an empirical trade-off between cost and accuracy, rather than a mechanism capable of reliably identifying harmful collaborators. Among the 1,671 questions triggering an upper-threshold exit, 1,364 (81.63%) retained the same correctness status, while 134 changed from incorrect to correct and 173 from correct to incorrect. This corresponds to a net accuracy decrease of 2.33 percentage points. A collaborator that performs better than the base model on the task may repeatedly correct its draft, in which case an upper-threshold exit can cut short useful guidance. A weaker collaborator may repeatedly steer the draft toward errors, in which case stopping collaboration can reduce interference. Our paired results show both patterns. For HY4→LM8, where the collaborator is stronger, termination changes 6 answers from incorrect to correct (W→C), but also changes 10 from correct to incorrect (C→W). Conversely, for LM8→HY4, where the collaborator is weaker, termination changes 18 answers from incorrect to correct and 8 from correct to incorrect. These results suggest that high influence can reflect either useful guidance from a stronger collaborator or potentially disruptive interference from a weaker one.

Table 2 Answer-correctness transitions under different CoStay exit conditions relative to SAFE

Base Model → CollaboratorLower-Threshold ExitUpper-Threshold ExitNo Exit
HY4 → LM8197 (7 / 12 / 178)101 (6 / 10 / 85)302 (0 / 0 / 302)
HY4 → Q3496 (5 / 7 / 84)336 (18 / 48 / 270)168 (0 / 0 / 168)
LM8 → HY4104 (5 / 5 / 94)177 (18 / 8 / 151)319 (0 / 0 / 319)
Q25 → HY4104 (4 / 4 / 96)226 (22 / 13 / 191)270 (0 / 0 / 270)
LM8 → Q3457 (4 / 2 / 51)421 (32 / 55 / 334)122 (0 / 0 / 122)
Q34 → LM8454 (18 / 20 / 416)8 (0 / 0 / 8)138 (0 / 0 / 138)
Q25 → Q3471 (1 / 3 / 67)402 (38 / 39 / 325)127 (0 / 0 / 127)
Q25 → Q25600 (0 / 0 / 600)0 (0 / 0 / 0)0 (0 / 0 / 0)
Overall1,683 (44 / 53 / 1,586)1,671 (134 / 173 / 1,364)1,446 (0 / 0 / 1,446)

Each cell reports the total number of examples, followed by counts in the order (W→C / C→W / Same). W→C denotes examples answered incorrectly by SAFE and correctly by CoStay. C→W denotes examples answered correctly by SAFE and incorrectly by CoStay. Same includes examples that both methods answered correctly or both answered incorrectly.

Overall, most exits preserve answer correctness. The lower-threshold rule removes potentially redundant collaborators while maintaining the original correctness status in 94.23% of cases, with only a 0.53-percentage-point net accuracy decrease. For the upper-threshold rule, although high influence may reflect either beneficial or harmful contributions, retaining this rule provides opportunities to reduce collaboration overhead; otherwise, the corresponding 1,671 (34.8%) cases may incur additional collaboration costs due to prolonged participation. Moreover, the weaker base LLMs have more to gain from collaboration, motivating a more conservative termination policy. Therefore, we retain the upper threshold to balance inference efficiency and answer accuracy.

Reviewer_ix17OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoStay, a dynamic collaborator elimination method for token level collaboration systems.

The terminology is that there is a base LLM (drafter) that generates draft text and then there are multiple collaborators that are verifying and correcting the generated text. Existing methods like SAFE and CoSD keep the collaborators running throughout the generation period, that may lead to higher costs even if their contributions are marginal.

CoStay proposes to continuously evaluate the value of a collaborator by looking at its impact on the disagreement intensity of the collective decision over a sliding window and remove it if it lies in the extremes (too low or too high). It does this by using the position of the first rejected draft token and uses a leave one out comparison to estimate each collaborator's marginal influence on the collective decision. Across various experiments, CoStay helps to reduce token cost of collaborators by 44% without losing much accuracy (99.12%).

Critical strengths
  • Approach is simple without the need for additional LLM inference passes. All the data necessary for counterfactual disagreement and stopping is alreay present from the existing collaborator passes.
Critical weaknesses
  • The disagreement metric defined measures the absolute difference between disagreement intensities and not necessarily based on correctness or quality. For e.g. say the generation involved heterogenous topics like plain text and law and one of the collaborators is significantly good at law compared to base LLM. Eventually this specialized collaborator might be removed. And This concern is particularly relevant given that CoStay achieves 3.66 pp lower accuracy than SAFE in the stronger collaborator setting, suggesting that useful collaboration may be lost through early stopping. Also, from the Table 3 the weaker collaborators have a higher active ratio (77.91%) than stronger collaborators (70.17%), which I am not sure is intended for useful collaboration.

  • The paper presents leave one out as leave one out as main component of CoStay framework, but all related muti-collaborator experiments are limited in scope (2 to 3 collaborators) delegated to Appendix. Leave one out in case of single collaborator amounts to the disagreement intensity and may not add any additional value.

Questions for authors
  • Can the authors compare CoStay against simpler stopping mechanisms applied to SAFE, such as fixed-round stopping and moving-average disagreement thresholding, under the same matched collaborator token budgets?

  • Can authors provide ablation demonstrating the value of leave one out estimation in multi-collaborator settings?

  • Can the authors provide evidence on why disagreement intensity during generation corresponds to overall more useful collaboration and a better quality of generated content.

6 comments
AuthorsOpenReview ↗

We added fixed-cutoff and sliding-window mean stopping baselines, as well as leave-one-out ablations in multi-collaborator settings. We also clarified what disagreement intensity captures about generation decisions and discussed its limitations as a proxy for generation quality. The revised manuscript added the relevant content and marked it in blue. We hope our response will address the reviewers' concerns.

A1: We added fixed-round and mean-based stopping ablations.

Thank you for this helpful suggestion. We added two ablation methods to the revised manuscript: CoStay-W(43), which uses a fixed stopping point, and CoStay-Mean, which replaces LCB/UCB with the sliding-window mean while keeping all other settings unchanged. CoStay-W uses a fixed-round of 43 calls to match CoStay's average budget of 42.74 calls. Its actual average is slightly lower because some tasks finish before reaching 43 calls. Only CoStay-W allows a predefined cutoff; CoStay and CoStay-Mean do not, as their stopping decisions are determined automatically by their respective criteria.

In the single-collaborator setting (Table 1), CoStay achieves 71.58% accuracy, outperforming CoStay-W(43) by 0.37 percentage points and CoStay-Mean by 0.89 percentage points. In multi-collaborator settings (Appendix H), CoStay improves average accuracy over both CoStay-Mean and CoStay-W(43) across the three two-collaborator configurations, while incurring a comparable token cost to CoStay-W(43). These results suggest that CoStay more effectively identifies when to deactivate collaborators, thereby better preserving the benefits of collaboration. Detailed results are provided in Appendix H.

Table 1: Overall performance across all single-collaborator configurations

MethodAccuracy↑Calls↓Activ Ratio↓Token Cost↓
CoStay-W(10)69.83%10.0018.25%49.91
CoStay-W(43)71.21%41.9065.94%208.88
CoStay-Mean70.69%27.3443.90%136.23
CoStay71.58%42.7462.10%212.70

A2/W2: We added multi-collaborator LOO ablations.

Thank you for this question. We agree that LOO is particularly valuable for individual influence attribution in multi-collaborator settings, while also supporting single-collaborator settings. To verify this, we add two ablation variants that replace LOO-based influence estimation with alternative disagreement measures. The three differ as follows

CoStay: dj,r=∣Ir(Sr)−Ir!(Sr∖Mj)∣d_{j,r}=\left|I_r(S_r)-I_r\\!\left(S_r\setminus\\{M_j\\}\right)\right|, Sr=M1,…,Mj,…S_r = \\{M_1,\dots,M_j,\dots\\}

CoStay-Local: dj,r=Ir(Mj)d_{j,r}= I_r({M_j})

CoStay-Global: dj,r=Ir(Sr)d_{j,r}=I_r(S_r)

As defined above, CoStay also supports a single-collaborator setting (equivalent to CoStay-Local).

As shown in Table 2, across the three two-collaborator configurations, CoStay reduces average token cost by 22.0% relative to CoStay-Local and by 23.5% relative to CoStay-Global, while maintaining comparable accuracy. Although CoStay-Local achieves slightly higher accuracy (72.61% vs. 71.94%), this comes at a higher computational cost. This suggests that LOO-based influence estimation provides a more favorable trade-off between accuracy and collaboration cost by better guiding collaborator deactivation. Detailed results across different configurations are provided in Appendix I.

Table 2: LOO ablation averaged over two-collaborator settings.

MethodAccuracy↑Calls↓Active Ratio↓Token Cost↓
CoStay-Local72.61%106.1771.05%527.92
CoStay-Global71.72%101.5067.37%538.54
CoStay71.94%82.6357.27%411.81

A3/W1: We clarified the role of disagreement intensity.

We apologize for the lack of clarity in our presentation, which may have caused this misunderstanding. Due to the inherent uncertainty of LLM reasoning, the quality of an individual token-generation decision cannot be reliably determined at the time it is made. This limitation is inherent to the generation process rather than specific to disagreement. Therefore, disagreement intensity reflects the magnitude of a collaborator's influence on the base LLM, but does not indicate whether that influence is beneficial or detrimental. Our goal is to achieve a balance between cost and accuracy.

Our experiments (Section 3.2, page 5) show that stronger base LLMs are more resilient to interference. Based on this observation, CoStay adopts a relatively high upper threshold to balance collaboration opportunities for weaker base LLMs against unnecessary interference for stronger ones. This design allows weaker base LLMs to benefit from collaboration while still providing stronger base LLMs with opportunities to deactivate collaborators, which also explains the accuracy loss observed with strong collaborators, as discussed in W1.

Compared with Base-only, CoStay improves accuracy from 69.25% to 71.58%. Compared with SAFE, it reduces token cost by 45.8%, with only a 1.07 pp accuracy decrease. Moreover, Figure 4 (Appendix C, page 15) shows that higher thresholds increase cost but do not necessarily improve accuracy, supporting the need for a chosen upper threshold. Overall, these results support CoStay's disagreement-based design for balancing accuracy and cost through adaptive collaboration. We acknowledge that the current design is not optimal and will further improve it in future work.

AuthorsOpenReview ↗

Additional notes regarding question Q3

Q3: Can the authors provide evidence on why disagreement intensity during generation corresponds to overall more useful collaboration and a better quality of generated content.

Thank you for raising this important question. CoStay uses disagreement to measure a collaborator's marginal influence on the generation process, rather than directly assessing the quality of its contribution. The rationale for the two thresholds is as follows.

Lower threshold. Persistently low disagreement indicates that removing a collaborator has little effect on draft acceptance. This suggests that the collaborator rarely provides differentiated information that substantially changes the current generation process, and its marginal contribution may therefore be limited. In this case, the additional computational overhead of continued collaboration may outweigh its potential benefits. Thus, the lower threshold provides a heuristic basis for terminating collaborations with consistently limited influence.

Upper threshold. Persistently high disagreement indicates that a collaborator has a substantial influence on generation, but this influence may be either beneficial or harmful. Ideally, harmful interference should be eliminated, whereas beneficial guidance should be retained. However, the actual effect of a collaborator's contribution depends on subsequent generation and the quality of the final answer, making it difficult to determine its value accurately in advance. These competing objectives create a trade-off between eliminating potentially harmful interference and preserving useful guidance. To balance these considerations, we introduce an upper threshold to limit prolonged collaboration, thereby managing the potential benefits of continued collaboration against its additional computational overhead.

The answer-correctness transitions in Table 3 provide empirical support for this design. For the lower-threshold exits, 1,586 out of 1,683 examples (94.24%) retained the same correctness status, with a net accuracy decrease of only 0.53 percentage points. This suggests that terminating collaborations with consistently limited influence generally incurs a small accuracy cost. For the upper-threshold exits, 1,364 out of 1,671 examples (81.63%) retained the same correctness status, while the remaining examples included both improvements (134 W→C) and degradations (173 C→W). These mixed outcomes confirm that high-influence collaboration can have either beneficial or harmful effects, supporting our decision to use the upper threshold as a compromise between preserving potentially useful guidance and limiting prolonged collaboration overhead.

Table 3 Answer-correctness transitions under different CoStay exit conditions relative to SAFE (ICLR 2026)

Base Model → CollaboratorLower-Threshold ExitUpper-Threshold ExitNo Exit
HY4 → LM8197 (7 / 12 / 178)101 (6 / 10 / 85)302 (0 / 0 / 302)
HY4 → Q3496 (5 / 7 / 84)336 (18 / 48 / 270)168 (0 / 0 / 168)
LM8 → HY4104 (5 / 5 / 94)177 (18 / 8 / 151)319 (0 / 0 / 319)
Q25 → HY4104 (4 / 4 / 96)226 (22 / 13 / 191)270 (0 / 0 / 270)
LM8 → Q3457 (4 / 2 / 51)421 (32 / 55 / 334)122 (0 / 0 / 122)
Q34 → LM8454 (18 / 20 / 416)8 (0 / 0 / 8)138 (0 / 0 / 138)
Q25 → Q3471 (1 / 3 / 67)402 (38 / 39 / 325)127 (0 / 0 / 127)
Q25 → Q25600 (0 / 0 / 600)0 (0 / 0 / 0)0 (0 / 0 / 0)
Overall1,683 (44 / 53 / 1,586)1,671 (134 / 173 / 1,364)1,446 (0 / 0 / 1,446)

Each cell reports the total number of examples, followed by counts in the order (W→C / C→W / Same). W→C denotes examples answered incorrectly by SAFE and correctly by CoStay. C→W denotes examples answered correctly by SAFE and incorrectly by CoStay. Same includes examples that both methods answered correctly or both answered incorrectly.

Thank you for the additional ablations and the answer-correctness transitions. The additional results seem to support my main concern. At a given matched budget, the fixed cutoff CoStay-W(43) comes close to CoStay overall and does slightly better with stronger collaborators at a lower cost, so most of the gain seems to come from stopping early rather than from the disagreement signal. The LOO ablation reads as a cost vs accuracy trade off against CoStay-Local rather than a clear win. I'm inclined to keep my current scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.