- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CoStay a method for dynamic collaborator pruning for speculative token-level collaboration. Existing methods today keep every collaborator active for the whole generation, which is costly. The author's propose a training-free method that decides when a collaborator should stop. It measures how much each collaborator changes draft rejection, tracks this over a sliding window with confidence bounds, and removes collaborators whose influence stays very low or very high.
Critical strengths
[S1] The efficiency gains are substantial. Collaborator token cost drops by about 46% compared with SAFE for about 1% decrease in accuracy, and throughput improves on all three tested configurations.
[S2] The analysis by collaborator strength is good. With identical models, CoStay cuts cost by over 80% with no accuracy loss, and with a weaker collaborator it limits the damage better than other methods. This comes at the trade-off of some accuracy with stronger collaborators, which the authors attribute to removing some useful participation
[S3] The bounded distinctiveness stopping idea is intuitive and well grounded. It removes collaborators that add almost nothing and those that intervene too much, and it builds on the ensemble view that diversity helps only when it is balanced
[S4] The comparison of CoStay with the fixed-cutoff version CoStay-W is good. The fixed cutoff is cheapest but loses clearly more accuracy with stronger collaborators. The authors attribute this to stopping useful collaboration without regard to what it contributes.
Critical weaknesses
[W1] The leave-one-out comparison is presented as a key part of the method, but all main experiments use only one collaborator. The cases where leave-one-out actually matters, with several collaborators, appear only in Table 8. It could be useful to move this into the main text.
[W2] The comparison of CoStay with CoStay-W, the fixed cutoff uses far fewer collaborator calls than CoStay (Table 2). A comparison at a similar cost would better show how much of the saving comes from the disagreement signal itself.
[W3] Once a collaborator is stopped, it never rejoins later rounds. A collaborator that was unhelpful early in an answer might still be useful later, for example in a later reasoning step, and it could be useful to mention or explore this as future works.
[W4] Because the stopping rule waits for a full window of 10 rounds, a collaborator can never be stopped before round 10. The window size itself is fixed rather than studied. It could be useful to discuss this design choice.
Questions for authors
Please address weaknesses.
We summarized the multi-collaborator results in the main text and added a fixed-cutoff comparison at a similar token cost. We also explained how the 10-round window affects the earliest stopping decision and discussed collaborator reactivation as future work. These changes have significantly improved the quality of the manuscript. The revised manuscript added the relevant content and marked it in blue. We hope they address the reviewers' concerns.
A1 We added multi-collaborator results to the main text.
Thank you for this valuable suggestion. Leave-one-out comparison is applicable to both single- and multiple-collaborator settings. Due to the page limit, we retain the detailed results in Table 8 in the appendix, while clarifying its role in the main text to better highlight its effectiveness in multi-collaborator scenarios. For instance, we added the following description to the main text:
We further evaluate CoStay in multi-collaborator settings to examine its scalability and the effectiveness of leave-one-out-based collaborator selection. The results show that adding more collaborators does not necessarily yield proportional accuracy gains, despite introducing additional inference overhead. For the three two-collaborator configurations, CoStay achieves an average accuracy of 71.94%, improving upon Base-only by 3.55 percentage points, while reducing collaborator token costs by 65.32% and 56.74% compared with SAFE and UniTE, respectively. Moreover, the benefit of additional collaborators is non-monotonic: adding HY4 to Q25$\rightarrow$Q34 reduces accuracy from 73.83% to 72.17%, while introducing another collaborator further increases token cost without improving accuracy. These results suggest that increasing the number of collaborators incurs additional inference costs, but does not guarantee commensurate accuracy improvements. Thus, using a single collaborator may offer a more favorable trade-off when its contribution is sufficient.
A2 We added cost-matched fixed-cutoff experiments.
Thank you for this valuable suggestion. We added the method to the revised manuscript: CoStay-W(43), which uses a fixed stopping point. Since CoStay requires an average of 42.74 collaborator calls, we set the fixed cutoff of CoStay-W to 43 to ensure a comparable collaboration budget. Its actual average number of calls is slightly lower because some tasks terminate before reaching the cutoff.
Unlike CoStay, a fixed-cutoff strategy requires the stopping budget to be manually specified in advance. A smaller cutoff may reduce computational cost but prematurely terminate useful collaboration (such as CoStay-W(10)), whereas a larger cutoff incurs more cost without necessarily improving accuracy. In contrast, CoStay adapts the stopping time to the observed disagreement. Experimental results show that CoStay achieves higher accuracy than CoStay-W(43) under a comparable collaboration budget, demonstrating the advantage of disagreement-guided adaptive deactivation over a manually predefined stopping point.
Table 1: Overall performance across all single-collaborator configurations
A3 We included collaborator reactivation in future work.
Thank you for this helpful suggestion. We have acknowledged this limitation in the revised manuscript and identified adaptive collaboration as a promising direction for future work. The revised text is as follows:
Future work will explore adaptive collaboration mechanisms to address these limitations. In particular, dynamically assigning non-uniform weights according to the observed reliability of each LLM and enabling the reactivation of previously stopped collaborators could better exploit complementary capabilities across different reasoning stages.
A4 We added a discussion on the window-size selection.
Thank you for this insightful comment. The full observation window is required to avoid premature stopping decisions based on insufficient observations or transient fluctuations. A smaller window enables faster responses but is more sensitive to short-term fluctuations, whereas a larger window provides more historical information at the cost of delayed stopping decisions and potentially unnecessary collaboration overhead.
In this work, we set the window size to W=10 as a reasonable trade-off between decision stability and responsiveness. Since the window size and the upper and lower thresholds are interdependent, jointly optimizing these parameters would substantially expand the search space and increase the experimental cost. More importantly, our primary focus is to evaluate the effectiveness of the proposed mechanism rather than extensively optimize its window size.