acceptodds
Withdrawn from ICLR 2027

STAR: Trajectory-Aware Sentence Routing for Efficient Multi-LLM Ensemble

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-LLM ensembles exploit the complementary capabilities of heterogeneous LLMs to improve inference without further increasing model scale. However, existing ensemble methods incur substantial computational waste by invoking multiple models at every generation step, while their reliance on local signals can result in suboptimal reasoning trajectories. In this work, we find that multi-LLM integration is sparse yet consequential at the sentence level: semantic disagreement arises only at a few critical states, while decisions at these states can shape subsequent reasoning trajectories beyond what local signals capture. To exploit this, we propose STAR, a sentence-level trajectory-aware routing framework that triggers multi-LLM integration only when needed and routes candidates according to their long-term trajectory value. STAR combines Sentence-level Disagreement Prediction (SDP) to predict cross-model semantic disagreement before candidate generation with Tree-Advantage Routing Optimization (TARO) to evaluate alternative trajectories and optimize routing using counterfactual advantages. Extensive experiments show that STAR achieves state-of-the-art accuracy among multi-LLM ensemble methods while reducing inference cost by up to 88.04%. Our code is available in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-LLM Ensemble, Sentence-Level Routing, Semantic Disagreement, Trajectory-Aware Routing, LLM Routing, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_NyPHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAR, a trajectory-aware sentence-level routing framework for efficient multi-LLM ensembles. The method addresses the computational redundancy of frequent multi-model collaboration and the limitations of local candidate selection. STAR introduces two key components: Sentence-level Disagreement Prediction (SDP), which identifies when multi-model collaboration is necessary, and Tree-Advantage Routing Optimization (TARO), which uses trajectory-level rewards and counterfactual advantages to optimize candidate selection. Experiments across mathematical reasoning, code generation, and general reasoning benchmarks demonstrate that STAR achieves superior average accuracy while significantly reducing inference costs compared with existing ensemble methods. Ablation studies and additional analyses further support the effectiveness and efficiency of the proposed framework.

Critical strengths

1. Clear Motivation and Novel Design: The paper addresses computational inefficiency in multi-LLM ensembles through a well-motivated combination of disagreement prediction and trajectory-aware routing.

2. Strong Experimental Results: Extensive experiments demonstrate improved accuracy and reduced inference costs, supported by comprehensive ablation studies and efficiency analyses.

Critical weaknesses
  1. Unclear Data Splits and Inconsistent Accuracy Reporting: The paper does not clearly explain whether the training data for SDP and TARO are strictly separated from the evaluation benchmarks, raising potential data leakage concerns. Moreover, some reported accuracy values appear inconsistent with the dataset sizes under single-run evaluation. For example, MATH500 contains 500 samples, meaning accuracy should change in increments of 0.2%, yet Table 1 reports 56.69%. Similarly, the reported 69.80% on HumanEval (164 samples) does not correspond to an integer number of correct predictions. The authors should clarify whether these results are averaged over multiple runs and provide details of the evaluation protocol.

  2. Questionable Assumption and Insufficient Evaluation of SDP: The method assumes that semantic disagreement indicates when multi-LLM collaboration is necessary. However, models may agree on incorrect reasoning or disagree while producing equally valid solutions. Therefore, semantic disagreement does not necessarily reflect the actual benefit of collaboration. Furthermore, the paper only reports the agreement between LLM-generated and human annotations and the overall ensemble activation rate, without directly evaluating SDP's precision, recall, F1-score, or false negative rate. These metrics are important for validating its ability to identify critical states.

  3. Limitations of TARO and Insufficient Justification of Its Design: The proposed max-value backup (Eq. 7) may produce overly optimistic value estimates, as a candidate is considered valuable if only one of its downstream trajectories achieves a high reward. Additionally, the counterfactual advantage (Eq. 8) may provide weak or zero learning signals when different candidate branches have similar returns. Although some ablations are provided, comparisons with simpler alternatives, such as ranking-based objectives or advantage-weighted classification, are missing.

  4. Potential Training–Inference Distribution Mismatch: SDP is trained using sentence-level states collected along trajectories generated by the base model (Algorithm 1). However, during inference, the context may contain sentences selected from different LLMs (Algorithm 3). This creates a potential distribution shift between training and inference states, which could affect disagreement prediction and routing reliability. The paper does not investigate this issue or provide robustness analysis under mixed-model trajectories.

  5. Incomplete Cost and Statistical Analysis: Although the paper provides extensive inference efficiency comparisons, it does not quantify the substantial offline costs associated with LLM-based semantic annotation, multi-model candidate generation, and tree rollouts. These costs are important for assessing the overall economic benefits of STAR. In addition, the paper lacks results across multiple random seeds, standard deviations, or confidence intervals, making it difficult to assess the statistical reliability of the reported improvements.

Questions for authors
  1. Clarification on Data Splits and Accuracy Reporting: Could the authors clarify whether the original questions used for SDP and TARO training are strictly separated from the evaluation benchmarks? Additionally, how are the accuracy values in Table 1 calculated? For example, MATH500 contains 500 samples, so accuracy should change in increments of 0.2% under single-run evaluation, yet 56.69% is reported. Similarly, 69.80% on HumanEval (164 samples) does not correspond to an integer number of correct predictions. Are these results averaged over multiple runs? If so, how many runs were conducted?

  2. Clarification on SDP Reliability and Collaboration Necessity: How do the authors validate that semantic disagreement actually identifies states where multi-LLM collaboration improves final task accuracy? Since models may agree on incorrect reasoning, could the authors report the precision, recall, and false negative rate of SDP, particularly on mixed-model trajectories during inference? These results would help determine whether the proposed disagreement criterion reliably identifies critical decision points.

Reviewer_zyRROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

STAR aims to reduce inference cost when combining heterogeneous language models. A sentence disagreement predictor decides when to invoke multiple models. The routing policy is trained with offline trees and counterfactual advantages to account for later reasoning outcomes. Experiments on math, code and reasoning tasks report higher average accuracy and lower inference cost than the tested ensemble methods.

Critical strengths
  1. Reducing repeated model calls is a useful goal for practical deployment. The paper also considers how intermediate model choices affect the final answer.
  2. Offline search keeps trajectory exploration separate from deployment. The frozen language models and small routing modules offer a practical way to reuse existing models.
  3. The evaluation covers several task types. Latency, throughput and memory measurements provide useful evidence beyond task accuracy.
Critical weaknesses
  1. The methodological novelty appears somewhat incremental. R2R[1] learns semantic divergence signals for routing. AlphaRouter[2] combines token disagreement prediction with offline tree training and counterfactual advantages. Comparisons under compatible model settings are needed to establish the added value of sentence routing and the revised learning rules in STAR.
  2. Semantically different sentences may lead to similar subsequent reasoning. The paper should examine whether disagreement labels reliably identify where to branch during trajectory exploration.
  3. Punctuation and token limits may split a reasoning step, especially in code. The human evaluation lacks details about the annotators and annotation procedure, and does not assess whether these boundaries are suitable for routing.
  4. The 70B baseline scores well below the results in its official model card on several tasks [3]. The task prompts, decoding settings and answer extraction need clarification to explain these gaps.
  5. The accuracy and cost curves omit MATH500, HumanEval and the overall average. The sampling budgets and changes to the original baseline settings also need justification.
  6. All experiments use the same Gemma base model and candidate pool. Results with another configuration are needed to assess whether the gains extend beyond this setup.

[1] R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing. NeurIPS 2025. [2] AlphaRouter: Token-level Routing Between SLM and LLM with Reinforcement Learning and Tree Search. ICML 2026. [3] The Llama 3 Herd of Models. arXiv, 2024.

Questions for authors

see weakness.

Reviewer_Vur1OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes STAR, a sentence-level ensemble of multiple heterogeneous LLMs (Gemma-2-2B, Mistral-7B-v0.3, Llama-3.1-8B) that write a single response together. At each sentence boundary, a lightweight MLP (SDP) reads the base model’s last-layer hidden state and predicts whether the three models would write semantically different next sentences. If no disagreement is predicted, Gemma writes the next sentence alone. Otherwise, all three models generate a candidate, and a learned router (TARO) selects one.

The SDP use agreement labels from a GPT-5.5 judge. TARO is trained offline with PPO. They test their results on GSM8K, MATH500, MBPP, HumanEval and ARC-C, STAR reports 71.74% average accuracy at $0.39. That beats five ensemble baselines and Llama-3.1-70B-Instruct.

Critical strengths

The motivating observation is clear and useful. Only about 12.6% of sentence positions show cross-model semantic disagreement, so process-level ensembles that call every model at every step waste most of their compute.

Branching the tree only at disagreement states, with sibling-relative advantages, is a sensible credit-assignment design.

Critical weaknesses

The baseline numbers seems to be inconsistent with other paper. For Llama-3.1-70B-Instruct, Table 1 reports 75.38 on GSM8K, 66.41 on HumanEval and 82.15 on ARC-C. However, Meta reports 80.5 on HumanEval, 95.1 on GSM8K and 94.8 on ARC Challenge. Please report prompts, if there is any generation limits and answer-extraction rules.

When we always runs the router (w/o SDP in Fig 3) , the accuracy drops to 60%. This is surprising, as AdaFuse and CBS, which also ensemble at every step are about 75%. I am not sure if the router behaves poorly.

Questions for authors
  • Why do Llama-3.1-8B and 70B score 12–20 points below published numbers? What are the prompts, token limits and extraction rules?
  • How many runs underlie each number, and is there variance?
  • What changes in TARO without SDP, and why does running the router at every step cost about 11 points?
Reviewer_GEkcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STAR, which decomposes sentence-level multi-model ensembling into two decisions: when to invoke multiple models and how to select among candidate sentences. SDP predicts cross-model semantic disagreement from a small model's hidden states, while TARO trains a router with offline trajectory search so that candidate selection accounts for downstream task outcomes. The overall idea is clear, and the reported accuracy and inference-cost improvements are attractive. However, important details about the data split, result aggregation, and validation of the central components remain unclear.

Critical strengths

The problem is practically relevant and the motivation is reasonable. The paper observes that only about 12.6% of sentence positions exhibit cross-model semantic disagreement in its sampled analysis. This observation provides a plausible motivation for avoiding unnecessary model calls through selective collaboration. The method has a clear division of responsibilities and addresses both quality and efficiency. SDP decides whether collaboration should be activated, whereas TARO selects the candidate. Tree search is performed during offline training; at inference time, the trained predictor and router are used, and multiple model candidates are generated only when collaboration is triggered. This design has the potential to benefit from trajectory-search supervision while keeping online inference overhead under control. The reported results are strong and the experimental coverage is broad. In Table 1, STAR improves average accuracy by 3.67 percentage points over AdaFuse, the strongest average-performing ensemble baseline, while the estimated inference cost decreases by about 48% ($0.75 to $0.39). The experiments cover mathematics, code, and general reasoning tasks, and include out-of-distribution tests, replacement of the semantic annotator, parameter analyses, and system-efficiency comparisons.

Critical weaknesses

(1) The relationship between training and test data, and the way accuracy is aggregated, is not sufficiently specified. Appendix B.4.2 gives split proportions for 20,000 sentence-level samples, but it does not state whether the split was first performed at the original-question level and sentence states were collected afterward. It is also unclear which questions were used to train TARO and how those questions relate to the evaluation data in Table 1. Consequently, I cannot determine whether different states or trajectories from the same question cross the training and test boundaries. This is a missing methodological detail rather than evidence of data leakage, but it directly affects how the reported generalization should be interpreted. Some reported values also do not appear to correspond to a single per-question evaluation under the stated sample counts. For 500 MATH500 questions, accuracy should change in increments of 0.2 percentage points, whereas Table 1 reports 56.69%. Likewise, 69.80% on HumanEval does not directly correspond to an integer number of passed problems out of 164 when there is one output per problem. These values may result from averaging across multiple runs or from a multi-sample estimate, but the paper does not say so. The paper should explicitly state the question-level data split, the relationship between the questions used to train SDP and TARO and the final evaluation questions, the calculation procedures for accuracy and pass@1, the number of runs, and the resulting variability. (2) The current analysis does not establish that SDP identifies the positions where collaboration is most useful. Appendix C.5 uses the similarity between the collaboration-trigger rate and the true disagreement rate as evidence for SDP. Similar aggregate rates, however, do not show that the predictor identifies the same positions. The reported 92.2% human agreement measures the semantic labels, not SDP's predictive performance; under class imbalance, overall agreement alone is also insufficient to assess the quality of disagreement detection. More importantly, semantic disagreement is not equivalent to collaboration benefit. Different reasoning paths may all lead to the correct answer, and several models may agree on the same wrong answer. The current evidence therefore does not distinguish gains caused by predicting semantic disagreement from gains caused by a generic sparse-calling policy. The authors should report disagreement-class precision and recall for SDP on independent test questions. They should also compare SDP with random triggering and a simple confidence-based trigger at the same collaboration rate or under the same inference budget. These comparisons would provide the most direct test of the paper's central motivation. (3) The necessity of TARO and the independent effects of the components are not adequately separated. The max-value backup records the best result found along a branch during search, but that value is not automatically the gain that the deployed router will realize when it follows the branch. This design may help discover promising paths, but the paper should explain how the router learns to reproduce their downstream benefits and why this more complex training target is preferable to simpler supervised learning. The current ablations also change multiple factors at once. When SDP is removed, TARO is retrained while the search budget remains fixed at B = 32. This may change both the branching density of the training tree and its effective search depth. Therefore, the roughly 11.5-percentage-point accuracy drop cannot be attributed entirely to the predictor. In addition, the experiment described as removing tree search also changes the selection rule to perplexity, so it does not fully isolate the effect of search from the effect of the selection criterion. At minimum, the paper should add a controlled comparison that keeps the trained router fixed and disables only SDP during inference. It should also compare TARO with a simple version that directly learns from supervised labels for the best candidate found by search. These controls would clarify whether the gains depend on the proposed design itself. (4) The comparison settings and cost accounting do not yet support the paper's broader claims. The paper reports that STAR outperforms a single 70B model, but the main evaluation does not sufficiently describe the prompt formats, generation lengths, and other settings used for the models. Based on the current PDF, it is not possible to determine whether the single-model baselines were evaluated under appropriate and consistent conditions. The paper also emphasizes the value of long-horizon trajectories, yet it does not directly compare TARO with the related tree-search ensemble methods discussed in the paper. This makes TARO's incremental contribution over prior approaches difficult to assess. The efficiency analysis benefits from reporting both cost and end-to-end latency. Nevertheless, the paper should clarify how prefix processing and cache overhead are counted, and how Table 2 matches different methods to the same 78.3% accuracy. Excluding offline annotation and search costs from inference cost is reasonable, but the resulting claims should be explicitly limited to inference-stage savings. The authors should provide the evaluation configurations for the key baselines, explain the accuracy-matching procedure in Table 2, and define the boundary of the cost accounting. Until these details are clarified, claims such as "outperforms a 70B model" and "achieves lower cost" should be limited to the model pool and experimental settings used in this paper.

Questions for authors

(1) Was the data split performed at the original-question level before collecting sentence states? Which questions were used to train SDP and TARO, and how do they overlap or differ from the questions used in the final evaluations? How exactly are accuracy and pass@1 computed, including the number of runs and the variability across runs? (2) On independent test questions, what are SDP's precision and recall for the disagreement class? Can the authors compare SDP with random triggering and a simple confidence trigger at the same collaboration rate or under the same inference budget? (3) Can the authors provide controlled ablations that keep a trained TARO router fixed while disabling only SDP at inference time? Can they compare TARO with a simpler supervised learner trained to select the best candidate found by search? How do the branching density and effective search depth change when SDP is removed while B = 32 is held fixed? (4) What prompt formats, generation lengths, and other evaluation settings are used for the key baselines, including the 70B single-model comparison? How does TARO compare directly with the related tree-search ensemble methods discussed in the paper? For Table 2, how are methods matched to 78.3% accuracy, and are prefix processing and cache overhead included in the reported cost and latency? Please also clarify that offline annotation and search costs are excluded from the inference-cost figures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.