- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a test-time method that combines responses obtained under alternative contexts. It constructs answer-support sequences and uses agreement and commitment to select and fuse evidence. The method is evaluated on in-context classification and generation tasks, as well as skill-augmented reasoning. Some experiments have been conducted.
Critical strengths
The tackled problem is well-motivated. Selecting a single demonstration count, ordering, or reasoning plan can discard useful evidence from other contexts. It provides a unified way to use these alternative responses across several task settings. This is the main strength for this paper. Another strength is that the paper reports ICL classification results for five models and skill-augmented reasoning results for two Qwen3 models. It also provides component analyses, budget curves, and inference-time measurements. These experiments make the proposal more credible than a result on one benchmark alone.
Critical weaknesses
- One major weakness is that the experiments do not sufficiently isolate the value of the consensus rule. Most baselines in Table 2 select or construct a context, whereas the proposed method obtains and aggregates predictions from multiple contexts. The paper does not compare AG+CM with simple aggregation—such as mean probability or voting—applied to the same saved predictions. Similarly, in the skill experiment, it uses diverse plan-conditioned responses while Self-Consistency samples under a fixed prompt; voting over its own response bank would be a more direct comparison. This matters because the observed gain may arise from response diversity or additional inference rather than the proposed consensus procedure. For context, Table 25 reports 71.92% accuracy at 16.06 seconds of inference per query for the proposed method on Llama 3 8B, versus 67.55% at 1.25 seconds for EPR; EPR also has a separate one-time preparation cost. A comparison using the same responses and a compute–accuracy analysis would substantially strengthen the central claim. The current analyses also do not directly isolate stable-position selection from choosing the final or highest-scoring position. Another major weakness is that the evidence for consistent gains is weaker than the headline averages suggest. On Llama 3 8B, the proposed method exceeds EPR’s twelve-task classification average by 4.37 points. The 29.84-point difference between these methods on CoLA contributes approximately 2.49 points to that average gap. Reporting CoLA’s standard MCC metric and an average excluding CoLA would clarify robustness; using accuracy consistently is not itself an error. The classification margin is also much smaller on Llama 3 70B (0.17 points over the strongest average baseline). In generation, Table 3 reports 56.0 for the proposed method versus 55.7 for EPR across six tasks, with no individual task on which it has the highest score. These results support the method’s competitiveness, but provide limited evidence for a consistent advantage across tasks and models.
Questions for authors
What is the performance for CoLA’s standard MCC metric?
3 comments
Author response to Reviewer T11b (2/2)
W3) Does stable-position selection help beyond choosing the highest-scoring position?
Across 12 tasks, CHORUS achieves the highest average score when AG and CM are combined with stable-position selection. The variant without stable-position selection uses the position with the highest class probability. Ordering selection remains unchanged in this comparison.
| Method | 12-task average score |
|---|---|
| CHORUS w/o stable-position selection | 66.56 |
| CHORUS | 70.20 |
We will incorporate these clarifications and results in the revised paper.
W4, Q) What is CoLA's performance in MCC, and how does the correction affect the averages?
After correcting the CoLA metric, CHORUS retains the highest 12-task average on each of the five models. We apologize for mixing up accuracy and MCC when reporting the original CoLA results. We now report MCC for every CoLA entry and have recalculated all 12-task averages accordingly. Each cell below shows CoLA MCC × 100 / 12-task average / 11-task average excluding CoLA.
| Method | Llama 3 8B | Qwen2.5 7B | Mistral 7B | Gemma 2 27B | Llama 3 70B |
|---|---|---|---|---|---|
| Random | 50.52 / 65.03 / 66.35 | 58.79 / 72.14 / 73.35 | 50.22 / 68.86 / 70.55 | 61.59 / 74.66 / 75.85 | 64.11 / 76.83 / 77.99 |
| TopK-BM25 | 54.75 / 66.96 / 68.07 | 63.15 / 73.60 / 74.55 | 52.70 / 70.08 / 71.66 | 62.35 / 75.15 / 76.31 | 65.85 / 78.15 / 79.26 |
| TopK-BERT | 62.01 / 66.94 / 67.38 | 64.77 / 74.00 / 74.84 | 55.41 / 69.87 / 71.19 | 68.59 / 75.60 / 76.23 | 67.82 / 77.81 / 78.72 |
| DPP-BERT | 57.69 / 67.01 / 67.86 | 63.55 / 74.38 / 75.37 | 56.11 / 69.90 / 71.16 | 67.56 / 74.89 / 75.56 | 69.61 / 78.86 / 79.70 |
| D.Va | 55.79 / 66.95 / 67.97 | 61.76 / 73.77 / 74.86 | 55.72 / 70.05 / 71.35 | 61.36 / 74.03 / 75.18 | 65.94 / 76.81 / 77.79 |
| EPR | 55.39 / 67.55 / 68.66 | 64.92 / 74.28 / 75.13 | 53.09 / 70.14 / 71.69 | 63.80 / 75.74 / 76.83 | 60.57 / 77.94 / 79.52 |
| TopicK | 51.27 / 66.63 / 68.02 | 62.47 / 72.25 / 73.14 | 56.77 / 68.61 / 69.69 | 55.34 / 72.72 / 74.30 | 63.81 / 76.44 / 77.58 |
| EASE | 55.37 / 65.16 / 66.05 | 61.52 / 71.72 / 72.65 | 52.53 / 68.58 / 70.04 | 55.22 / 73.51 / 75.17 | 62.76 / 74.19 / 75.22 |
| Se2 | 47.51 / 63.75 / 65.23 | 63.15 / 71.65 / 72.42 | 53.01 / 68.42 / 69.82 | 62.41 / 72.42 / 73.33 | 66.93 / 74.48 / 75.17 |
| SeDPO | 52.79 / 65.14 / 66.27 | 60.11 / 71.30 / 72.32 | 54.09 / 67.62 / 68.85 | 66.48 / 74.05 / 74.74 | 61.62 / 75.12 / 76.34 |
| DR-IKE | 41.93 / 60.66 / 62.37 | 59.97 / 69.47 / 70.33 | 50.62 / 66.84 / 68.32 | 52.10 / 68.02 / 69.46 | 61.30 / 72.31 / 73.31 |
| Adaptive-k | 51.40 / 57.57 / 58.13 | 60.78 / 69.70 / 70.51 | 51.57 / 67.42 / 68.86 | 64.38 / 70.43 / 70.98 | 63.67 / 72.69 / 73.52 |
| CHORUS | 64.57 / 70.20 / 70.71 | 63.15 / 74.88 / 75.94 | 57.52 / 71.09 / 72.32 | 68.84 / 76.36 / 77.04 | 70.05 / 79.03 / 79.85 |
After excluding CoLA, CHORUS still achieves the highest 11-task average on each of the five models. The strongest baseline varies by model.
W5) Is the advantage consistent across tasks and models, including Llama 3 70B and generation?
We agree that the gains over the strongest baseline are small on Llama 3 70B (DPP-BERT) and generation (EPR). However, the strongest baseline changes across models. For example, EPR scores higher on average than DPP-BERT on Llama 3 8B and Gemma 2 27B, while DPP-BERT scores higher on Llama 3 70B. In contrast, CHORUS achieves the highest average score on every evaluated model and setting. For each task, EPR uses a language model to label candidate demonstrations based on the probability it assigns to the correct answer. It then trains a separate retriever on those labels to select demonstrations for new queries. We hope the reviewer will consider that CHORUS achieves these results with the same consensus procedure across ICL classification, generation, and skill-augmented reasoning, without additional training.
As the discussion period draws to a close, we would be grateful if you could take another look at our rebuttal and revised manuscript. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.
Author response to Reviewer T11b (1/2)
We are grateful for your thoughtful feedback. Our revisions to the paper will reflect the points discussed below. Following your feedback, we now evaluate CoLA with MCC, its standard evaluation metric, and have recalculated the affected averages.
W1) How does CHORUS compare with simple aggregation in ICL and skill-augmented reasoning?
In ICL, we applied majority voting and probability averaging under the same inference budget as CHORUS (Llama 3 8B, 12 tasks).
In skill-augmented reasoning, CHORUS and the consistency baselines use the same set of reasoning-plan prompts. We apologize that our original description did not make this clear. The main-text results below use a maximum budget of six responses and report accuracy (%) as mean ± SD over three runs.
We will incorporate these clarifications and comparisons in the revised paper.
W2) Does the gain come from additional inference, and how do compute and accuracy trade off?
When aggregation rules use the same responses, they share the same model inference cost and token budget. We report score comparisons for both skill-augmented reasoning and ICL in W1. In skill-augmented reasoning, CHORUS and the consistency baselines use the same reasoning-plan policy. When applied to the same six CHORUS-generated responses, each aggregation rule uses the same number of generated tokens, averaging 3,086 per query for Qwen3-4B and 3,096 for Qwen3-8B. When applied to the same responses, CHORUS requires no additional model forward passes and therefore has the same model inference cost as simpler consensus rules. Its consensus step took less than 0.1% of inference time per query in both ICL and skill-augmented reasoning.
Compared with ICL methods that predict from a single selected context, CHORUS incurs additional inference cost because it evaluates multiple contexts per query. We further reduce this cost by adding batching to the implementation reported in the paper. We evaluate all context lengths across the four demonstration orderings in one batch (Batched inference). An attention mask ensures that each query sees only the demonstrations in its own context. With early stopping (Batched inference with early stopping), we process demonstrations in blocks of 10 and apply the paper’s stopping rule within each block. Each ordering stops once its predicted class has remained unchanged for 10 consecutive evaluated context lengths. Orderings that continue reuse the KV cache for the next block.
For Llama 3 8B, we measured inference time for both batched implementations.
Batching preserves CHORUS’s consensus rule. With early stopping, CHORUS’s inference time is comparable to that of EPR (1.50 vs. 1.25 seconds per query), with a modest score loss of about 1.5 points relative to full CHORUS. EPR trains a retriever for each task, whereas CHORUS applies the same consensus procedure across tasks without additional training.
We will incorporate these clarifications and results in the revised paper.