acceptodds
Under review as a conference paper at ICLR 2027

Beyond Context Selection: Consensus over Responses to Alternative Contexts

Abstract

Large language models can produce different predictions depending on how inference-time context is composed or applied. Existing approaches have largely addressed this variation by deciding which context to provide before prediction. We instead ask whether the target model's responses to alternative contexts can be used to adapt the final decision to each query. We propose CHORUS, a test-time consensus framework that complements context selection by jointly using responses from alternative ways of composing or applying the supplied context, without training a separate selector. CHORUS combines answers that are repeatedly supported across alternative contexts with answers that receive strong support from a single context. The same principle applies to different demonstration compositions in in-context learning and to different reasoning plans applied to fixed skill documents. Across seven LLMs, 12 classification tasks, six structured and short-form generation tasks, and four skill-augmented reasoning domains, CHORUS shows strong performance against the evaluated selection and consistency baselines. Further analyses show that their relative strengths change with the amount of available context and across tasks, and that adapting consensus decisions to the current query improves prediction. These results support using responses to alternative contexts as test-time evidence beyond context selection alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
In-context learning, test-time consensus, context utilization, skill-augmented reasoning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T11bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a test-time method that combines responses obtained under alternative contexts. It constructs answer-support sequences and uses agreement and commitment to select and fuse evidence. The method is evaluated on in-context classification and generation tasks, as well as skill-augmented reasoning. Some experiments have been conducted.

Critical strengths

The tackled problem is well-motivated. Selecting a single demonstration count, ordering, or reasoning plan can discard useful evidence from other contexts. It provides a unified way to use these alternative responses across several task settings. This is the main strength for this paper. Another strength is that the paper reports ICL classification results for five models and skill-augmented reasoning results for two Qwen3 models. It also provides component analyses, budget curves, and inference-time measurements. These experiments make the proposal more credible than a result on one benchmark alone.

Critical weaknesses
  • One major weakness is that the experiments do not sufficiently isolate the value of the consensus rule. Most baselines in Table 2 select or construct a context, whereas the proposed method obtains and aggregates predictions from multiple contexts. The paper does not compare AG+CM with simple aggregation—such as mean probability or voting—applied to the same saved predictions. Similarly, in the skill experiment, it uses diverse plan-conditioned responses while Self-Consistency samples under a fixed prompt; voting over its own response bank would be a more direct comparison. This matters because the observed gain may arise from response diversity or additional inference rather than the proposed consensus procedure. For context, Table 25 reports 71.92% accuracy at 16.06 seconds of inference per query for the proposed method on Llama 3 8B, versus 67.55% at 1.25 seconds for EPR; EPR also has a separate one-time preparation cost. A comparison using the same responses and a compute–accuracy analysis would substantially strengthen the central claim. The current analyses also do not directly isolate stable-position selection from choosing the final or highest-scoring position. Another major weakness is that the evidence for consistent gains is weaker than the headline averages suggest. On Llama 3 8B, the proposed method exceeds EPR’s twelve-task classification average by 4.37 points. The 29.84-point difference between these methods on CoLA contributes approximately 2.49 points to that average gap. Reporting CoLA’s standard MCC metric and an average excluding CoLA would clarify robustness; using accuracy consistently is not itself an error. The classification margin is also much smaller on Llama 3 70B (0.17 points over the strongest average baseline). In generation, Table 3 reports 56.0 for the proposed method versus 55.7 for EPR across six tasks, with no individual task on which it has the highest score. These results support the method’s competitiveness, but provide limited evidence for a consistent advantage across tasks and models.
Questions for authors

What is the performance for CoLA’s standard MCC metric?

3 comments
AuthorsOpenReview ↗

Author response to Reviewer T11b (1/2)

We are grateful for your thoughtful feedback. Our revisions to the paper will reflect the points discussed below. Following your feedback, we now evaluate CoLA with MCC, its standard evaluation metric, and have recalculated the affected averages.

W1) How does CHORUS compare with simple aggregation in ICL and skill-augmented reasoning?

In ICL, we applied majority voting and probability averaging under the same inference budget as CHORUS (Llama 3 8B, 12 tasks).

Method12-task average score ± three-run SD
Majority voting68.79 ± 0.04
Probability averaging68.60 ± 0.39
CHORUS70.20 ± 0.53

In skill-augmented reasoning, CHORUS and the consistency baselines use the same set of reasoning-plan prompts. We apologize that our original description did not make this clear. The main-text results below use a maximum budget of six responses and report accuracy (%) as mean ± SD over three runs.

MethodQwen3-4BQwen3-8B
Self-Consistency (Majority)77.33 ± 0.3476.21 ± 0.36
Self-Consistency (Weighted)77.55 ± 0.2176.43 ± 0.69
Adaptive-Consistency77.33 ± 0.3476.21 ± 0.36
CHORUS78.98 ± 0.3179.69 ± 0.17

We will incorporate these clarifications and comparisons in the revised paper.

W2) Does the gain come from additional inference, and how do compute and accuracy trade off?

When aggregation rules use the same responses, they share the same model inference cost and token budget. We report score comparisons for both skill-augmented reasoning and ICL in W1. In skill-augmented reasoning, CHORUS and the consistency baselines use the same reasoning-plan policy. When applied to the same six CHORUS-generated responses, each aggregation rule uses the same number of generated tokens, averaging 3,086 per query for Qwen3-4B and 3,096 for Qwen3-8B. When applied to the same responses, CHORUS requires no additional model forward passes and therefore has the same model inference cost as simpler consensus rules. Its consensus step took less than 0.1% of inference time per query in both ICL and skill-augmented reasoning.

Compared with ICL methods that predict from a single selected context, CHORUS incurs additional inference cost because it evaluates multiple contexts per query. We further reduce this cost by adding batching to the implementation reported in the paper. We evaluate all context lengths across the four demonstration orderings in one batch (Batched inference). An attention mask ensures that each query sees only the demonstrations in its own context. With early stopping (Batched inference with early stopping), we process demonstrations in blocks of 10 and apply the paper’s stopping rule within each block. Each ordering stops once its predicted class has remained unchanged for 10 consecutive evaluated context lengths. Orderings that continue reuse the KV cache for the next block.

For Llama 3 8B, we measured inference time for both batched implementations.

MethodPreparation (total hours for 12 tasks)Inference (s/query)Average score
EPR≈27.241.2567.55
CHORUS, original implementation–16.0670.20
CHORUS, original implementation with early stopping–4.3068.67
CHORUS (Batched inference)–3.8570.20
CHORUS (Batched inference with early stopping)–1.5068.67

Batching preserves CHORUS’s consensus rule. With early stopping, CHORUS’s inference time is comparable to that of EPR (1.50 vs. 1.25 seconds per query), with a modest score loss of about 1.5 points relative to full CHORUS. EPR trains a retriever for each task, whereas CHORUS applies the same consensus procedure across tasks without additional training.

We will incorporate these clarifications and results in the revised paper.

AuthorsOpenReview ↗

Author response to Reviewer T11b (2/2)

W3) Does stable-position selection help beyond choosing the highest-scoring position?

Across 12 tasks, CHORUS achieves the highest average score when AG and CM are combined with stable-position selection. The variant without stable-position selection uses the position with the highest class probability. Ordering selection remains unchanged in this comparison.

Method12-task average score
CHORUS w/o stable-position selection66.56
CHORUS70.20

We will incorporate these clarifications and results in the revised paper.

W4, Q) What is CoLA's performance in MCC, and how does the correction affect the averages?

After correcting the CoLA metric, CHORUS retains the highest 12-task average on each of the five models. We apologize for mixing up accuracy and MCC when reporting the original CoLA results. We now report MCC for every CoLA entry and have recalculated all 12-task averages accordingly. Each cell below shows CoLA MCC × 100 / 12-task average / 11-task average excluding CoLA.

MethodLlama 3 8BQwen2.5 7BMistral 7BGemma 2 27BLlama 3 70B
Random50.52 / 65.03 / 66.3558.79 / 72.14 / 73.3550.22 / 68.86 / 70.5561.59 / 74.66 / 75.8564.11 / 76.83 / 77.99
TopK-BM2554.75 / 66.96 / 68.0763.15 / 73.60 / 74.5552.70 / 70.08 / 71.6662.35 / 75.15 / 76.3165.85 / 78.15 / 79.26
TopK-BERT62.01 / 66.94 / 67.3864.77 / 74.00 / 74.8455.41 / 69.87 / 71.1968.59 / 75.60 / 76.2367.82 / 77.81 / 78.72
DPP-BERT57.69 / 67.01 / 67.8663.55 / 74.38 / 75.3756.11 / 69.90 / 71.1667.56 / 74.89 / 75.5669.61 / 78.86 / 79.70
D.Va55.79 / 66.95 / 67.9761.76 / 73.77 / 74.8655.72 / 70.05 / 71.3561.36 / 74.03 / 75.1865.94 / 76.81 / 77.79
EPR55.39 / 67.55 / 68.6664.92 / 74.28 / 75.1353.09 / 70.14 / 71.6963.80 / 75.74 / 76.8360.57 / 77.94 / 79.52
TopicK51.27 / 66.63 / 68.0262.47 / 72.25 / 73.1456.77 / 68.61 / 69.6955.34 / 72.72 / 74.3063.81 / 76.44 / 77.58
EASE55.37 / 65.16 / 66.0561.52 / 71.72 / 72.6552.53 / 68.58 / 70.0455.22 / 73.51 / 75.1762.76 / 74.19 / 75.22
Se247.51 / 63.75 / 65.2363.15 / 71.65 / 72.4253.01 / 68.42 / 69.8262.41 / 72.42 / 73.3366.93 / 74.48 / 75.17
SeDPO52.79 / 65.14 / 66.2760.11 / 71.30 / 72.3254.09 / 67.62 / 68.8566.48 / 74.05 / 74.7461.62 / 75.12 / 76.34
DR-IKE41.93 / 60.66 / 62.3759.97 / 69.47 / 70.3350.62 / 66.84 / 68.3252.10 / 68.02 / 69.4661.30 / 72.31 / 73.31
Adaptive-k51.40 / 57.57 / 58.1360.78 / 69.70 / 70.5151.57 / 67.42 / 68.8664.38 / 70.43 / 70.9863.67 / 72.69 / 73.52
CHORUS64.57 / 70.20 / 70.7163.15 / 74.88 / 75.9457.52 / 71.09 / 72.3268.84 / 76.36 / 77.0470.05 / 79.03 / 79.85

After excluding CoLA, CHORUS still achieves the highest 11-task average on each of the five models. The strongest baseline varies by model.

W5) Is the advantage consistent across tasks and models, including Llama 3 70B and generation?

We agree that the gains over the strongest baseline are small on Llama 3 70B (DPP-BERT) and generation (EPR). However, the strongest baseline changes across models. For example, EPR scores higher on average than DPP-BERT on Llama 3 8B and Gemma 2 27B, while DPP-BERT scores higher on Llama 3 70B. In contrast, CHORUS achieves the highest average score on every evaluated model and setting. For each task, EPR uses a language model to label candidate demonstrations based on the probability it assigns to the correct answer. It then trains a separate retriever on those labels to select demonstrations for new queries. We hope the reviewer will consider that CHORUS achieves these results with the same consensus procedure across ICL classification, generation, and skill-augmented reasoning, without additional training.

As the discussion period draws to a close, we would be grateful if you could take another look at our rebuttal and revised manuscript. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.

Reviewer_BRpvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CHORUS, a training free method for improving LLM predictions by means of considering responses that are produced under different contexts, as against using a single context. The method computes scores to result from those responses and applies two complementary aggregation strategies: Agreement and Commitment. It also evaluates the method on 7 models across 12 classification tasks, 6 generation tasks and 4 skill-augmented reasoning domains. Their claim is that adapting consensus decisions to the current query improves prediction and their results support the use of responses to alternative contexts as test-time evidence beyond context selection alone.

Critical strengths
  1. Meaningful Problem Focus: The paper addresses a highly practical and relevant limitation of context-based LLM inference, specifically that prediction reliability can fluctuate heavily based on the exact demonstration or reasoning instructions provided.

  2. Extensive Evaluation: The empirical evaluation spans a commendable breadth, covering 7 models across multiple task types (12 classification, 6 generation, and 4 skill-augmented reasoning domains).

  3. Thorough System Analysis: The paper provides valuable depth regarding the combination of Agreement and Commitment, alongside helpful analyses of skill sensitivity, computational overhead, and early-stopping mechanisms.

Critical weaknesses
  1. Confounded Procedural Variations: In the skill experiments, CHORUS alters both the generation procedure and the aggregation rule simultaneously, making it impossible to cleanly isolate what portion of the gain stems from the aggregation rule itself. Specifically, while CHORUS samples across varying reasoning plans, the consistency baselines are restricted to sampling from a fixed prompt.

  2. Missing Critical Baselines: Standard, essential baselines—such as Majority Voting and Confidence-Weighted Voting evaluated over the exact same plan-conditioned responses—are completely omitted. This gap extends to the In-Context Learning experiments as well. A direct comparison against ordinary voting on identical generated responses, rather than fixed-prompt samples, is required to demonstrate that stability selection and fusion provide a clear advantage over simpler alternatives.

  3. Unclear Computational and Token Efficiency: The additional context processing required by CHORUS is substantial, yet the evaluation fails to establish if simpler ensemble methods can achieve comparable accuracy with the same or lower computation. It remains unverified if CHORUS outperforms basic ensembles when matched for inference latency or processed token budgets. Furthermore, the paper lacks clarity on how generated token counts differ between the proposed plan-conditioned responses and the fixed-prompt baseline samples.

  4. Under-Justified Metric Formulation and Marginal Significance: The interpretation and formulation of the "Commitment" metric lacks a solid foundational justification. In Equation 9, confidence magnitude is essentially eliminated in low-entropy or binary tasks; when ε is negligible, disparate distributions like (0.51, 0.49) and (0.99, 0.01) become functionally identical after normalization. Consequently, preserving the largest normalized support fails to preserve the strongest original confidence. Furthermore, the empirical gains are marginal (e.g., a 0.17 improvement on Llama-3-70B and a 0.3 average gain on generation tasks) and are averaged across different evaluation metrics, making the statistical reliability and practical impact of the method difficult to verify without explicit confidence intervals.

Questions for authors
  1. Isolation of Aggregation Gains: In the skill experiments, the generation procedure (varying reasoning plans) and the aggregation rule are altered simultaneously. Could you provide an ablation study that applies ordinary Majority Voting or Confidence-Weighted Voting to the exact same plan-conditioned responses generated by CHORUS? This is necessary to isolate how much of the gain stems from the aggregation rule versus the decoding procedure itself.

  2. Baseline Prompts: Why were the consistency baselines restricted to sampling from a fixed prompt while CHORUS sampled across diverse plans? Please clarify how the generated token counts and total inference latency compare between these two setups.

  3. Commitment Metric Formulation: In Equation 9, the confidence magnitude appears to be heavily suppressed in binary or low-entropy tasks after normalization (e.g., transforming distinct distributions like (0.51, 0.49) and (0.99, 0.01) into nearly identical outputs when ε is negligible). Can you provide a task-level ablation of this aggregate normalization to justify why preserving the largest normalized support is preferable to preserving the strongest raw confidence?

  4. Statistical Significance: Given the narrow empirical margins on certain configurations (e.g., the 0.17 gain on Llama-3-70B and the 0.3 average gain on generation tasks), could you provide confidence intervals or error bars across independent runs to verify the statistical reliability of these improvements?

3 comments
AuthorsOpenReview ↗

Author response to Reviewer BRpv (1/2)

We appreciate your helpful comments and will address the points you raised in the revised paper. Following Reviewer T11b's feedback, we now evaluate CoLA with MCC, its standard evaluation metric, and have recalculated the affected averages.

W1, W2, Q1) How does CHORUS compare with simple aggregation in skill-augmented reasoning and ICL?

We apologize that our original description of the skill-augmented reasoning comparison was unclear. In the controlled comparison, CHORUS and the voting baselines use the same reasoning-plan prompts. For each query, we applied CHORUS, majority voting, and confidence-weighted voting to exactly the same six CHORUS-generated responses. Table 4 reports the resulting accuracies. CHORUS achieves the highest mean accuracy on both models.

MethodQwen3-4BQwen3-8B
Self-Consistency (Majority)77.33 ± 0.3476.21 ± 0.36
Self-Consistency (Weighted)77.55 ± 0.2176.43 ± 0.69
Adaptive-Consistency77.33 ± 0.3476.21 ± 0.36
CHORUS78.98 ± 0.3179.69 ± 0.17

Values are mean accuracy (%) ± SD over three runs. Figure 2 shows the results across response budgets.

In ICL, CHORUS achieves a higher average score than majority voting and probability averaging under the same inference budget. We evaluate all three methods on Llama 3 8B across 12 classification tasks.

Method12-task average score ± three-run SD
Majority voting68.79 ± 0.04
Probability averaging68.60 ± 0.39
CHORUS70.20 ± 0.53

We will clarify the evaluation setup and include the ICL comparison in the revised paper.

W3, Q2) How do latency and token usage compare with simpler ensembles under the same plan policy?

As clarified in our responses to W1, W2, and Q1, the skill-augmented reasoning results in the paper compare CHORUS with other ensemble methods using the same set of reasoning plans under matched response budgets. When aggregating the same six CHORUS-generated responses, all methods use identical generated-token counts, averaging 3,086 per query for Qwen3-4B and 3,096 for Qwen3-8B. CHORUS requires no additional model forward passes, and its consensus postprocessing took less than 0.1% of inference time per query in both ICL and skill-augmented reasoning.

In ICL, CHORUS also achieves a higher average score than majority voting and probability averaging under the same inference budget. We evaluate all three methods on Llama 3 8B across 12 classification tasks.

Method12-task average score ± three-run SD
Majority voting68.79 ± 0.04
Probability averaging68.60 ± 0.39
CHORUS70.20 ± 0.53

We additionally report the computational cost relative to standard ICL baselines. Compared with ICL methods that predict from a single selected context, CHORUS incurs additional inference cost because it evaluates multiple contexts per query. We further reduce this cost by adding batching to the implementation reported in the paper. We evaluate all context lengths across the four demonstration orderings in one batch (Batched inference). An attention mask ensures that each query sees only the demonstrations in its own context. With early stopping (Batched inference with early stopping), we process demonstrations in blocks of 10 and apply the paper’s stopping rule within each block. Each ordering stops once its predicted class has remained unchanged for 10 consecutive evaluated context lengths. Orderings that continue reuse the KV cache for the next block.

For Llama 3 8B, we measured inference time for both batched implementations.

MethodPreparation (total hours for 12 tasks)Inference (s/query)Average score
EPR≈27.241.2567.55
CHORUS, original implementation–16.0670.20
CHORUS, original implementation with early stopping–4.3068.67
CHORUS (Batched inference)–3.8570.20
CHORUS (Batched inference with early stopping)–1.5068.67

Batching preserves CHORUS’s consensus rule. With early stopping, CHORUS’s inference time is comparable to that of EPR (1.50 vs. 1.25 seconds per query), with a modest score loss of about 1.5 points relative to full CHORUS. EPR trains a retriever for each task, whereas CHORUS applies the same consensus procedure across tasks without additional training.

We will incorporate these clarifications and results in the revised paper.

AuthorsOpenReview ↗

Author response to Reviewer BRpv (2/2)

W4, Q3) Does Eq. 9 suppress confidence magnitude, and does normalization outperform raw probabilities?

We agree that Eq. 9 largely removes confidence magnitude in binary tasks. In our same-response AG+CM comparison, binary-task averages are close, with a small numerical advantage for normalization.

We normalize support across candidates at each trajectory position to compare relative support across contexts, whose raw confidence can vary with demonstration choice and order [1]. The preferred answer at each position is preserved, so preference changes along the trajectory remain available for consensus and stable position selection. With three or more candidates, normalized scores also retain differences in relative support. Task-level effects vary, while normalization yields slightly higher averages in both task groups. The first table reports Llama 3 8B scores, using the same responses and unchanged selection and fusion rules, with equal task weights.

AG+CM task groupEq. 9Raw
Multi-class (4 tasks)73.5273.13
Binary (8 tasks)68.5368.23

Across 12 tasks, CHORUS achieves the highest average score when AG and CM are combined with stable-position selection. The variant without stable-position selection uses the position with the highest class probability. Ordering selection remains unchanged in this comparison.

Method12-task average score
CHORUS w/o stable-position selection66.56
CHORUS70.20

We will incorporate these clarifications and results in the revised paper.

[1] Zihao Zhao et al. Calibrate Before Use: Improving Few-Shot Performance of Language Models. ICML 2021.

W4, Q4) Are the small gains on Llama 3 70B and generation statistically reliable?

We agree that the gains over the strongest baseline in each setting, DPP-BERT for Llama 3 70B and EPR for generation, are small. The differences in the two settings raised by the reviewer and their 95% confidence intervals are shown below.

SettingCompared withDifference [95% CI]
ICL, Llama 3 70BBest baseline: DPP-BERT+0.17 [−1.45, +1.80]
Generation, 6 tasksBest baseline: EPR+0.25 [−1.55, +2.14]

Both intervals include zero, so these results do not establish statistically significant improvements.

We hope the reviewer will also consider how consistently CHORUS performs across settings. The strongest baseline varies across models and benchmarks. CHORUS achieves the best or near-best average scores across ICL classification, generation, and skill-augmented reasoning. In ICL, a strong learned baseline such as EPR uses the language model to score candidate demonstrations on each task’s training set and then trains a separate retriever for that task. CHORUS forms consensus directly from the target LLM’s own response signals at inference time, using the same rule across tasks and models without additional training.

As the discussion period draws to a close, we would be grateful if you could take another look at our rebuttal and revised manuscript. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.

Reviewer_Hp7kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a test-time ensembeling method that aggregates predictions across different demonstration orderings and lengths for ICL, and different reasoning plans for skill-augmented reasoning.

Critical strengths

The method seems to work well compared to other evaluated baselines (at least for ICL classification, and skill augmented reasoning). ICL generation seems hit and miss.

Critical weaknesses

I don’t really understand the motivation behind doing this. Figure 1 shows preferences don’t transfer across models for both tasks, so how does this motivate using “agreement across responses”

The paper in general isn’t well written, and is hard to follow. For example reading the abstract “Existing approaches have largely addressed this variation by deciding which context to provide before prediction. We instead ask……final decision to each query” it’s not clear at all what was being done before and what you’re suggesting to do. Generally speaking, reading the introduction it is hard to get much out of it because everything reads very vague.

There is a lot of work on test-time ensembling. What is precisely new here other than a more elaborate test-time heuristic? The motivation for using the different metrics like agreement, commitment, and stability doesn’t seem strong. There are some ablations, but it’s not clear why these metrics and their combination works well. What does each one buy you separately? How does the proposed method compare with simpler baselines like averaging probabilities or majority voting using the same predictions and inference budget?

The method also uses a lot of compute compared to icl baselines, what if they were run computed-matched

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Author response to Reviewer Hp7k (1/3)

Thank you for your constructive feedback. We will incorporate the changes outlined below into the revised manuscript. Following Reviewer T11b's feedback, we now evaluate CoLA with MCC, its standard evaluation metric, and have recalculated the affected averages.

W1) How does Figure 1 motivate using agreement across responses? CHORUS aggregates responses produced by a single target model for the same query under alternative contexts. In ICL, these alternatives vary the number and ordering of demonstrations. In skill-augmented reasoning, they vary the reasoning plan used to apply the same skill document.

Figure 1 shows that a context choice that works well for one model may not work well for another. The key implication is that the final decision should be grounded in the target model’s own response signals, since context preferences obtained from another model may not reflect how the target model uses the same information. CHORUS therefore generates responses under alternative contexts and seeks consensus among these target-model signals. To make fuller and more reliable use of these signals, CHORUS combines support shared across responses with strong individual support and uses stability to select the evidence for each query.

Thus, the connection is that model-dependent context preferences motivate making the final decision through consensus among the target model’s own responses. We will clarify this motivation in the revised introduction.

W2) What did prior approaches do, and what does CHORUS do differently? Thank you for pointing out this lack of clarity. We will revise the abstract and introduction to make the distinction explicit:

  1. Context-selection methods, including BM25, TopK-BERT, and EPR, decide what information to provide before prediction, but CHORUS makes the final decision from the target model’s own response signals. In our ICL evaluation, learned retrieval methods such as EPR require a separate retriever for each task, trained on that task’s own data. CHORUS instead forms consensus directly from the target model’s response signals at inference time, using the same rule across tasks and models without additional training. Figure 1 motivates using the target model’s own responses, because context preferences can differ across models. In ICL, these responses are obtained by varying demonstration counts and orderings. In skill-augmented reasoning, they are generated using different reasoning plans for the same skill document.

  2. CHORUS instead introduces an efficient consensus procedure based on the target model’s own response signals. It combines agreement and commitment to aggregate evidence across responses, and uses stability to identify a point where the predicted answer remains consistent as more evidence is considered. The procedure therefore determines both how to combine response evidence and which point in the response sequence to use for the final prediction.

We will revise the abstract and introduction to clarify how CHORUS differs from prior context-selection methods and how its consensus procedure works.

W3) What is new compared with existing test-time ensembling? CHORUS forms consensus over different ways the target model uses context. We introduce a unified consensus procedure for this setting and adapt existing consistency methods for comparison. The distinction lies in how CHORUS combines answer support and selects the evidence used for the final decision:

  1. CHORUS combines shared support with strong individual support. Majority voting counts answer occurrences, while confidence-weighted voting accumulates weighted votes. CHORUS explicitly separates two signals: Agreement aggregates support shared across stable response sequences, whereas Commitment retains the strongest normalized support that any individual sequence provides for each candidate answer. Combining these signals allows the final decision to consider both broadly supported answers and strong evidence that is not shared across all sequences.

  2. CHORUS selects stable evidence for each signal before fusion. Agreement and Commitment can select different sequence positions, based on whether their predicted answers remain stable. Commitment additionally requires a support margin. Their selected distributions are then combined. Adaptive-Consistency adjusts the number of samples based on confidence in the majority answer. CHORUS instead makes an explicit selection of which evidence each consensus signal contributes to the final prediction.

This design makes the final decision depend on both the distribution of support across sequences and its persistence across sequence positions. We will clarify these distinctions in the revised manuscript. See W4 for component ablations and W5 for comparisons with simple aggregation using identical responses and inference budgets.

AuthorsOpenReview ↗

Author response to Reviewer Hp7k (2/3)

W4) What does each of agreement, commitment, and stability contribute? AG captures shared support, CM preserves strong individual support, and CHORUS combines these signals with stable evidence selection. The following component comparison illustrates their roles on Llama 3 8B across 12 classification tasks.

The single-ordering baseline selects the highest-probability answer and position along the relevance ordering. AG and CM use the same normalization and stable-position selection as CHORUS but keep only one signal. CHORUS applies the complete procedure, including normalization, stable evidence selection, and fusion.

MethodAverage score
Single-ordering argmax65.45
AG69.40
CM69.94
AG+CM at the last position (no stable-position selection)68.71
CHORUS70.20
  1. AG improves the average score by 3.95 points by incorporating support across sequences. Its geometric-mean aggregation favors candidates supported by multiple selected sequences. This provides corroborating evidence beyond the highest-probability prediction from a single ordering.

  2. CM improves the average score by 4.49 points by preserving strong individual support. Its maximum-based aggregation retains the strongest support for each candidate across sequences. This allows an informative response sequence to influence the decision even when other sequences do not provide comparable support.

  3. CHORUS improves over AG and CM by 0.80 and 0.26 points, respectively, by combining the two signals. The stronger rule changes across tasks: AG leads on five of the 12 tasks and CM on five (two ties), and CHORUS scores above the average of AG and CM on eight tasks (Figure 3 of the paper). Both rules already select stable evidence, so these gains come from fusing shared support with strong individual support. This design considers both how support is distributed across sequences and whether the predicted answer persists across positions.

We will clarify these component roles and experimental settings in the revised manuscript. W5 provides comparisons with probability averaging and majority voting using identical responses and inference budgets.

W5) How does CHORUS compare with probability averaging and majority voting using the same predictions and inference budget? CHORUS improves the average score over simple aggregation within the same inference budget. We applied majority voting and probability averaging under the same inference budget as CHORUS (Llama 3 8B, 12 classification tasks).

Method12-task average score ± three-run SD
Majority voting68.79 ± 0.04
Probability averaging68.60 ± 0.39
CHORUS70.20 ± 0.53

The paper’s skill response-budget figure also shows higher mean accuracy for CHORUS than both Self-Consistency variants at matched budgets of four to six responses on Qwen3-4B and Qwen3-8B. At each budget, all methods aggregate the same plan-conditioned responses.

We will incorporate these comparisons in the revised paper.

AuthorsOpenReview ↗

Author response to Reviewer Hp7k (3/3)

W6) How does CHORUS compare with the ICL baselines when compute is matched? We examine compute in both directions: simple baselines are given the same inference as CHORUS, and CHORUS is run with less inference. The results below are for ICL classification on Llama 3 8B across 12 tasks.

  1. With the same inference, CHORUS has the highest average score. The majority-voting and probability-averaging baselines in W5 use exactly the responses that CHORUS uses, so their compute is identical.

  2. Compared with ICL methods that predict from a single selected context, CHORUS incurs additional inference cost because it evaluates multiple contexts per query. We further reduce this cost by adding batching to the implementation reported in the paper. We evaluate all context lengths across the four demonstration orderings in one batch (Batched inference). An attention mask ensures that each query sees only the demonstrations in its own context. With early stopping (Batched inference with early stopping), we process demonstrations in blocks of 10 and apply the paper’s stopping rule within each block. Each ordering stops once its predicted class has remained unchanged for 10 consecutive evaluated context lengths. Orderings that continue reuse the KV cache for the next block.

For Llama 3 8B, we measured inference time for both batched implementations.

MethodPreparation (total hours for 12 tasks)Inference (s/query)Average score
EPR≈27.241.2567.55
CHORUS, original implementation–16.0670.20
CHORUS, original implementation with early stopping–4.3068.67
CHORUS (Batched inference)–3.8570.20
CHORUS (Batched inference with early stopping)–1.5068.67

Batching preserves CHORUS’s consensus rule. With early stopping, CHORUS’s inference time is comparable to that of EPR (1.50 vs. 1.25 seconds per query), with a modest score loss of about 1.5 points relative to full CHORUS. EPR trains a retriever for each task, whereas CHORUS applies the same consensus procedure across tasks without additional training.

We will incorporate these clarifications and results in the revised paper.

As the discussion period draws to a close, we would be grateful if you could take another look at our rebuttal and revised manuscript. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.