- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes length-conditioned majority voting, which assigns each sampled answer a logistic weight based on its reasoning length. Two parameters start with a preference for shorter responses and are adapted across incoming questions using ordinary majority agreement as pseudo-labels. The method does not require future questions or correct-answer labels, but collects the current question's responses before updating and voting. The language model remains frozen. Experiments span four existing protocols, comparing unweighted voting, length-based selection, and confidence-based aggregation. Reported averages favor LCMV, and an ablation suggests an improvement over fixed length weighting.
Critical strengths
-
The online adaptation setting is worthwhile. Updating a weighting rule as questions arrive, without access to future questions or correctness labels, is a meaningful constraint compared with fitting on a complete offline collection. Adapting only two scalars adds little computation once responses exist and requires no language-model fine-tuning. Relative to Shortest Majority Vote, the contribution is this online adaptation, rather than length weighting itself. A reliable improvement under these constraints would be practically useful.
-
The evaluation includes relevant breadth and controls. The four protocols cover reasoning and instruction-tuned models, different sample counts, mathematics, and GPQA. Table 5 directly compares fixed, online, and offline weighting, rather than reporting only comparisons to ordinary majority voting. Repeating generation seeds and question orders is a sensible attempt to examine variability in the sequential setting.
Critical weaknesses
-
The incremental value of online adaptation is not established convincingly enough. Table 5 is the most direct evidence: online weighting improves the three protocol averages over fixed weighting by 0.7, 1.0, and 1.5 points. These gains are promising, but their paired uncertainty is unclear. The reported 25 runs combine five generation seeds with five question orders per seed; separating those sources of variation matters when assessing reliability. The broader comparisons do not resolve this: Section 5.2 imports MV, FFS, and Shortest MV in Table 1, and all baselines in Table 3, from earlier publications, whereas LCMV uses newly generated responses. Matching nominal protocols does not ensure identical samples or parsing. Some baselines are reproduced, but the principal comparisons still need evaluation on the same responses. The experimental argument should center on whether adaptation reliably improves over fixed weighting and established inexpensive aggregators, with benchmark breadth supporting that conclusion.
-
The learning signal needs a clearer interpretation and mechanism analysis. Equation 5 labels agreement with the sampled majority, which can itself be wrong. Learning that target does not automatically yield the correctness probability asserted in Equation 4. This is not a reason to dismiss the heuristic: reweighting can still change the winner and improve accuracy. However, the paper should show whether adaptation repairs incorrect majority answers more often than fixed weighting, and how often it damages correct ones. That analysis would explain when the online update contributes useful information rather than reinforcing the majority's existing preferences.
Questions for authors
- Contribution clarification: What is the paired improvement and uncertainty for Online versus Fixed LCMV in Table 5? How do MV and Shortest MV compare on exactly the same responses and grader?
- Mechanism clarification: When ordinary MV is incorrect, how often does adaptation recover the correct answer compared with fixed weighting, and how often does it damage a correct MV answer? Does the learned weight track actual correctness or only majority agreement?
- Online robustness clarification: How much variation comes from response generation versus question ordering, particularly the first question used for normalization? Does the benefit over fixed weighting persist across those choices?
3 comments
Answers to Reviewer 7W89 (2nd part)
Q3: Equation 5 labels agreement with the sampled majority, which can itself be wrong. Learning that target does not automatically yield the correctness probability asserted in Equation 4. This is not a reason to dismiss the heuristic: reweighting can still change the winner and improve accuracy. However, the paper should show whether adaptation repairs incorrect majority answers more often than fixed weighting, and how often it damages correct ones. That analysis would explain when the online update contributes useful information rather than reinforcing the majority's existing preferences. (...) When ordinary MV is incorrect, how often does adaptation recover the correct answer compared with fixed weighting, and how often does it damage a correct MV answer? Does the learned weight track actual correctness or only majority agreement?
A3: The simplest way to quantitatively measure how frequently our method repairs incorrect majority answers (and, conversely, how often it damages correct ones), is by using accuracy. In fact, by definition, accuracy is the number of correct responses relative to the total number of responses. Therefore, a higher accuracy for Online LCMV compared to MV shows precisely how often our approach improves the pseudo-labels obtained using Eq. (5). For the sake of brevity, we show below only the mean values, taken from Tab. 1, 2 and 3, calculated for each protocol by averaging across all the datasets and models used:
| MV | Online LCMV | Fixed LCMV | |
|---|---|---|---|
| Ave. Tab. 1 | 57.1 | 58.9±0.6 | 58.2±0.7 |
| Ave. Tab. 2 | 71.8 | 73.7±1.3 | 72.7±1.2 |
| Ave. Tab. 3 | 56.7 | 62.4±0.4 | 60.9±0.5 |
| Ave. Tab. 6 | 65.8 | 67.4 | - |
| Average | 62.9 | 65.6 | 63.9 |
In the above table we also added the average accuracy of the first protocol (Fig. 2), obtained across all the values of and all the models, whose detailed results are reported in Tab. 6. This comparison shows that we consistently improve over MV, independently of the setting, demonstrating that our approach is able to improve the pseudo-labes computed with standard MV (Eq. (5)). Note that other methods do not always achieve an improvement over MV. For instance, when using protocol 1, proposed by Short-3@k, the latter underperforms compared to MV when using DeepSeek-R1-Distill-Qwen-32B (Fig. 2a and Tab. 6). In the third protocol (Tab. 2), proposed by FFS, FFS is, on average, inferior to MV. In the above table, we also added the average results of Fixed LCMV, which again show that our unsupervised adaptation strategy improves over the fixed-prior variant.
Furthermore, following your suggestion, we analyzed how often Online LCMV changes the aggregated MV decision (Eq. (2)) from correct to incorrect, and vice versa. To reduce the number of possible combinations, we used an example with . Specifically, we used DeepSeek-R1-Distill-Qwen-32B tested on HMMT February 2025 from Tab. 6. Using , we get 1,140 different subsets of responses for a single question (please see the description of Protocol 1 on lines 311–313). Given that HMMT February 2025 consists of 30 questions, the experiment was repeated for a total of 1,140 x 30 = 34,200 runs. In the table below, column “Response configuration” indicates the configuration of the votes used in Eq. (2). For instance, “3C” means 3 correct answers, “2C–1E” means 2 correct and 1 incorrect answer, while “1C–1E–1E” means 1 correct and 2 different, incorrect answers. Column “Number of runs” indicates the number of runs for which MV produced that configuration. Finally, “Incorrect → Correct” and “Correct → Incorrect” denote the number of times Online LCMV turned an incorrect MV decision into a correct one, and vice versa.
| Response configuration | Number of runs | Incorrect → Correct | Correct → Incorrect |
|---|---|---|---|
| 3C | 6,938 | 0 | 0 |
| 2C–1E | 4,026 | 0 | 4 |
| 1C–1E–1E | 5,074 | 1,889 | 693 |
| 2E–1C | 1,303 | 10 | 0 |
| 3E | 1,500 | 0 | 0 |
| 2E–1E | 4,997 | 0 | 0 |
| 1E–1E–1E | 10,362 | 0 | 0 |
| Total | 34,200 | 1,899 | 697 |
In the example shown in the table above, most of the changes are focused on the configuration “1C–1E–1E”. In this configuration, since there is no winner, MV chooses at random, thus only one-third of the 5,074 runs, on average, will be correct. For the remaining two-thirds of incorrect decisions, Online LCMV corrects 1,889 runs while damaging 693 correct ones. Overall, Online LCMV corrects 1,899 decisions and damages 697. The difference, 1,202, when compared to the total (34,200), corresponds to an increase in accuracy of approximately 3.5%, as shown in Table 6 in the entry corresponding to this experiment.
Answers to Reviewer 7W89 (3rd part)
Q4: How much variation comes from response generation versus question ordering, particularly the first question used for normalization? Does the benefit over fixed weighting persist across those choices?
A4: As mentioned in Sec. 5.1 and 5.2, we randomly permute the question order to show that our method (both the Fixed and the Online versions) is not sensitive to this order, including the normalization based on the first question. This robustness is demonstrated by the low standard deviation obtained in Tab. 1, 2, and 3. For instance, in Tab. 1, the standard deviation for Online LCMV is lower than that of the other methods for which it was possible to repeat the experiments using different random seeds. The same result was obtained with the first experimental protocol (Fig. 2 and Table 6). Also in that case, the average standard deviation (Tab. 6) is lower for Online LCMV with respect to MV and Short-3@k in 12 and 13 out of 16 comparisons, depending on the adopted base LLM (lines 840-847). Note that in the first protocol, the question order is fixed but the experiments are based on randomly extracting subsets of CoTs drawn from a set of 20 generations, thus in every subset the first question is different (see App. C.1 for more details). Finally, please note that also Fixed LCMV was tested using the same random order, thus the benefit shown in Tab. 5 takes into account the first-question normalization (Sec. 5.3).
Answers to Reviewer 7W89
We thank the Reviewer for their work and feedback, and for appreciating our method, its unsupervised and online strategy and its usefulness, as well as for appreciating the diversity and breadth of our experimental evaluation. We provide responses to all questions below.
Q1: The incremental value of online adaptation is not established convincingly enough. Table 5 is the most direct evidence: online weighting improves the three protocol averages over fixed weighting by 0.7, 1.0, and 1.5 points. These gains are promising, but their paired uncertainty is unclear. The reported 25 runs combine five generation seeds with five question orders per seed; separating those sources of variation matters when assessing reliability. (...) What is the paired improvement and uncertainty for Online versus Fixed LCMV in Table 5?
A1: Following your suggestion, we report below Tab. 5 in which the results are computed with different combinations of random seeds. Specifically, the first column “5 seeds x 5 permutations” corresponds to the 25 runs used for each model and dataset in Tab. 1-3 and summarized in Tab. 5. On the other hand, “5 generative seeds” corresponds to using only 5 random seeds for the autoregressive CoT generation with temperature, without any question order permutation. Finally, “1 generative seed, 5 permutation seeds” indicates the use of only one random seed for generation and 5 random seeds for permutation.
The results in the table above show that there is no substantial difference in the use of multiple seeds for generation and/or permutation.
Q2: Section 5.2 imports MV, FFS, and Shortest MV in Table 1, and all baselines in Table 3, from earlier publications, whereas LCMV uses newly generated responses. Matching nominal protocols does not ensure identical samples or parsing. Some baselines are reproduced, but the principal comparisons still need evaluation on the same responses. (...) How do MV and Shortest MV compare on exactly the same responses and grader?
A2: Unfortunately, it is not very clear to us what you mean by "same responses and grader". If you mean that the comparison must be conducted using the same ground-truth answers, then, of course we use the same answers as the other approaches (as these are unique and no others exist). Furthermore, we have not only used the same answers, but—where available—we have also adopted the same string parsers, verifiers, prompts and generation hyperparameters proposed in the papers from which we drew our experimental protocols and against which we benchmarked our work (Sections 5.1, 5.2, and App. B). This, in fact, implicitly favors our competitors.
If, on the other hand, by "responses" you are referring to the CoTs generated by other methods, please note that these are not publicly available, and therefore we do not have access to them. This is, in fact, a quite common situation: authors which generate CoTs using non-deterministic sampling (i.e., with a temperature other than 0) very rarely make the CoTs generated by their models public. What is typically disclosed is the temperature value used, along with other details—such as the verifier and/or the prompt—which allow for a fair comparison of the resulting accuracy. For further details, please refer to App. B.