- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper argues that mixing candidate embeddings at the input can lead to single-candidate dominance during layerwise processing. It proposes DIVER, which independently processes candidate tokens before fusing their attention Values at selected internal layers. The experiments show improvements across multiple models and benchmarks. However, it remains unclear whether the suppressed candidates contain useful information and whether the performance gains justify the additional computation. Stronger accuracy improvements or lower computational costs would strengthen the contribution.
Critical strengths
-
The paper is well motivated by a concrete layerwise analysis rather than intuition alone.
-
The experimental coverage is broad enough to make the gains credible. DIVER is evaluated on seven backbones from 1.7B to 32B, on math, science, and code benchmarks, and achieves the highest average accuracy on every backbone.
Critical weaknesses
-
The fusion layers are selected by a per-model calibration procedure on 1,000 synthetic arithmetic tasks and then fixed for all domains. Performance is sensitive to this choice (Table 4: fusing at layers 1–4 or all layers is yields lower average accuracy than 21–24), but the paper does not examine whether the selection is robust: it does not report what layers would be chosen when calibrating on science or code examples, nor how accuracy changes when the window is shifted by a few layers.
-
The accuracy gains come with additional computational costs. The current comparisons do not sufficiently establish whether DIVER offers a better accuracy–runtime trade-off than alternatives under a comparable budget.
Questions for authors
-
Are candidates whose influence weakens under embedding mixing more likely to lead to correct or efficient solutions? Can the authors analyze their downstream outcomes?
-
Under the same runtime budget, does DIVER outperform CoT with multiple samples and voting, or other inference-time methods?
-
Fusion layers are selected using simple arithmetic tasks. Although these layers yield gains across several benchmarks, how sensitive are the selected layers and final performance to the calibration samples or task domain?
5 comments
Response to Reviewer PHJ9 [2/2]
W2&Q2: DIVER's performance and efficiency advantages under comparable computational budgets
To better evaluate the trade-off between accuracy and computation time, we have added experiments under comparable computational budgets, comparing DIVER with the CoT Sampling and Soft Thinking baselines combined with Self-Consistency.
First, we compare DIVER, CoT Sampling+SC@3, and Soft Thinking+SC@3 on AIME 2024 and AIME 2025. We use SC@3 as a common low-budget setting for both baselines. At this setting, both baselines have runtimes close to that of DIVER. Each method is evaluated eight times, and we report the average accuracy and average runtime across the two datasets.
| Method | Model | Average Time | Average Accuracy |
|---|---|---|---|
| CoT Sampling + SC@3 | Qwen3-4B-Thinking-2507 | 41.26 min | 79.79% |
| Soft Thinking + SC@3 | Qwen3-4B-Thinking-2507 | 51.88 min | 80.63% |
| DIVER (ours) | Qwen3-4B-Thinking-2507 | 50.21 min | 81.67% |
| CoT Sampling + SC@3 | Qwen3-8B | 38.99 min | 75.83% |
| Soft Thinking + SC@3 | Qwen3-8B | 48.44 min | 73.33% |
| DIVER (ours) | Qwen3-8B | 37.68 min | 75.21% |
Under comparable computational budgets, DIVER improves accuracy on Qwen3-4B-Thinking-2507 by 1.04 and 1.88 percentage points over Soft Thinking+SC@3 and CoT Sampling+SC@3, respectively. On Qwen3-8B, DIVER achieves performance comparable to CoT Sampling+SC@3 and improves accuracy over Soft Thinking+SC@3 by 1.88 percentage points. These results demonstrate that DIVER exhibits good performance.
More importantly, DIVER can also be combined with self-consistency method. We therefore further compare DIVER+SC@8, CoT Sampling+SC@30, and Soft Thinking+SC@30 on AIME 2024 and AIME 2025, reporting the average accuracy and average runtime across the two datasets. Under this setting, DIVER+SC@8 achieves higher accuracy than both baselines on both models.
| Model | Method | Time | Accuracy |
|---|---|---|---|
| Qwen3-4B-Thinking-2507 | CoT Sampling + SC@30 | 6.91h | 88.33% |
| Qwen3-4B-Thinking-2507 | Soft Thinking + SC@30 | 8.62h | 88.33% |
| Qwen3-4B-Thinking-2507 | DIVER + SC@8 | 6.69h | 90% |
| Qwen3-8B | CoT Sampling + SC@30 | 6.38h | 80.00% |
| Qwen3-8B | Soft Thinking + SC@30 | 8.03h | 78.33% |
| Qwen3-8B | DIVER + SC@8 | 5.02h | 83.33% |
These results demonstrate that DIVER not only exhibits good performance but can also be combined with self-consistency voting. We have added these supplementary experiments to the appendix of the paper and highlighted them in blue.
Q1: Are candidates whose influence is weakened by embedding mixing more likely to produce correct or efficient solutions?
Predictions after embedding mixing tend to favor the top-1 candidate, while other candidates have relatively less influence. To examine the effects of these candidates with weakened influence on subsequent reasoning, we select the 1st, 10th, and 50th high-entropy positions in the reasoning trajectories for the 30 questions in AIME 2025, replace the top-1 token at each selected position with the top-2 or top-3 token, and then continue generation. We record the final accuracy, the average number of generated tokens, and the numbers of problems that change from incorrect to correct and from correct to incorrect relative to the baseline.
| Replacement Position | Candidate | Accuracy | Average Number of Generated Tokens | Baseline Incorrect → Correct | Baseline Correct → Incorrect |
|---|---|---|---|---|---|
| No replacement | top-1 baseline | 66.67% | 18,728 | – | – |
| 1st high-entropy position | top-2 | 70.00% | 16,671 | 4 | 3 |
| 1st high-entropy position | top-3 | 63.33% | 19,027 | 2 | 3 |
| 10th high-entropy position | top-2 | 63.33% | 18,601 | 2 | 3 |
| 10th high-entropy position | top-3 | 70.00% | 17,624 | 3 | 2 |
| 50th high-entropy position | top-2 | 70.00% | 18,099 | 3 | 2 |
| 50th high-entropy position | top-3 | 56.67% | 19,434 | 1 | 4 |
The results show that candidates other than top-1 can lead to correct solutions that the baseline fails to obtain, but replacing top-1 can also turn previously correct answers into incorrect ones. For example, selecting top-2 at the first high-entropy position corrects four questions answered incorrectly by the baseline while causing three previously correct answers to become incorrect. The average generation length decreases from 18,728 to 16,671. These findings indicate that alternative candidates can support successful reasoning, while the top-1 candidate also remains valuable. This motivates DIVER’s design of combining information from top-1 and alternative candidates to preserve it for subsequent reasoning.
I thank the authors for their detailed response and the substantial effort devoted to the additional experiments. These results are helpful, but several concerns remain.
-
The accuracy–cost advantage remains unclear. (W2, Q2) At the lower budget, DIVER performs similarly to CoT+SC@3: on Qwen3-8B, accuracy is slightly lower (75.21 vs. 75.83); on Qwen3-4B, the improvement is 1.88 percentage points with approximately 22% more runtime. At the higher budget, Figure 6 appears to suggest that CoT+SC largely plateaus around SC@8, making a comparison at that operating point particularly relevant. A fuller accuracy–runtime curve would help establish the trade-off more clearly. The higher-budget accuracy margins also correspond to only one or two questions out of 60, which makes uncertainty estimates especially valuable.
-
The statistical robustness of the improvements is difficult to assess. (W2/Q2) The reported average gains over CoT are 1.15–2.90 percentage points. Although I appreciate the eight-run evaluations added for some comparisons, means alone do not show how consistently these gains hold. For the same default Qwen3-8B configuration, the single-run AIME 2024 ablation result is 83.33, whereas the eight-run mean is 78.75. This difference does not invalidate the results, but it makes reporting variability particularly relevant when interpreting relatively small improvements.
-
The layer-selection results and candidate analysis leave some questions unresolved. (W1, Q3; Q1) The selected window shows substantially higher patching recovery than adjacent windows (approximately 72–74% vs. 5–15%), yet the corresponding downstream accuracy differences are much smaller; a shifted window also performs better on Qwen3-8B GPQA. Thus, the connection between the recovery measure and downstream performance remains unclear to me. Similarly, the token-replacement experiment reports 15 incorrect-to-correct changes and 17 correct-to-incorrect changes. This shows that alternative candidates can affect outcomes, but provides limited evidence that preserving them yields a systematic benefit.
Beyond the issues raised in my initial review, I also have some secondary concerns about the single-run ablations, the unclear criterion for choosing K. On reconsideration, I feel my initial soundness assessment was somewhat optimistic. I will maintain my original weak-reject recommendation.
Response to Reviewer PHJ9 [1/2]
We thank the reviewer for the positive comments on our research motivation, layerwise analysis, and experimental coverage. Below, we provide responses to the reviewer’s questions about fusion layer selection, computational cost, and the downstream effects of alternative candidates.
W1&Q3: Fusion layer selection and sensitivity to window shifts
To examine whether the Value-path patching results depend on the task domain, we repeat the analysis on a controlled code dataset, keeping all other settings the same as in the arithmetic experiments. The table below reports recovery toward the second candidate. For both models, recovery at L21–24 is notably higher than at either adjacent window, providing support for retaining this window when moving from the controlled arithmetic setting to the controlled code setting.
Moreover, to evaluate the robustness of fusion layer selection, we shift the original L21–24 window four layers earlier and later. We compare L17–20, L21–24, and L25–28 on mathematical reasoning, scientific question answering, and code generation tasks using Qwen3-4B-Thinking-2507 and Qwen3-8B. Since AIME 2024 is a small dataset, we perform eight evaluation runs on this dataset and report the mean accuracy. The results are as follows:
QWEN 4B Thinking 2507
QWEN 8B
On both models, the original L21–24 configuration achieves the highest average accuracy across tasks. This indicates that our selected fusion layers exhibit good robustness across different tasks. We have added the relevant experiments and discussion to the paper and highlighted them in blue.