- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DySCO, a training-free intervention algorithm that estimates the relevance of historical tokens by extracting attention signals from internal retrieval heads (QRHeads) at each decoding step, and amplifies the attention logits of the most relevant tokens across all heads. It shows gains on long-context benchmarks, and the authors provide a FlashAttention-compatible implementation, offering a novel perspective on test-time compute optimization.
Critical strengths
-
DySCO is pioneering in translating the internal mechanism of QRHeads into an adaptive decoding guide. The dynamic rescaling mechanism based on internal signals is highly intuitive and elegant.
-
The algorithm is architecturally plug-and-play, and the authors make a clever low-level modification to FlashAttention, directly injecting into the tiled online Softmax computation. This design avoids the memory OOM and is a highly valuable engineering highlight.
-
On long-context tasks, it performs remarkably well, not only effectively mitigating context decay but even surpassing an 8-way Self-Consistency majority-voting baseline.
Critical weaknesses
- The paper involves too many hyperparameters, which weakens the robustness of the method.
1.1 (Top-p and selection count) Although a threshold and a maximum cutoff are set, can the authors report how many tokens are actually selected per decoding step, to demonstrate the method's sparsity in real runtime?
1.2 (Smoothing parameter ) The paper hard-codes to without providing ablation experiments.
1.3 (Aggregation early-stopping layer) The paper specifies stopping at layer 17-20 on Qwen3-8B to extract QRHeads, but provides no ablation study. Besides, the layer distribution of retrieval heads differs across models (e.g., Llama, Mixtral) and scales (4B vs. 8B). This hard-coded layer choice hinders automatic transferability to low-cost environments. Is there a unified, automatic criterion for selecting the stopping layer to ensure cross-model transferability?
- The paper makes a highly clever bias injection into the FlashAttention online Softmax mechanism. Modifying the FlashAttention kernel is tricky and of significant engineering value. I would be glad if the authors could open-source the core CUDA/Triton injection code via an anonymous link during the rebuttal period.
Questions for authors
The method increases the attention weight of selected tokens at every step. From my own experience with such methods, this continuous intervention is highly likely to cause garbled output or model collapse.The authors employ a smoothing mechanism and a maximum intervention token cutoff to limit the scope of intervention. Beyond these two, do the authors have any additional operations for preventing collapse? Why does this global, persistent intervention not damage the model's basic generation capability?
2 comments
Thank you for the detailed response. I still have some concerns about the hyperparameters. Many choices remain insufficiently justified. For instance, if performance is insensitive to the smoothing parameter, what is the rationale for choosing 0.4, and can its necessity be demonstrated? Moreover, if smoothing is needed, why is the module apparently insensitive to this parameter? This seems somewhat contradictory. These questions also extend to other hyperparameters such as and . That said, I find the paper interesting and will raise my score to 3.
Thank you for your feedback!
Thank you for your thoughtful comments. Please find answers below.
Q1: The method introduces several hyperparameters, which may limit its robustness.
A1: Thank you for raising this point! We acknowledge that DySCO introduces several hyperparameters. However, we do not tune hyperparameters separately for each dataset. We also provide additional analyses in Appendix J to demonstrate DySCO's robustness to reasonably variant hyperparameter settings.
Q1 (a): How many tokens are actually selected per decoding step, to demonstrate the method's sparsity in real runtime?
A1 (a): Thank you for the suggestion. We report the fraction of tokens selected by DySCO using Qwen3-8B (Thinking). Following the hyperparameter settings in our paper, we use a selection threshold of p=0.975, with maximum selection budgets of 4,096 tokens at 64K and 8,192 tokens at 128K. The results show that DySCO selects only approximately 5.1–8.2% of available tokens, with the maximum top-k cutoff reached in around 30.0% of decoding steps.
Q1(b): How does the smoothing factor (momentum coefficient) affect DySCO's performance?
A1(b): Thank you for this question! In our preliminary experiments, we found that performance was relatively insensitive to the smoothing factor and warm-up window. We therefore fixed these hyperparameters across all experiments. We provide additional results on MRCR and LongBench v2 under different momentum coefficients, as shown below. The results show that performance remains relatively stable across different momentum values, with only modest variations on both benchmarks. We will incorporate this additional analysis into the revision.
Q1(c): Is the early stopping layer hardcoded? Is there a unified, automatic criterion for selecting the stopping layer to ensure cross-model transferability?
A1(c): Thank you for this question! We would like to clarify a small misunderstanding. The early stopping layer is not a hardcoded hyperparameter but is automatically determined based on the selected QRHeads. Specifically, following prior work on retrieval heads (Wu et al., 2024; Zhang et al., 2025), we select the top 16 QRHeads for each model and set the early stopping layer to the deepest layer containing any of these heads. This ensures that we collect attention weights from selected QRHeads without unnecessary computation in subsequent layers. The same selection criterion applies across different models without manually tuning the stopping layer. We will clarify this procedure in our revision.
Q2: Modifying the FlashAttention kernel requires substantial engineering effort. Could the authors open-source the core CUDA/Triton code for attention-bias injection through an anonymous repository link?
A2: Thank you for your question. We provide the implementation of attention-bias injection for flash attention in the supplementary materials.
Q3: Beyond the smoothing mechanism and maximum token-selection budget, are there any additional mechanisms to prevent generation collapse? Why does the global attention intervention not degrade the model's general generation capabilities?
A3: Thank you for this insightful question! We believe this is largely due to our relatively conservative intervention strategy, which uses a high top-p threshold (p >= 0.95) and moderate intervention strength.
In our preliminary experiments, we tested more aggressive interventions, such as a lower top-p threshold (p = 0.6) combined with a high intervention strength (greater than 5.0), and observed occasional repetition in generated outputs. Our final design adopts a higher top-p threshold to retain most of the attention mass, together with a moderate intervention strength to avoid excessively amplifying selected tokens. With these settings, we generally do not observe noticeable degradation in generation quality.