acceptodds
Under review as a conference paper at ICLR 2027

DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models

Abstract

Understanding and reasoning over long contexts is a crucial capability for language models (LMs). Although recent models support increasingly long context windows, their accuracy often deteriorates as input length grows: models struggle to keep attention focused on the most relevant context throughout decoding. We propose DySCO, a training-free decoding algorithm for long-context reasoning. At each decoding step, DySCO uses retrieval heads–a small subset of attention heads specialized for long-context retrieval—to identify task-relevant tokens, and explicitly up-weights attention to them across all heads. DySCO applies directly to off-the-shelf LMs and is compatible with FlashAttention, as its intervention reduces to a per-token logit bias inside the attention kernel. Across three model families, spanning instruction-tuned and reasoning models, DySCO consistently improves performance on challenging long-context reasoning benchmarks, with relative gains of 22% and 21% for Qwen3-8B on MRCR and LongBenchV2 at 128K context length. DySCO incurs about 1.8x decoding latency but makes more effective use of test-time compute, outperforming self-consistency baselines with up to 8 parallel samples. Further analysis shows that both dynamic rescaling and retrieval-head guided selection are important to its effectiveness, and provides insights into decoding-time attention behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Long-context, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WAc3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DySCO, a training-free intervention algorithm that estimates the relevance of historical tokens by extracting attention signals from internal retrieval heads (QRHeads) at each decoding step, and amplifies the attention logits of the most relevant tokens across all heads. It shows gains on long-context benchmarks, and the authors provide a FlashAttention-compatible implementation, offering a novel perspective on test-time compute optimization.

Critical strengths
  1. DySCO is pioneering in translating the internal mechanism of QRHeads into an adaptive decoding guide. The dynamic rescaling mechanism based on internal signals is highly intuitive and elegant.

  2. The algorithm is architecturally plug-and-play, and the authors make a clever low-level modification to FlashAttention, directly injecting into the tiled online Softmax computation. This design avoids the memory OOM and is a highly valuable engineering highlight.

  3. On long-context tasks, it performs remarkably well, not only effectively mitigating context decay but even surpassing an 8-way Self-Consistency majority-voting baseline.

Critical weaknesses
  1. The paper involves too many hyperparameters, which weakens the robustness of the method.

1.1 (Top-p and selection count) Although a threshold pp and a maximum cutoff KK are set, can the authors report how many tokens are actually selected per decoding step, to demonstrate the method's sparsity in real runtime?

1.2 (Smoothing parameter γ\gamma) The paper hard-codes γ\gamma to 0.40.4 without providing ablation experiments.

1.3 (Aggregation early-stopping layer) The paper specifies stopping at layer 17-20 on Qwen3-8B to extract QRHeads, but provides no ablation study. Besides, the layer distribution of retrieval heads differs across models (e.g., Llama, Mixtral) and scales (4B vs. 8B). This hard-coded layer choice hinders automatic transferability to low-cost environments. Is there a unified, automatic criterion for selecting the stopping layer to ensure cross-model transferability?

  1. The paper makes a highly clever bias injection into the FlashAttention online Softmax mechanism. Modifying the FlashAttention kernel is tricky and of significant engineering value. I would be glad if the authors could open-source the core CUDA/Triton injection code via an anonymous link during the rebuttal period.
Questions for authors

The method increases the attention weight of selected tokens at every step. From my own experience with such methods, this continuous intervention is highly likely to cause garbled output or model collapse.The authors employ a smoothing mechanism and a maximum intervention token cutoff to limit the scope of intervention. Beyond these two, do the authors have any additional operations for preventing collapse? Why does this global, persistent intervention not damage the model's basic generation capability?

2 comments
AuthorsOpenReview ↗

Thank you for your feedback!

Thank you for your thoughtful comments. Please find answers below.

Q1: The method introduces several hyperparameters, which may limit its robustness.

A1: Thank you for raising this point! We acknowledge that DySCO introduces several hyperparameters. However, we do not tune hyperparameters separately for each dataset. We also provide additional analyses in Appendix J to demonstrate DySCO's robustness to reasonably variant hyperparameter settings.

Q1 (a): How many tokens are actually selected per decoding step, to demonstrate the method's sparsity in real runtime?

A1 (a): Thank you for the suggestion. We report the fraction of tokens selected by DySCO using Qwen3-8B (Thinking). Following the hyperparameter settings in our paper, we use a selection threshold of p=0.975, with maximum selection budgets of 4,096 tokens at 64K and 8,192 tokens at 128K. The results show that DySCO selects only approximately 5.1–8.2% of available tokens, with the maximum top-k cutoff reached in around 30.0% of decoding steps.

Num. Selected TokensFraction of Selected TokensSteps Reaching Top-k Cutoff
MRCR 128K4,9785.1%31.8%
LongBench v2 128K5,0656.1%25.4%

Q1(b): How does the smoothing factor (momentum coefficient) affect DySCO's performance?

A1(b): Thank you for this question! In our preliminary experiments, we found that performance was relatively insensitive to the smoothing factor and warm-up window. We therefore fixed these hyperparameters across all experiments. We provide additional results on MRCR and LongBench v2 under different momentum coefficients, as shown below. The results show that performance remains relatively stable across different momentum values, with only modest variations on both benchmarks. We will incorporate this additional analysis into the revision.

MomentumMRCR 128KLongBench v2 128K
0.319.932.7
0.4 (Paper)19.634.7
0.519.034.7

Q1(c): Is the early stopping layer hardcoded? Is there a unified, automatic criterion for selecting the stopping layer to ensure cross-model transferability?

A1(c): Thank you for this question! We would like to clarify a small misunderstanding. The early stopping layer is not a hardcoded hyperparameter but is automatically determined based on the selected QRHeads. Specifically, following prior work on retrieval heads (Wu et al., 2024; Zhang et al., 2025), we select the top 16 QRHeads for each model and set the early stopping layer to the deepest layer containing any of these heads. This ensures that we collect attention weights from selected QRHeads without unnecessary computation in subsequent layers. The same selection criterion applies across different models without manually tuning the stopping layer. We will clarify this procedure in our revision.


Q2: Modifying the FlashAttention kernel requires substantial engineering effort. Could the authors open-source the core CUDA/Triton code for attention-bias injection through an anonymous repository link?

A2: Thank you for your question. We provide the implementation of attention-bias injection for flash attention in the supplementary materials.


Q3: Beyond the smoothing mechanism and maximum token-selection budget, are there any additional mechanisms to prevent generation collapse? Why does the global attention intervention not degrade the model's general generation capabilities?

A3: Thank you for this insightful question! We believe this is largely due to our relatively conservative intervention strategy, which uses a high top-p threshold (p >= 0.95) and moderate intervention strength.

In our preliminary experiments, we tested more aggressive interventions, such as a lower top-p threshold (p = 0.6) combined with a high intervention strength (greater than 5.0), and observed occasional repetition in generated outputs. Our final design adopts a higher top-p threshold to retain most of the attention mass, together with a moderate intervention strength to avoid excessively amplifying selected tokens. With these settings, we generally do not observe noticeable degradation in generation quality.

Thank you for the detailed response. I still have some concerns about the hyperparameters. Many choices remain insufficiently justified. For instance, if performance is insensitive to the smoothing parameter, what is the rationale for choosing 0.4, and can its necessity be demonstrated? Moreover, if smoothing is needed, why is the module apparently insensitive to this parameter? This seems somewhat contradictory. These questions also extend to other hyperparameters such as pp and KK. That said, I find the paper interesting and will raise my score to 3.

Reviewer_bC6vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DySCO, a training-free decoding method for improving long-context language model performance. Motivated by the observation that query-focused retrieval heads (QRHeads) retain useful relevance rankings even as attention mass on relevant context decreases, DySCO dynamically identifies important tokens from QRHead attention distributions and increases their attention weights across all heads during decoding. The method combines attention aggregation, momentum-based token selection, and additive attention-logit biasing. Experiments on Qwen3, Llama, and Mixtral models show improvements on long-context reasoning and retrieval tasks, supported by comparisons with uniform attention scaling, random-head and static-scaling ablations, self-consistency, and efficiency measurements.

Critical strengths

Clear motivation and a simple, effective intervention. The paper connects an empirical observation about retrieval-head behavior under increasing context length to a concrete decoding-time method. The attention intervention is mathematically well-defined, requires no parameter updates, and can be applied to existing models.

Strong empirical results across challenging long-context tasks. DySCO achieves substantial gains on reasoning-oriented benchmarks, including MRCR, LongBenchV2, and Clipper. For Qwen3-8B at 128K, the reported improvements over YaRN alone are 3.5 absolute points on MRCR and 6.1 points on LongBenchV2 (Section 4.1). The experiments also cover multiple model scales, direct-recall tasks, and an MoE architecture.

Meaningful ablations and practical implementation analysis. Table 3 evaluates both random retrieval-head selection and static token scaling, supporting the importance of QRHead-guided dynamic selection. The paper also includes hyperparameter robustness analyses, comparisons with self-consistency, and measured eager-attention and Flash-compatible implementations (Appendix D). These experiments substantially strengthen the empirical contribution.

Critical weaknesses

1. The mechanistic explanation is not fully established.

The Path Traversal analysis (Section 2.2, Figure 3) shows that QRHeads continue to rank relevant edges relatively highly as context length increases, while their absolute attention mass decreases. This is a useful observation, and the performance improvements from DySCO support the effectiveness of attention redistribution. However, the evidence does not establish reduced attention mass as the principal cause of long-context prediction failures. Other factors, including the quality of retrieved representations and downstream utilization of relevant information, may also contribute.

Furthermore, the diagnostic analysis focuses on a controlled path-retrieval task and a specific intermediate reasoning step. It is therefore unclear how directly the observed attention behavior explains improvements on more complex long-document reasoning tasks. A more direct connection between token selection, relevant evidence, and successful predictions would strengthen the mechanistic claims.

2. The necessity of applying retrieval-derived attention biases to all heads remains insufficiently isolated.

DySCO estimates token relevance using a small set of QRHeads but applies the resulting bias across all attention heads (Section 3.1). This is a reasonable design, but different heads may perform different computational functions, and a relevance signal derived from retrieval heads need not be equally beneficial across all heads.

The random-head and static-scaling ablations in Table 3 establish the importance of the relevance source and dynamic updating. However, they do not isolate the effect of the intervention scope. Comparing global rescaling with interventions restricted to QRHeads, selected layers, or non-retrieval heads would clarify whether global propagation is essential to the observed improvements.

3. The evaluation protocol and practical accuracy–latency trade-off require further clarification.

(a) Development and evaluation independence. DySCO's hyperparameters are selected using MRCR, which is also included among the main evaluation benchmarks (Section 4 and Appendix G). The authors describe the search procedure and apply the selected configurations to other datasets, which provides meaningful evidence of cross-task transfer. Nevertheless, the paper should explicitly clarify whether the MRCR instances used for hyperparameter selection are disjoint from those used for final evaluation. Without this information, the MRCR gains are more difficult to interpret as independent test results.

(b) Practical decoding cost. DySCO requires an additional partial forward pass during decoding. The authors transparently report measured eager-attention and Flash-compatible performance in Appendix D. For Qwen3-8B with a 128K-token input and 4K-token output, Flash-compatible throughput decreases from 13.9 to 7.9 tokens/s (Table 5). Thus, although the additional FLOPs are relatively small compared with long-context prefilling, the decoding-time slowdown remains substantial. A more explicit discussion of the accuracy–latency trade-off would help establish when DySCO is preferable to standard decoding or alternative test-time scaling strategies.

Questions for authors

1.Mechanistic evidence: Can the authors quantify the relationship between dynamically selected tokens and ground-truth relevant evidence across decoding steps, particularly on tasks beyond Path Traversal? To what extent do the results support attention-mass dilution as a causal explanation rather than a correlated symptom?

2.Intervention scope: How does DySCO perform when the attention bias is applied only to QRHeads or selected layers instead of all heads? Can such restricted interventions retain most of the accuracy gains while reducing decoding overhead?

3.Evaluation independence: Are the MRCR development instances used for hyperparameter selection strictly disjoint from the MRCR instances used in the final reported evaluation? How are the respective datasets constructed?

4.Efficiency: Given the measured Flash-compatible slowdown, under which input/output length regimes does DySCO offer the most favorable accuracy–latency trade-off? Can the authors clarify how the comparison with self-consistency changes when accounting for wall-clock latency and parallel execution?

4 comments
AuthorsOpenReview ↗

Thank you for your feedback!

Thank you for your valuable feedback. Please find answers below.

Q1: Can authors provide more mechanistic evidence? Can authors quantify the relationship between selected tokens and ground-truth relevant evidence across decoding steps on tasks beyond Path Traversal?

A1: Thank you for raising this point! To provide additional mechanistic evidence, we conduct token-level analyses on MRCR 128K using Qwen3-4B. We choose MRCR because it allows us to directly measure how DySCO's token selection aligns with ground-truth evidence.

We first first compare 16 QRHeads against 16 layer-matched randomly selected heads, using identical decoding states on vanilla decoding. At each decoding step, we select the top 4,096 tokens based on averaged attention scores. We then measure the fraction of ground-truth target tokens covered by this selection. As shown below, QRHeads achieve substantially higher target-token coverage, and are more effective at identifying relevant evidence than randomly selected heads.

HeadTarget-Token Coverage
QRHeads39.1%
Random Heads (Layer-Matched)22.9%

We further analyze the tokens selected for upweighting during DySCO intervention, using the hyperparameters from our main experiments (top-p = 0.975, top-k = 8,192). At each decoding step, we measure the fraction of tokens receiving an attention upweight separately within the target needles and background context, and average these fractions across decoding steps. The results (Table below) shows that tokens within the ground-truth target needles are much more likely to be upweighted than background tokens.

Tokens Upweighted
Needles47.51%
Background Context4.41%

These results demonstrate that DySCO preferentially identifies and upweights ground-truth relevant evidence during decoding. While we acknowledge that attention-mass dilution may be influenced by other confounding factors, our findings provide mechanistic evidence that DySCO effectively redirects attention toward relevant evidence, which leads to substantial performance improvements. We will incorporate these additional analyses and clarify the our mechanistic claims in our revisions.


Q2: How does DySCO perform when the attention bias is applied only to QRHeads or selected layers instead of all heads?

A2: Thank you for this insightful suggestion! Please refer to our general response on intervention scope. Our additional experiments show that restricting DySCO's intervention to later layers still substantially outperforms vanilla decoding, while incurring only modest additional latency (approximately 1.2×).


**Q3:**Are the MRCR development instances used for hyperparameter selection disjoint from the MRCR instances used in the final reported evaluation?

A3: Thank you for raising this point! We acknowledge that we tuned hyperparameters using the full MRCR validation set, since MRCR contains only 100 validation examples. We will explicitly clarify this limitation in the revised paper. We did not use the synthetic Path Traversal task for hyperparameter selection, as its original setup only supports context lengths up to 32K.

Nevertheless, we believe our results provide strong evidence that DySCO's improvements generalize.

  1. In our paper, DySCO also achieves substantial improvements on LongBench v2 and CLIPPER using the same hyperparameters selected on MRCR, without any benchmark-specific tuning. We additionally evaluate DySCO on LongMemEval (Wu et al., 2024) at 128K using Qwen3-8B (Thinking), directly transferring the hyperparameters without further tuning. As shown below, DySCO improves accuracy on this dataset as well.
LongMemEval 128K
Vanilla59.5
UniAttn59.0
DySCO63.5
  1. In Table 7 of the appendix, we evaluate DySCO under 5 different hyperparameter configurations. All configurations outperform vanilla decoding, and some even achieve better performance on benchmarks other than MRCR.

Overall, while we acknowledge the potential evaluation bias on MRCR, the consistent improvements across multiple benchmarks, together with the robustness across hyperparameters, suggest that DySCO's benefits are not due to overfitting to MRCR. We will clarify our hyperparameter selection procedure and incorporate the additional results into the revised paper. Thank you for the suggestion!


Q4: The comparison with self-consistency does not account for parallel decoding.

A4: Thank you for raising this point. Please refer to our general response on the additional comparisons with self-consistency.

Reviewer_bC6vOpenReview ↗

Follow-up on the Author Rebuttal

Thank you for your response to my question. The rebuttal provides useful additional evidence, particularly the intervention-scope ablations, which address Q2. However, three concerns remain.

Q1: Mechanistic evidence. The new analyses support relevant-token selection but do not establish that attention dilution causes prediction failures, as suggested by the causal language in Sections 1 and 2.2.

Q3: Evaluation independence. Although Section 4 identifies MRCR as a development dataset, the authors confirm that the full MRCR validation set was used for hyperparameter selection. Thus, the prominently reported MRCR gains lack independent evaluation. Results on other benchmarks provide evidence of cross-task transfer but do not establish generalization to unseen MRCR examples.

Q4: Efficiency. The additional comparisons clarify the trade-off with batched self-consistency, but full DySCO still incurs approximately 1.7–1.8× the decoding latency of vanilla decoding, with a similar slowdown across the input and output lengths reported in Table 6. The restricted variants reduce latency but retain only part of the accuracy gains.

Gven the remaining limitations, I am inclined to maintain my rating of 2.

AuthorsOpenReview ↗

Thank you for your follow up

Thank you for your follow-up. We would like to further clarify the remaining points.

Q1: Mechanistic evidence: The new analysis do not establish that attention dilution causes prediction failures

A1 We would like to clarify that our analyses and intervention-scope ablations provide additional direct evidence that attention dilution is an important contributing factor to prediction failures, and causally correcting attention allocation substantially mitigates these failures. Specifically, rescaling attention only within QRHeads yields smaller improvements than applying the same rescaling across all attention heads. Importantly, DySCO directly intervenes on attention allocation without modifying model parameters. The consistent performance improvements from this intervention, particularly the additional gains from global rescaling over QRHead-only rescaling, provide causal evidence that insufficient attention to relevant information across layers contributes to long-context failures.

MethodMRCR 128KLongBench v2 128K
No Intervention (Vanilla)16.128.6
QROnly Intervention17.932.1
Full Intervention19.634.7

In addition, we do not claim that attention dilution as the only exclusive cause of such failures, and attention dilution itself may arise from other factors, such as embedding collapse. Nevertheless, our combined evidence (selection evidence alignment and intervention-scope ablations), provides strong support for attention dilution as an important and actionable contributing factor. We will clarify this distinction in the revised paper. Thanks again for your suggestion.


Q3: While the results demonstrate cross-task transfer. Do DySCO's improvements persist when hyperparameters are selected on a separate MRCR validation set?

A3: Thank you for your follow-up. We test using disjoint validation and test sets to test independent evaluation. Given the limited number of MRCR samples, we perform 5 random trials, each independently resampling 150 samples for validation and 150 for testing. Thus, both vanilla and DySCO performance may differ across rows due to different test splits.

We search over top-p from [0.95, 0.975], top-k [4096, 8192], and strength [2.0, 2.5]. For each trial, we select params on the validation split and report performance on the held-out test split, along with results on LongBench v2 128K (same set).

Trial(p, k, strength)MRCR 128K VanillaMRCR 128K DySCOLB 128K VanillaLB 128K DySCO
1(0.95, 8192, 2.5)15.919.228.631.7
2(0.975, 8192, 2.5)16.320.528.634.7
3(0.975, 8192, 2.5)17.018.628.634.7
4(0.95, 8192, 2.5)17.218.128.631.7
5(0.975, 8192, 2.5)15.319.828.634.7

The results show that hyperparameter selection is stable across different splits, with only variation in top-p. DySCO consistently outperforms vanilla inference on independent test splits, with gains ranging from 0.9 to 4.5 points. The improvements also transfer to LongBench v2. We will clarify these results in our revision.


Q4: Full DySCO incurs additional latency, while restricted variants reduce overhead but retain only part of the accuracy gains.

A4: Thank you for the follow-up. We would like to emphasize that our additional experiments explicitly compare DySCO against batched self-consistency at matched wall-clock latency, accounting for parallel decoding. DySCO outperforms self-consistency (M4, B2) under the same latency budget, improving performance on both MRCR and LongBenchv2. In addition, our late-layer variants offer a practical alternative for latency-sensitive settings. Specifically, the restricted variant matches self-consistency's performance on MRCR while achieving approximately 33% lower wall-clock latency.

We would also like to clarify that DySCO is explicitly designed to trade additional latency for improved accuracy. Our comparisons demonstrate that this additional cost is justified by stronger performance than self-consistency at matched wall-clock latency. We would appreciate your further clarification on which aspects of the accuracy–latency trade-off remain insufficiently supported, or what additional comparisons would affect your assessment. We will clarify this trade-off in the revised paper. Thank you for your suggestion.

Thank you for the additional experiments and clarifications. The new MRCR results substantially address my concern about evaluation independence, but the causal claims about attention dilution and the latency overhead of full DySCO remain concerns. I appreciate the authors' efforts, would prefer to maintain my current rating of 2.

Reviewer_kETeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DySCO, a training-free decoding method for long-context language models. At each decoding step, DySCO runs a partial forward pass up to the middle layers to aggregate attention scores from pre-identified retrieval heads, selects top context tokens with momentum smoothing, and runs a full forward pass that adds a positive logit bias to those tokens across all heads. Experiments on Qwen3, Llama-3.1, and Mixtral show accuracy gains on long-context reasoning benchmarks.

Critical strengths
  • The diagnostic study on path traversal clearly shows that retrieval heads maintain high relative ranking of relevant spans over long contexts even as their absolute attention mass gets diluted.
  • The method requires no model training and integrates cleanly into FlashAttention by adding a per-token bias inside the online softmax update.
  • Evaluations across multiple model families show consistent accuracy gains on multi-step chain-of-thought reasoning tasks at 64K and 128K context lengths.
Critical weaknesses
  • Running layers 1 through 20 twice at every decoding step is computationally wasteful and causes a 1.8x decoding slowdown, taking over 20 minutes per query at 128K context in Table 6.
  • Hyperparameters are tuned directly on MRCR-64K and MRCR-128K as described in Section 4 and Appendix G, which invalidates MRCR as an unbiased evaluation benchmark.
  • Broadcasting the same logit bias to every head across all layers disrupts non-retrieval heads, causing net accuracy degradation on LongBenchV2 when chain-of-thought is disabled in Figure 6.
  • Gains on 128K direct recall tasks in Table 2 are marginal, with Qwen3-32B improving by only 0.2 points on average and degrading on InfMC.
  • The comparison against self-consistency overlooks batched parallel decoding with shared prefix caches and relies on multiple-choice tasks where small-sample voting easily ties.
Questions for authors
  1. Since retrieval heads reside in layers 17 to 20, why not apply the attention bias only to layers 21 and above within a single forward pass instead of recomputing the first 20 layers from scratch at every token?
  2. Because tuning hyperparameters directly on MRCR-64K and MRCR-128K leaks test benchmark information, how much accuracy gain remains on MRCR and LongBenchV2 if hyperparameters are selected strictly on the synthetic path traversal task?
  3. In Figure 6, DySCO hurts LongBenchV2 accuracy when thinking mode is turned off. Does restricting the logit intervention to a subset of middle or upper layers prevent this degradation?
2 comments
AuthorsOpenReview ↗

Thank you for your feedback!

Thank you for your thoughtful comments and feedback! Please find the answers below.

Q1: Why not apply the attention bias only to layers 21 and above within a single forward pass, rather than recomputing the first 20 layers at every decoding step?

A1: Thank you for this insightful suggestion! Please refer to our general response on intervention scope. Our additional experiments show that restricting DySCO's intervention to later layers still substantially outperforms vanilla decoding, while incurring only modest additional latency (approximately 1.2×).


Q2: In Figure 6, DySCO reduces LongBench v2 accuracy when thinking mode is disabled. Does restricting the intervention to a subset of layers prevent this degradation?

A2: Thank you for this valuable suggestion! Our additional experiments show that restricting DySCO's intervention to later layers mitigate the performance degradation observed on LongBench v2 in the non-thinking setting. As shown below, the narrower intervention scope variants match vanilla decoding. We will integrate this analysis in our revised paper.

MethodLongBench v2 64KLongBench v2 128K
Vanilla38.632.6
DySCO (Full)37.431.6
DySCO (18–36)39.632.6
DySCO (24–36)38.632.6
DySCO (QRHeads only)39.133.7

Q3: Does tuning hyperparameters on MRCR introduce bias into the reported MRCR results?

A3: Thank you for raising this point! We acknowledge that we tuned hyperparameters using the full MRCR validation set, since MRCR contains only 100 validation examples. We will explicitly clarify this procedure in the revised paper. We did not use the synthetic Path Traversal task for hyperparameter selection, as its original setup only supports context lengths up to 32K.

Nevertheless, we believe our results provide strong evidence that DySCO's improvements generalize.

  1. In our paper, DySCO also achieves substantial improvements on LongBench v2 and CLIPPER using the same hyperparameters selected on MRCR, without any benchmark-specific tuning. We additionally evaluate DySCO on LongMemEval (Wu et al., 2024) at 128K using Qwen3-8B (Thinking), directly transferring the hyperparameters without further tuning. As shown below, DySCO improves accuracy on LomgMevlEval
LongMemEval 128K
Vanilla59.5
UniAttn59.5
DySCO63.5
  1. In Table 7 of the appendix, we evaluate DySCO under 5 different hyperparameter configurations. All configurations outperform vanilla decoding, and some even achieve better performance on benchmarks other than MRCR.

Overall, while we acknowledge the potential evaluation bias on MRCR, the consistent improvements across multiple benchmarks, together with the robustness across hyperparameters, suggest that DySCO's benefits are not due to overfitting to MRCR. We will clarify our hyperparameter selection procedure and incorporate the additional results into the revised paper. Thank you for the suggestion!


Q4: The comparison with self-consistency does not account for parallel decoding. Voting on multiple choice questions with a small number of samples can result in ties.

A4: Thank you for raising this point. Please refer to our general response on the additional comparisons with self-consistency. We show DySCO can be more efficient than self-consistency on MRCR as well under the same GPU setting.

Thank you for your review.

Regarding Q3 (Does tuning hyperparameters on MRCR introduce bias into the reported MRCR results?). We would like to provide results that use disjoint validation and test sets to test independent evaluation. Given the limited number of MRCR samples, we perform 5 random trials, each independently resampling 150 samples for validation and 150 for testing. Thus, both vanilla and DySCO performance may differ across rows due to different test splits.

We search over top-p from [0.95, 0.975], top-k [4096, 8192], and strength [2.0, 2.5]. For each trial, we select params on the validation split and report performance on the held-out test split, along with results on LongBench v2 128K (same set).

Trial(p, k, strength)MRCR 128K VanillaMRCR 128K DySCOLB 128K VanillaLB 128K DySCO
1(0.95, 8192, 2.5)15.919.228.631.7
2(0.975, 8192, 2.5)16.320.528.634.7
3(0.975, 8192, 2.5)17.018.628.634.7
4(0.95, 8192, 2.5)17.218.128.631.7
5(0.975, 8192, 2.5)15.319.828.634.7

The results show that hyperparameter selection is stable across different splits, with only variation in top-p. DySCO consistently outperforms vanilla inference on independent test splits, with gains ranging from 0.9 to 4.5 points. The improvements also transfer to LongBench v2. We will clarify these results in our revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.