- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies evidence use in distractor-rich contexts through residual-stream measurements and controlled evidence relocation. It proposes ResiBand, which applies causal local subtraction followed by exponential smoothing and a learned low-rank residual correction, while keeping pretrained backbone weights frozen. Across three backbones, the reported three-seed mean multi-hop QA improvement over the specified Q/V-LoRA baseline is 19.30, 10.95, and 1.93 percentage points, respectively. Operator ablations, evidence-relocation experiments, and interventions examine how filtering and learned residual writes affect evidence discrimination. The results support the utility of the proposed adaptation pathway, while the specific causal role of local common-mode dominance remains less firmly established.
Critical strengths
-
A useful adaptation method supported by comparisons across backbones. Table 1 reports positive mean improvements over the specified LoRA baseline on all nine multi-hop backbone–benchmark combinations, with fewer trainable parameters. On Qwen3-1.7B, Table 3 also compares against an adapter with the same rank, parameter count, and training budget: ResiBand reaches a multi-hop average of 22.41 versus 12.00. This control supports a contribution from the filtered input beyond simply adding trainable residual capacity.
-
Controlled diagnostics that go beyond attention visualization. The relocation experiments preserve the question, answer, document/token multisets, and overall length while exchanging support and control positions. The paper combines these measurements with generated-answer performance, K/V deletion, and learned-write interventions. For example, Table 2 reports mean F1 of 0.447 for ResiBand versus 0.191 for adapter-only on the controlled relocation suite, with a paired source-bootstrap interval for the difference. This provides meaningful evidence of improved behavior in the constructed setting.
Critical weaknesses
-
The proposed causal explanation is not fully isolated. Sections 3 and 5.5 establish associations among residual statistics, routing, and answer discrimination, and show causal effects of enabling the complete trained module. However, the complete module changes both the temporal representation and the learned residual correction. These interventions do not specifically establish that suppressing the measured common-mode component causes the behavioral gain. Likewise, the increase in the contrast/common-mode ratio after filtering is compatible with the filter’s construction. The Q/K/V patches in Appendix C.7 provide additional causal evidence, but do not resolve this identification issue; their largest reported single-layer effect is 11.6% of the full On–Off margin gain. This does not rule out a distributed mechanism, but it limits how strongly the particular causal account is established.
-
The scope of the comparative claims exceeds the strongest available controls. Table 3 supports filtering relative to the unfiltered adapter, but the incremental multi-hop gain of the complete band over high-pass alone is 1.48 points, measured with one training seed. Its robustness is therefore unclear. Furthermore, the LoRA results establish superiority under the reported fixed Q/V adaptation contract, rather than over well-tuned PEFT generally. Appendix B specifies method-specific learning rates but leaves important optimization, prompting, and generation/scoring details unspecified. These limitations matter when judging which part of the architecture is necessary and how broadly its comparative advantage should be interpreted.
Questions for authors
-
Mechanism clarification: Which reported intervention specifically distinguishes common-mode suppression from other changes introduced by the trained residual branch? If this distinction is not identified by the current experiments, would you narrow the causal interpretation while retaining the supported behavioral and diagnostic findings?
-
Component contribution: Do you have paired results across training seeds for the full-band versus high-pass-only comparison? If not, can you qualify the claim that the longer-horizon smoothing stage contributes reliably, given the 22.41 versus 20.93 single-seed result in Table 3?
-
Comparison and reproducibility clarification: Please specify the optimizer/schedule, development procedure for the method-specific learning rates, prompt templates, generation limits and stopping rules, and completion-boundary scoring rules. Were these evaluation choices identical across arms? Please also clarify whether the inference-region masks used for deployment and latency reporting were selected on a separate development set.
I would be happy to raise my score if some of the questions or concerns are addressed.
9 comments
Response to Reviewer dMpz (Part 2)
Please specify the optimizer/schedule, development procedure for the method-specific learning rates, prompt templates, generation limits and stopping rules, and completion-boundary scoring rules. Were these evaluation choices identical across arms? Please also clarify whether the inference-region masks used for deployment and latency reporting were selected on a separate development set.
Thank you for raising these reproducibility and comparison concerns. Within each dataset and backbone, all methods used the same training budget, prompt template, and evaluation procedure. We followed a standard adapter-training setup with AdamW and cosine decay. Generation was greedy, limited to 64 new tokens, and stopped at EOS or the length limit. We scored only the generated suffix after the prompt boundary using normalized whole-completion token F1. These choices were identical across all comparison arms.
We used for LoRA, following a commonly recommended adapter learning rate. ResiBand learns an additive correction from normalized, filtered representations and does not use LoRA's explicit scaling; we therefore fixed its learning rate at as an adapter-style heuristic. Both values were fixed across backbones rather than selected using the evaluation sets.
To check whether the comparison was caused by an unfavorable LoRA learning rate, we additionally performed a five-point, three-seed sensitivity sweep on Qwen3-1.7B ( to ), while keeping the LoRA architecture, training budget, and evaluation protocol unchanged.
| Method | LR | MuSiQue | HotpotQA | 2Wiki | MultiHop avg. | GSM8K | MMLU |
|---|---|---|---|---|---|---|---|
| LoRA | 1e-5 | 3.97 ± 0.21 | 7.60 ± 0.16 | 7.14 ± 0.23 | 6.23 ± 0.19 | 52.61 ± 1.11 | 35.45 ± 0.27 |
| LoRA | 3.16e-5 | 5.34 ± 0.13 | 9.52 ± 0.39 | 7.49 ± 0.48 | 7.45 ± 0.31 | 47.55 ± 2.17 | 35.75 ± 0.10 |
| LoRA | 1e-4 | 6.59 ± 0.68 | 9.76 ± 0.59 | 7.86 ± 0.30 | 8.07 ± 0.38 | 49.57 ± 3.93 | 36.36 ± 0.71 |
| LoRA | 3.16e-4 | 7.16 ± 0.36 | 9.99 ± 0.10 | 8.05 ± 0.07 | 8.40 ± 0.08 | 51.46 ± 3.45 | 36.94 ± 0.07 |
| LoRA | 1e-3 | 7.64 ± 0.28 | 10.11 ± 0.57 | 8.29 ± 0.13 | 8.68 ± 0.31 | 49.72 ± 0.82 | 37.26 ± 0.63 |
| ResiBand | 3e-4 | 16.10 ± 4.98 | 36.29 ± 6.40 | 29.75 ± 7.43 | 27.38 ± 6.10 | 51.21 ± 2.35 | 37.65 ± 0.30 |
ResiBand remains substantially stronger on MultiHop than every LoRA learning rate tested (27.38 versus a maximum of 8.68), while remaining comparable on GSM8K and MMLU. Thus, the advantage cannot be explained solely by insufficient learning-rate tuning of the reported LoRA baseline.
Regarding the inference-region masks, these were introduced as a layer ablation to understand where ResiBand corrections contribute. ResiBand reconditions the residual stream to help frozen downstream layers use available evidence more effectively. This ablation shows that, across all three backbones, early and middle regions yield the two highest multi-hop averages, while late-only execution is weakest. This identifies intervention depth as an important design choice and motivates early-to-middle placement as a starting point for follow-up residual-correction methods. The original latency masks followed these ablation findings; they were not selected on a separate development set.
We hope the additional results help address your concerns and give you greater confidence in our work. Thank you again for your thoughtful feedback and consideration!
Looking forward to your reply
Dear Reviewer dMpz,
As the rebuttal deadline approaches, could you please take a look at our responses? We are looking forward to your reply!
Feel free to let us know if you have any other concerns. Thanks for your time and effort!
Best Regards,
Authors of Submission 32574
Response to Reviewer dMpz (Part 1)
Thank you for your thoughtful and constructive review.
Thank you for raising this important question. It helped us strengthen the causal evidence for our proposed mechanism. To test the contribution of common-mode suppression, we restore the suppressed component with all other conditions held fixed. Specifically, we replace Lτ(h−Lϵ(h)) with Lτ(h).
Results on Qwen3-1.7B , reevaluated on an RTX 4090, are shown below using whole-completion F1:
Restoring the common-mode component sharply reduces F1 on all three benchmarks. This provides direct intervention evidence that common-mode suppression contributes to ResiBand's performance.
The independent 200-source diagnostic cohort connects this effect to evidence routing: compared with unmodified ResiBand, restoring the common-mode component reduces support-minus-control attention-mass contrast by 0.0525 and the gold-versus-counter first-answer-token log-probability margin by 7.29. These interventions directly support a causal contribution of common-mode suppression to evidence use in the trained ResiBand model.Together, these results support a causal contribution of common-mode suppression: restoring the shared background weakens evidence routing and reduces answer correctness.
Thank you for raising this point. In this work, we focus on a gap between evidence access and effective use: relevant evidence remains available, but the model struggles to distinguish it from distractors. We find that its contrast with distractors can be weak relative to a dominant locally shared component in the residual stream. This motivates a simple, targeted design: subtract the short-horizon average to suppress this shared background and expose the remaining contrast. The longer-horizon smoothing stage then serves to stabilize the resulting signal, aggregating persistent variation and attenuating isolated token-wise fluctuations before the learned residual write.
The incremental benefit of this complementary stabilization stage is model-dependent. The original ablation in Table 3 used one backbone and one training seed. To evaluate its contribution more broadly, we extended the ResiBand-versus-HP-only comparison to Qwen3-1.7B and Qwen3-8B with three paired training seeds each, reporting all benchmarks:
On Qwen3-1.7B, the two configurations have close mean MultiHop scores, with a paired difference of -1.19 ± 2.39 points; the mean change is small relative to between-seed variability. On Qwen3-8B, adding smoothing improves MultiHop by 11.26 ± 4.79 points, with positive gains in all three seeds and the largest gain on 2Wiki. This gain is accompanied by a 21.60 ± 10.42 percentage-point reduction in outputs containing repeated trigrams, consistent with the intended stabilizing role. GSM8K and MMLU show only small mean changes on both backbones. These results motivate retaining both stages as a shared configuration across the tested backbones, combining targeted contrast recovery with complementary stabilization.