acceptodds
Under review as a conference paper at ICLR 2027

ResiBand: Recovering Evidence Contrast under Local Common-Mode Dominance

Abstract

Many language-model tasks require identifying and combining a few pieces of relevant evidence in distractor-rich contexts. Models may use this evidence yet fail to distinguish it reliably from competing information. We investigate this difficulty in the residual stream. Controlled evidence-relocation experiments show that support–distractor differences remain present but are small relative to a dominant component shared by nearby residual-stream states. This imbalance is associated with weaker evidence discrimination in subsequent attention layers. We introduce ResiBand, a causal residual operator that subtracts a short-horizon exponential average, smooths the remaining variation over a longer horizon, and maps the filtered signal into a learned low-rank residual correction. The fixed filter exposes variation relative to the local background, while the learned mapping adapts the correction to the task with pretrained weights frozen. Under a shared configuration across backbones, ResiBand improves average F1 across three multi-hop QA benchmarks over LoRA by 19.30, 10.95, and 1.93 percentage points on Qwen3-1.7B, Qwen3-8B, and Llama-3.1-8B, respectively, while requiring fewer trainable parameters. Mechanistic analyses and residual interventions indicate that the learned corrections improve discrimination between supporting evidence and distractors. These results support temporal filtering and learned residual correction for better use of available evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-hop reasoning, context utilization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dMpzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies evidence use in distractor-rich contexts through residual-stream measurements and controlled evidence relocation. It proposes ResiBand, which applies causal local subtraction followed by exponential smoothing and a learned low-rank residual correction, while keeping pretrained backbone weights frozen. Across three backbones, the reported three-seed mean multi-hop QA improvement over the specified Q/V-LoRA baseline is 19.30, 10.95, and 1.93 percentage points, respectively. Operator ablations, evidence-relocation experiments, and interventions examine how filtering and learned residual writes affect evidence discrimination. The results support the utility of the proposed adaptation pathway, while the specific causal role of local common-mode dominance remains less firmly established.

Critical strengths
  • A useful adaptation method supported by comparisons across backbones. Table 1 reports positive mean improvements over the specified LoRA baseline on all nine multi-hop backbone–benchmark combinations, with fewer trainable parameters. On Qwen3-1.7B, Table 3 also compares against an adapter with the same rank, parameter count, and training budget: ResiBand reaches a multi-hop average of 22.41 versus 12.00. This control supports a contribution from the filtered input beyond simply adding trainable residual capacity.

  • Controlled diagnostics that go beyond attention visualization. The relocation experiments preserve the question, answer, document/token multisets, and overall length while exchanging support and control positions. The paper combines these measurements with generated-answer performance, K/V deletion, and learned-write interventions. For example, Table 2 reports mean F1 of 0.447 for ResiBand versus 0.191 for adapter-only on the controlled relocation suite, with a paired source-bootstrap interval for the difference. This provides meaningful evidence of improved behavior in the constructed setting.

Critical weaknesses
  • The proposed causal explanation is not fully isolated. Sections 3 and 5.5 establish associations among residual statistics, routing, and answer discrimination, and show causal effects of enabling the complete trained module. However, the complete module changes both the temporal representation and the learned residual correction. These interventions do not specifically establish that suppressing the measured common-mode component causes the behavioral gain. Likewise, the increase in the contrast/common-mode ratio after filtering is compatible with the filter’s construction. The Q/K/V patches in Appendix C.7 provide additional causal evidence, but do not resolve this identification issue; their largest reported single-layer effect is 11.6% of the full On–Off margin gain. This does not rule out a distributed mechanism, but it limits how strongly the particular causal account is established.

  • The scope of the comparative claims exceeds the strongest available controls. Table 3 supports filtering relative to the unfiltered adapter, but the incremental multi-hop gain of the complete band over high-pass alone is 1.48 points, measured with one training seed. Its robustness is therefore unclear. Furthermore, the LoRA results establish superiority under the reported fixed Q/V adaptation contract, rather than over well-tuned PEFT generally. Appendix B specifies method-specific learning rates but leaves important optimization, prompting, and generation/scoring details unspecified. These limitations matter when judging which part of the architecture is necessary and how broadly its comparative advantage should be interpreted.

Questions for authors
  • Mechanism clarification: Which reported intervention specifically distinguishes common-mode suppression from other changes introduced by the trained residual branch? If this distinction is not identified by the current experiments, would you narrow the causal interpretation while retaining the supported behavioral and diagnostic findings?

  • Component contribution: Do you have paired results across training seeds for the full-band versus high-pass-only comparison? If not, can you qualify the claim that the longer-horizon smoothing stage contributes reliably, given the 22.41 versus 20.93 single-seed result in Table 3?

  • Comparison and reproducibility clarification: Please specify the optimizer/schedule, development procedure for the method-specific learning rates, prompt templates, generation limits and stopping rules, and completion-boundary scoring rules. Were these evaluation choices identical across arms? Please also clarify whether the inference-region masks used for deployment and latency reporting were selected on a separate development set.

I would be happy to raise my score if some of the questions or concerns are addressed.

9 comments
AuthorsOpenReview ↗

Response to Reviewer dMpz (Part 1)

Thank you for your thoughtful and constructive review.

Which reported intervention specifically distinguishes common-mode suppression from other changes introduced by the trained residual branch?

Thank you for raising this important question. It helped us strengthen the causal evidence for our proposed mechanism. To test the contribution of common-mode suppression, we restore the suppressed component with all other conditions held fixed. Specifically, we replace Lτ(h−Lϵ(h))L_{\tau}(h - L_{\epsilon}(h)) with Lτ(h)L_{\tau}(h).

Results on Qwen3-1.7B , reevaluated on an RTX 4090, are shown below using whole-completion F1:

ConditionMuSiQueHotpotQA2WikiMultiHop avg.
ResiBand15.99 ± 5.3836.06 ± 7.7429.46 ± 7.4427.17 ± 6.69
Common-mode restored1.76 ± 0.523.28 ± 0.993.76 ± 0.832.93 ± 0.73

Restoring the common-mode component sharply reduces F1 on all three benchmarks. This provides direct intervention evidence that common-mode suppression contributes to ResiBand's performance.

The independent 200-source diagnostic cohort connects this effect to evidence routing: compared with unmodified ResiBand, restoring the common-mode component reduces support-minus-control attention-mass contrast by 0.0525 and the gold-versus-counter first-answer-token log-probability margin by 7.29. These interventions directly support a causal contribution of common-mode suppression to evidence use in the trained ResiBand model.Together, these results support a causal contribution of common-mode suppression: restoring the shared background weakens evidence routing and reduces answer correctness.

Do you have paired results across training seeds for the full-band versus high-pass-only comparison?

Thank you for raising this point. In this work, we focus on a gap between evidence access and effective use: relevant evidence remains available, but the model struggles to distinguish it from distractors. We find that its contrast with distractors can be weak relative to a dominant locally shared component in the residual stream. This motivates a simple, targeted design: subtract the short-horizon average to suppress this shared background and expose the remaining contrast. The longer-horizon smoothing stage then serves to stabilize the resulting signal, aggregating persistent variation and attenuating isolated token-wise fluctuations before the learned residual write.

The incremental benefit of this complementary stabilization stage is model-dependent. The original ablation in Table 3 used one backbone and one training seed. To evaluate its contribution more broadly, we extended the ResiBand-versus-HP-only comparison to Qwen3-1.7B and Qwen3-8B with three paired training seeds each, reporting all benchmarks:

BackboneMethodMuSiQueHotpotQA2WikiMultiHop avg.GSM8KMMLU
Qwen3-1.7BResiBand16.10 ± 4.9836.29 ± 6.4029.75 ± 7.4327.38 ± 6.1051.21 ± 2.3537.65 ± 0.30
Qwen3-1.7BHP-only15.39 ± 4.9239.61 ± 11.0130.70 ± 8.8328.57 ± 8.2550.82 ± 0.9437.69 ± 0.27
Qwen3-8BResiBand15.30 ± 4.6122.38 ± 7.0233.39 ± 2.1323.69 ± 4.3772.38 ± 4.2445.34 ± 0.55
Qwen3-8BHP-only11.78 ± 0.2114.20 ± 2.2711.32 ± 1.1312.43 ± 1.1971.92 ± 4.2645.29 ± 0.72

On Qwen3-1.7B, the two configurations have close mean MultiHop scores, with a paired difference of -1.19 ± 2.39 points; the mean change is small relative to between-seed variability. On Qwen3-8B, adding smoothing improves MultiHop by 11.26 ± 4.79 points, with positive gains in all three seeds and the largest gain on 2Wiki. This gain is accompanied by a 21.60 ± 10.42 percentage-point reduction in outputs containing repeated trigrams, consistent with the intended stabilizing role. GSM8K and MMLU show only small mean changes on both backbones. These results motivate retaining both stages as a shared configuration across the tested backbones, combining targeted contrast recovery with complementary stabilization.

AuthorsOpenReview ↗

Response to Reviewer dMpz (Part 2)

Please specify the optimizer/schedule, development procedure for the method-specific learning rates, prompt templates, generation limits and stopping rules, and completion-boundary scoring rules. Were these evaluation choices identical across arms? Please also clarify whether the inference-region masks used for deployment and latency reporting were selected on a separate development set.

Thank you for raising these reproducibility and comparison concerns. Within each dataset and backbone, all methods used the same training budget, prompt template, and evaluation procedure. We followed a standard adapter-training setup with AdamW and cosine decay. Generation was greedy, limited to 64 new tokens, and stopped at EOS or the length limit. We scored only the generated suffix after the prompt boundary using normalized whole-completion token F1. These choices were identical across all comparison arms.

We used 10−410^{-4} for LoRA, following a commonly recommended adapter learning rate. ResiBand learns an additive correction from normalized, filtered representations and does not use LoRA's explicit α/r=2\alpha/r = 2 scaling; we therefore fixed its learning rate at 3×10−43 \times 10^{-4} as an adapter-style heuristic. Both values were fixed across backbones rather than selected using the evaluation sets.

To check whether the comparison was caused by an unfavorable LoRA learning rate, we additionally performed a five-point, three-seed sensitivity sweep on Qwen3-1.7B (10−510^{-5} to 10−310^{-3}), while keeping the LoRA architecture, training budget, and evaluation protocol unchanged.

MethodLRMuSiQueHotpotQA2WikiMultiHop avg.GSM8KMMLU
LoRA1e-53.97 ± 0.217.60 ± 0.167.14 ± 0.236.23 ± 0.1952.61 ± 1.1135.45 ± 0.27
LoRA3.16e-55.34 ± 0.139.52 ± 0.397.49 ± 0.487.45 ± 0.3147.55 ± 2.1735.75 ± 0.10
LoRA1e-46.59 ± 0.689.76 ± 0.597.86 ± 0.308.07 ± 0.3849.57 ± 3.9336.36 ± 0.71
LoRA3.16e-47.16 ± 0.369.99 ± 0.108.05 ± 0.078.40 ± 0.0851.46 ± 3.4536.94 ± 0.07
LoRA1e-37.64 ± 0.2810.11 ± 0.578.29 ± 0.138.68 ± 0.3149.72 ± 0.8237.26 ± 0.63
ResiBand3e-416.10 ± 4.9836.29 ± 6.4029.75 ± 7.4327.38 ± 6.1051.21 ± 2.3537.65 ± 0.30

ResiBand remains substantially stronger on MultiHop than every LoRA learning rate tested (27.38 versus a maximum of 8.68), while remaining comparable on GSM8K and MMLU. Thus, the advantage cannot be explained solely by insufficient learning-rate tuning of the reported LoRA baseline.

Regarding the inference-region masks, these were introduced as a layer ablation to understand where ResiBand corrections contribute. ResiBand reconditions the residual stream to help frozen downstream layers use available evidence more effectively. This ablation shows that, across all three backbones, early and middle regions yield the two highest multi-hop averages, while late-only execution is weakest. This identifies intervention depth as an important design choice and motivates early-to-middle placement as a starting point for follow-up residual-correction methods. The original latency masks followed these ablation findings; they were not selected on a separate development set.

We hope the additional results help address your concerns and give you greater confidence in our work. Thank you again for your thoughtful feedback and consideration!

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer dMpz,

As the rebuttal deadline approaches, could you please take a look at our responses? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 32574

Reviewer_Jtq3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies evidence use in contexts with distractors. The authors link poor evidence selection to components shared by nearby residual states. ResiBand subtracts a local average, smooths the remaining signal, and applies a learned low-rank residual update. The backbone remains frozen during adaptation. Experiments cover question answering and reasoning across language models, with controlled tests of evidence use.

Critical strengths
  1. The paper identifies a relevant failure mode: useful evidence may be present but poorly distinguished from distractors.
  2. The method uses a fixed filter and a small learned residual update. It adapts the model without changing the backbone.
  3. The reported gains span several language models. Controlled interventions also support changes in evidence use.
Critical weaknesses
  1. The default all-layer design is insufficiently justified, and the main quality and latency results use different layer settings. Disabling many jointly trained updates greatly improves question answering. The best active region also varies across backbones and tasks. An execution policy fixed on development data is needed to assess quality, stability, and cost for the same configuration.

  2. The reported question-answering gains are difficult to interpret because whole-completion F1 is affected by answer length and extra explanation. Changing output length can improve this score without correcting an answer. Answer margins and deletion tests support improved evidence use, but do not quantify how many benchmark answers become correct. The main benchmarks need consistent final-answer extraction and additional correctness checks.

  3. The experiments do not establish that common-mode suppression explains the gains, although they support improved evidence use. Moving evidence also changes its position and local context. Other interventions test the learned corrections without isolating the effect of the shared component. Varying that component while keeping evidence content and position fixed would provide a more direct test.

  4. The experiments do not establish that the multi-hop QA gains reflect better integration of multiple supporting facts. The relocation and deletion tests treat the support bundle as a whole. These tests support evidence use, but do not show whether the model combines each fact required by the question. Changing one necessary fact at a time would provide a more direct test of multi-hop evidence integration.

Questions for authors

Please refer to the Weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Jtq3 (Part 1)

Thank you for your thoughtful and constructive review.

The default all-layer design is insufficiently justified, and the main quality and latency results use different layer settings. Disabling many jointly trained updates greatly improves question answering. The best active region also varies across backbones and tasks. An execution policy fixed on development data is needed to assess quality, stability, and cost for the same configuration.

We would like to clarify that our evaluation follows a progressive experimental design, and that all-layer execution is intended as a reference configuration rather than the final or universally optimal deployment design.

First, we evaluate ResiBand with all layers enabled (Table 1) to establish its overall effectiveness under a unified configuration across different backbones. Even without any layer selection, ResiBand consistently outperforms LoRA on multi-hop QA across the three backbones. These results demonstrate the effectiveness of our proposed residual-conditioning mechanism under the default all-layer configuration.

Second, we conduct layer-region ablations (Table 4) to further investigate the contributions of learned corrections at different depths. The results reveal that early and middle layers generally contribute more strongly to multi-hop reasoning. Importantly, these experiments are intended to characterize depth-dependent behavior rather than establish an optimal layer-selection algorithm.

Third, we evaluate the inference efficiency of the high-performing configurations identified in the ablation. The quality results for these exact configurations are reported in Table 4, while their corresponding latency overheads are reported in Section 5.6, ensuring consistency between the quality and efficiency measurements.

We agree that developing a systematic execution policy selected on independent development data is an interesting direction. However, our primary contribution is the residual-conditioning mechanism and its effectiveness, rather than optimal layer scheduling. Since the all-layer configuration already demonstrates consistent improvements without any selection policy, we consider systematic layer selection a complementary optimization problem and leave its investigation to future work.

The reported question-answering gains are difficult to interpret because whole-completion F1 is affected by answer length and extra explanation. […] The main benchmarks need consistent final-answer extraction and additional correctness checks.

We agree that the benchmark gains need a direct check of answer correctness. We therefore evaluated final-answer correctness using the same extraction rule for both methods. The table below reports final-answer exact match (EM).

BackboneMethodMuSiQueHotpotQA2WikiAvg.
Qwen3-1.7BLoRA0.00 ± 0.000.27 ± 0.310.00 ± 0.000.09 ± 0.10
ResiBand11.44 ± 2.1225.20 ± 3.1222.93 ± 4.8119.86 ± 2.70
Qwen3-8BLoRA0.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00
ResiBand17.56 ± 4.0223.60 ± 8.8528.53 ± 4.7023.23 ± 5.51
Llama-3.1-8BLoRA0.17 ± 0.290.07 ± 0.120.20 ± 0.350.14 ± 0.25
ResiBand8.17 ± 6.5021.67 ± 11.7218.27 ± 6.1316.03 ± 7.64

LoRA's very low EM prompted us to inspect its outputs. Some responses begin with the correct answer but continue with extra text. These continuations can remain in the extracted answer and prevent an exact match.

We therefore added an answer-prefix check that credits a complete gold answer or accepted alias at the start of a response, regardless of trailing text.

BackboneMethodMuSiQueHotpotQA2WikiAvg.
Qwen3-1.7BLoRA14.11 ± 1.6730.47 ± 2.8427.80 ± 5.6024.13 ± 2.97
ResiBand20.72 ± 1.5139.40 ± 0.7233.73 ± 1.8931.29 ± 1.01
Qwen3-8BLoRA25.06 ± 0.5936.27 ± 4.2029.20 ± 4.8030.17 ± 2.56
ResiBand45.44 ± 2.9472.07 ± 2.4249.80 ± 5.7655.77 ± 3.56
Llama-3.1-8BLoRA18.72 ± 1.3620.13 ± 0.9919.40 ± 1.3119.42 ± 0.89
ResiBand43.33 ± 1.0965.47 ± 8.0455.67 ± 2.3954.82 ± 3.56

As these results show, ResiBand is more accurate than LoRA on all three QA benchmarks across all three backbones, under both strict EM and the prefix check.

AuthorsOpenReview ↗

Response to Reviewer Jtq3 (Part 2)

The experiments do not establish that common-mode suppression explains the gains, although they support improved evidence use. Moving evidence also changes its position and local context. Other interventions test the learned corrections without isolating the effect of the shared component. Varying that component while keeping evidence content and position fixed would provide a more direct test.

Thank you for your feedback, which helped us strengthen the evidence for our proposed mechanism. We restore the suppressed component while keeping the evidence content, positions, and all other conditions fixed to test the contribution of common-mode suppression. Specifically, we replace Lτ(h−Lϵ(h))L_{\tau}(h - L_{\epsilon}(h)) with Lτ(h)L_{\tau}(h).

We evaluated the Qwen3-1.7B interventions on an RTX 4090 using the same answer checks as above. The table includes final-answer EM and answer-prefix correctness, together with the original whole-completion F1:

MetricConditionMuSiQueHotpotQA2WikiAvg.
Whole-completion F1ResiBand15.99 ± 5.3836.06 ± 7.7429.46 ± 7.4427.17 ± 6.69
Common-mode restored1.76 ± 0.523.28 ± 0.993.76 ± 0.832.93 ± 0.73
Final-answer EMResiBand11.22 ± 1.9525.20 ± 3.8322.60 ± 5.0119.67 ± 3.12
Common-mode restored0.56 ± 0.512.47 ± 1.400.27 ± 0.311.10 ± 0.69
Answer-prefix correctnessResiBand20.56 ± 1.4239.33 ± 1.8033.27 ± 1.9731.05 ± 1.50
Common-mode restored0.44 ± 0.424.07 ± 2.535.00 ± 2.253.17 ± 1.65

Restoration lowers all three scores on every benchmark. The drop remains under the prefix check, so it is not explained solely by extra text in the output.

To see how this relates to evidence use, we also examined the same intervention on an independent diagnostic set of 200 examples. Restoration reduces the attention contrast between supporting evidence and control text by 0.0525. It also lowers the log-probability margin between the gold answer and the counter-answer at the first answer token by 7.29. Together, these results support a causal contribution of common-mode suppression: restoring the shared background weakens evidence routing and reduces answer correctness.

The experiments do not establish that the multi-hop QA gains reflect better integration of multiple supporting facts. The relocation and deletion tests treat the support bundle as a whole. These tests support evidence use, but do not show whether the model combines each fact required by the question. Changing one necessary fact at a time would provide a more direct test of multi-hop evidence integration.

To address this point, we test whether replacing the supporting facts from either document reduces final-answer accuracy.

For each HotpotQA example, we identify the annotated supporting sentences in the two supporting documents, A and B. We replace the supporting sentences from one document at a time with equally many irrelevant tokens, keeping token positions fixed. We apply matched replacements to irrelevant text as controls. All conditions use the same ResiBand checkpoint. We generate complete answers under the same decoding protocol and measure extracted-answer exact match:

InterventionExtracted-answer EM (%)
Unmodified prompt37.19
Replace supporting facts A24.62
Replace supporting facts B27.14
Replace irrelevant text A36.68
Replace irrelevant text B37.69

Replacing the supporting facts from either document substantially reduces final-answer EM, while matched irrelevant replacements have little effect. These results show that information from each supporting document contributes to final-answer accuracy in this diagnostic.

We hope our responses and additional results address your concerns and strengthen your confidence in our conclusions. Thank you again for your time and consideration!

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer Jtq3,

As the rebuttal deadline approaches, could you please take a look at our responses? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 32574

Reviewer_n4hHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author propose ResiBand, a parameter efficient way to finetune the model and add architectural change to better distinguish relevant evidence from distractors.

Critical strengths

The experiments are conducted on different models.

Critical weaknesses
  1. Why calculate F1 score? The tasks in the paper can be calculated with accuracy?
  2. Have you tuned the parameters for the baseline LoRA method? (https://arxiv.org/abs/2602.04998)
  3. Why the accuracy decreases on qwen3-8B GSM8K?
  4. It seems that the method is not only a PEFT method but can be applied as a general architectural change?
  5. It is unclear whether this will cause degradation and whether this method scales.
Questions for authors

Check weakness above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer n4hH (Part 1)

Thank you for your review and for taking the time to share your concerns.

Before addressing the individual questions, we would like to clarify the central contribution of our work. Our primary contribution is identifying and addressing a previously underexplored limitation in LLM evidence utilization. We find that relevant evidence can remain accessible in the context while its distinguishing signal is obscured by a dominant local common-mode component in the residual stream. Motivated by this observation, we introduce ResiBand, which suppresses locally shared components and learns a low-rank residual correction to improve evidence discrimination. Importantly, our contribution is supported by both empirical improvements and mechanistic evidence, rather than solely by comparisons against LoRA. Across three backbones, ResiBand improves average multi-hop QA performance over LoRA (Table 1). Furthermore, the parameter-matched adapter-only comparison (Table 3), controlled evidence-relocation experiments (Table 2), and mechanistic interventions (Section 5.5) demonstrate that the improvements are associated with better discrimination and utilization of relevant evidence. We address the specific concerns below:

Why calculate F1 score? The tasks in the paper can be calculated with accuracy?

We chose token F1 for the multi-hop QA benchmarks because it gives partial credit when an answer partially matches the reference, complementing the complete-match criterion of EM. To check final-answer correctness directly, we supplemented the original whole-completion scores with EM, using one fixed, gold-independent answer extractor for all methods. GSM8K already uses numerical-answer EM, and MMLU uses multiple-choice accuracy.

BackboneMethodMuSiQueHotpotQA2WikiAvg.
Qwen3-1.7BLoRA0.00 ± 0.000.27 ± 0.310.00 ± 0.000.09 ± 0.10
ResiBand11.44 ± 2.1225.20 ± 3.1222.93 ± 4.8119.86 ± 2.70
Qwen3-8BLoRA0.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00
ResiBand17.56 ± 4.0223.60 ± 8.8528.53 ± 4.7023.23 ± 5.51
Llama-3.1-8BLoRA0.17 ± 0.290.07 ± 0.120.20 ± 0.350.14 ± 0.25
ResiBand8.17 ± 6.5021.67 ± 11.7218.27 ± 6.1316.03 ± 7.64

ResiBand has higher mean EM in all nine comparisons. Some LoRA outputs begin with a correct answer but continue with repeated or unrelated text, which can prevent an exact match. As a separate check that ignores trailing text, we count responses beginning with a complete gold answer or accepted alias. ResiBand also scores higher under this rule: 31.29 versus 24.13, 55.77 versus 30.17, and 54.82 versus 19.42 on the three backbones, respectively. As these results show, the gains also hold when evaluated by answer accuracy.

Have you tuned the parameters for the baseline LoRA method? (https://arxiv.org/abs/2602.04998)

For the original LoRA baseline, we used a commonly recommended adapter-finetuning learning rate of 1e-4 (Hugging Face TRL documentation), fixed across backbones without development-set tuning. To assess sensitivity to this choice, as raised in the cited study, we additionally evaluated five log-spaced LoRA learning rates spanning 1e-5–1e-3. For a compact comparison, we show the complete three-seed Qwen3-1.7B sweep below:

MethodLRMuSiQue F1HotpotQA F12Wiki F1Avg. F1GSM8KMMLU
LoRA1e-53.97 ± 0.217.60 ± 0.167.14 ± 0.236.23 ± 0.1952.61 ± 1.1135.45 ± 0.27
LoRA3.16e-55.34 ± 0.139.52 ± 0.397.49 ± 0.487.45 ± 0.3147.55 ± 2.1735.75 ± 0.10
LoRA1e-46.59 ± 0.689.76 ± 0.597.86 ± 0.308.07 ± 0.3849.57 ± 3.9336.36 ± 0.71
LoRA3.16e-47.16 ± 0.369.99 ± 0.108.05 ± 0.078.40 ± 0.0851.46 ± 3.4536.94 ± 0.07
LoRA1e-37.64 ± 0.2810.11 ± 0.578.29 ± 0.138.68 ± 0.3149.72 ± 0.8237.26 ± 0.63
ResiBand3e-416.10 ± 4.9836.29 ± 6.4029.75 ± 7.4327.38 ± 6.1051.21 ± 2.3537.65 ± 0.30

On Qwen3-1.7B, ResiBand achieves higher average MultiHop F1 than all five LoRA settings (27.38 versus a maximum of 8.68), with GSM8K and MMLU scores close to the best LoRA results in the sweep. Thus, the average QA advantage on this backbone persists beyond the original LoRA learning rate in the three-seed sensitivity analysis.

AuthorsOpenReview ↗

Response to Reviewer n4hH (Part 2)

Why the accuracy decreases on qwen3-8B GSM8K?

To better understand this decrease, we stratified GSM8K performance by the number of reasoning steps. On Qwen3-8B, both LoRA and ResiBand improve over Base on problems requiring fewer reasoning steps but fall below it on those requiring more steps. Because the two adapted models use the same HotpotQA–GSM8K mixture and exhibit a similar step-dependent trend, this result is more consistent with a shared fine-tuning distribution effect than with a ResiBand-specific failure. In particular, examples requiring many reasoning steps are relatively rare in the training mixture, which may weaken some of the strong base model’s existing capability on such problems. This pattern does not change the matched comparison between adaptation methods: under the same training data and budget, ResiBand consistently outperforms LoRA on multi-hop QA across all three backbones.

It seems that the method is not only a PEFT method but can be applied as a general architectural change?

As outlined above, the key idea is to help the model’s existing attention use available evidence more effectively. Our diagnostics link poor evidence discrimination to a strong background shared by nearby residual states. ResiBand provides a targeted intervention: suppress this background and stabilize the remaining signal before a learned low-rank correction. PEFT lets us test this mechanism while keeping pretrained weights fixed. Incorporating the design into pretraining or full-model fine-tuning remains future work.

It is unclear whether this will cause degradation and whether this method scales.

We considered both concerns in our evaluation design, testing ResiBand across three backbones and three training seeds on contextual QA, arithmetic, and general knowledge. MMLU was not used in adaptation training, and its mean accuracy improves on all three backbones (Table 1). This provides evidence of general knowledge retention alongside the contextual QA gains. GSM8K also improves on two backbones, with the small Qwen3-8B decrease discussed above.

We evaluated models ranging from 1.7B to 8B within our available compute budget. Across this range, ResiBand improves contextual QA performance on all three backbones (Table 1). These gains across model sizes highlight ResiBand's potential for use in future model architectures. Evaluation on larger models and integration into pretraining or full-model fine-tuning remain future work.

We hope these clarifications and additional results will be helpful when you reconsider your assessment. Thank you again for your time and consideration!

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer n4hH,

As the rebuttal deadline approaches, could you please take a look at our responses? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 32574

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.