acceptodds
Under review as a conference paper at ICLR 2027

HiLA: Historical-Layer Attention for Low-Rank Adaptation

Abstract

Parameter-efficient fine-tuning (PEFT) has become an important paradigm for adapting large language models (LLMs) to downstream tasks. However, existing methods such as LoRA typically perform local low-rank updates conditioned only on the current-layer input, leaving task-relevant representations from preceding layers underutilized. Our analysis shows that shallow and intermediate layers already contain information that can influence final predictions, motivating the explicit use of cross-layer representations for low-rank adaptation. To this end, we propose HiLA (Historical Layer Attention), a PEFT method that dynamically integrates cross-layer representational changes into low-rank updates. HiLA constructs differences between the current and preceding-layer states and uses the current LoRA bottleneck representation as a query to retrieve relevant historical information through attention. We evaluate HiLA on mathematical reasoning, commonsense reasoning, and open-domain dialogue generation across four backbones with different architectures and scales. HiLA achieves the highest average commonsense accuracy on all four backbones and the best performance on five of six dialogue-generation metrics. Notably, on challenging mathematical reasoning tasks, HiLA consistently outperforms all low-rank baselines and even surpasses full fine-tuning on Llama-3.2-3B with only 1.03% trainable parameters. These results indicate that attention-based cross-layer representations effectively alleviate the limitations of existing PEFT methods. Our code is available at https://anonymous.4open.science/r/HiLA-8209/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_VNP8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HiLA, a parameter-efficient fine-tuning method that augments LoRA with a token-wise attention mechanism applied to the differences between current and historical block representations. The LoRA bottleneck serves as a query, while shared projections generate historical keys and values; the retrieved representation is then injected before the LoRA up-projection. The evaluation spans mathematical reasoning, commonsense reasoning, and dialogue generation, supported by component ablations, representation diagnostics, and inference-resource measurements. The most pronounced improvements are observed in mathematical reasoning tasks.

Critical strengths
  1. The methodological design is concrete and well-supported by targeted ablations. By operating on historical representation differences, the approach grants adapters access to preceding layers with minimal parameter overhead. The systematic comparisons—spanning raw states versus adjacent/cross-layer differences, various aggregation strategies, normalization schemes, and injection configurations—provide strong empirical grounding for the architectural choices, moving beyond mere aggregate benchmark reporting.

  2. The mathematical-reasoning gains are substantial. On Llama-3.1-8B, HiLA improves over the reported low-rank baselines on both mathematical datasets, although full fine-tuning remains stronger on MATH. These results support a useful adapter contribution despite the qualifications below.

Critical weaknesses
  1. Statistical reliability is insufficiently characterized. Table 7 lists five seeds for commonsense/dialogue and only seed 42 for mathematics, but the main results do not report dispersion or clearly explain seed aggregation. Some improvements are small, including 0.26 percentage points in the Qwen3-8B commonsense average and 0.02 points in dialogue BERTScore-F1. The large mathematical gains are more compelling, but their robustness across seeds and the comparability of baseline tuning remain important uncertainties. These are limitations of the empirical support, rather than evidence of a flawed construction.

  2. Inference overhead materially limits the practical efficiency advantage. Table 16 reports an end-to-end latency of 455.66 ms and a throughput of 20.83 tokens/s for HiLA, compared to 242.33 ms and 40.81 tokens/s for standard LoRA. This represents an approximate 1.88× increase in latency and a 50% reduction in throughput. While the authors transparently disclose this trade-off, it significantly curtails the practical appeal of the parameter savings. Crucially, because the input-dependent historical attention mechanism cannot be statically merged into the base model weights (unlike standard LoRA), this inference penalty is permanent. The manuscript must explicitly clarify whether the baselines in Table 16 were evaluated with merged weights, as this fundamentally dictates how the deployment trade-off should be interpreted.

Questions for authors
  1. Clarification and focused validation: Are the commonsense/dialogue results averages over seeds 41–45, and what is their dispersion? Do the main mathematical gains persist across additional seeds, and what validation-based tuning was used for HiLA and the strongest low-rank baselines?

  2. Inference configuration and weight merging: For the latency and throughput measurements in Table 16, were the mergeable baselines (e.g., standard LoRA) evaluated with their adapter weights merged into the base model? Clarifying the exact inference configuration is essential for a fair assessment of the deployment overhead.

  3. The component ablations and zero/shuffled-history interventions provide useful evidence about the historical branch, but inference-time disruption also changes the inputs expected by the trained model. Is there a comparison with a separately trained, parameter-matched nonlinear adapter using only current-layer information? Such a control would help distinguish the benefit of historical information from that of additional adaptation capacity.

6 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

Your comments on statistical reliability, the inference measurement protocol, and additional adapter capacity are greatly appreciated. In response, we add two independent training seeds for mathematical reasoning, a code-generation task with multi-seed evaluation, multi-seed statistics for commonsense reasoning and dialogue generation, and an independently trained parameter-matched current-layer nonlinear adapter as a control. We respond to each point below.


(W1, Q1) Multi-seed statistics, confidence intervals, and validation-set selection

We add experimental statistics for four task categories to test whether HiLA's gains depend on particular conditions and clarify how each method selects its validation set.

1. Commonsense reasoning: five-seed results

Main Table 3 of the revised manuscript reports the eight-task average accuracy and standard deviation over five random seeds for four backbones. The comparison between HiLA and the strongest baseline on each backbone is as follows:

BackboneStrongest baseline (%)HiLA (%)Gain (%)
Qwen2.5-1.5BDoRA: 79.21±0.3679.21\pm0.3679.61±0.3279.61\pm0.32+0.40
Qwen3-8BRaSA: 87.91±0.3587.91\pm0.3588.17±0.3088.17\pm0.30+0.26
Llama-3.2-3BMoRA: 82.76±0.3482.76\pm0.3482.97±0.2882.97\pm0.28+0.21
Llama-3.1-8BRaSA: 85.41±0.3785.41\pm0.3785.79±0.3185.79\pm0.31+0.38

HiLA achieves the highest eight-task average accuracy on all four backbones, with standard deviations ranging from 0.28 to 0.32. Its gains over the strongest baselines range from 0.21% to 0.40%.

2. Open-domain dialogue generation: five-seed results

Main Table 4 of the revised manuscript reports the means and standard deviations of six generation metrics for Llama-3.2-3B on ConvAI2 over five random seeds. The comparison with the strongest baseline, DoRA, is as follows:

MetricHiLA mean ±\pm std.DoRA mean ±\pm std.Mean difference
BLEU16.25±0.1116.25\pm0.1116.03±0.0716.03\pm0.07+0.22
BERTScore-F187.48±0.0287.48\pm0.0287.46±0.0287.46\pm0.02+0.02
BERTScore-R87.21±0.0287.21\pm0.0287.20±0.0287.20\pm0.02+0.01
BERTScore-P87.82±0.0687.82\pm0.0687.78±0.0387.78\pm0.03+0.04
METEOR21.18±0.3321.18\pm0.3321.29±0.0821.29\pm0.08−0.11
ROUGE-L26.33±0.0926.33\pm0.0926.22±0.0626.22\pm0.06+0.11

HiLA has the highest mean on five of the six metrics, while its METEOR score is 0.11 lower than DoRA's. For the small BERTScore-F1 gain, the difference is 0.02, and both methods have a standard deviation of 0.02. We therefore report both means and standard deviations to show the size of this gap relative to random variation.

3. Mathematical reasoning: two additional independent training seeds

The original main table reports results for only training seed 42. Following your suggestion, we conduct two additional independent training runs with separate random seeds. Together with the original seed 42, they form three independent training results. Appendix A.6, Table 12 of the revised manuscript reports the mean, sample standard deviation, and 95% confidence interval over the three seeds:

BackboneDatasetHiLA mean ±\pm std. (%)95% CIStrongest baseline (%)Gain (%)Two-sided pp value
Llama-3.2-3BGSM8K69.24±0.8369.24\pm0.83[67.18,71.31][67.18,71.31]RaSA: 53.15+16.090.00089
Llama-3.2-3BMATH19.35±1.3419.35\pm1.34[16.02,22.68][16.02,22.68]RaSA: 12.56+6.790.01273
Llama-3.1-8BGSM8K79.93±0.6879.93\pm0.68[78.25,81.62][78.25,81.62]DoRA: 69.14+10.790.00132
Llama-3.1-8BMATH33.06±0.1933.06\pm0.19[32.59,33.53][32.59,33.53]PiSSA: 30.12+2.940.00139

All four two-sided one-sample tt tests reach p<0.05p<0.05. For each of the four model-dataset combinations, the lowest HiLA result among the three seeds is higher than the fixed reported value of the corresponding low-rank baseline.

For FFT, all three Llama-3.2-3B seeds exceed FFT on GSM8K and MATH, whose FFT results are 65.13% and 16.84%, respectively. The three Llama-3.1-8B seeds also remain above FFT on GSM8K, whose FFT result is 73.39%, but remain below FFT on MATH, whose result is 36.36%. These cross-seed results further support the stability of the mathematical-reasoning advantage across repeated training runs.

AuthorsOpenReview ↗

Response by Authors, Part 2

4. Code generation: new task and three-seed evaluation

In addition to the three original task categories, we add a code-generation task. We fine-tune Llama-3.1-8B on Magicoder-Evol-Instruct-110K and evaluate the functional correctness of generated programs on HumanEval+. Main Table 2 reports the results for training seed 42:

MethodPass@1Pass@5Pass@10
LoRA57.7667.1869.18
MoRA58.0267.6869.40
RaSA58.2467.4569.90
GraLoRA58.9067.9669.65
Spectral Adapter59.1868.6370.55
CKA-based LoRA58.5168.2470.16
TensLoRA57.4966.7268.75
HiLA66.9875.1778.12

HiLA improves over the strongest baseline, Spectral Adapter, by 7.80%, 6.54%, and 7.57%, respectively.

To evaluate the stability of this result, we independently train three models with three random seeds and generate 50 candidate programs for each problem under the same evaluation configuration. Appendix A.7, Table 13 reports the means, standard deviations, and 95% confidence intervals over the three seeds:

MetricHiLA mean ±\pm std. (%)95% CIStrongest baseline (%)Gain (%)Two-sided pp value
Pass@164.86±1.8464.86\pm1.84[60.29,69.43][60.29,69.43]59.18+5.680.0332
Pass@573.28±1.6773.28\pm1.67[69.13,77.43][69.13,77.43]68.63+4.650.0404
Pass@1076.15±1.7876.15\pm1.78[71.73,80.57][71.73,80.57]70.55+5.600.0321

All three HiLA means exceed the reported value of the best baseline, Spectral Adapter[2], and the lower bound of each 95% confidence interval also exceeds the corresponding baseline. The two-sided one-sample tt tests against the fixed baseline values all reach p<0.05p<0.05.

In addition, Appendix A.8, Table 14 adds a code-generation comparison with Echo-LoRA[3] and IFC-LoRA[4]. Because their official code repositories are unavailable, we implement both methods according to their paper descriptions and evaluate them with the same training configuration:

MethodPass@1Pass@5Pass@10
LoRA57.7667.1869.18
Echo-LoRA59.4068.6070.50
IFC-LoRA59.5068.8070.80
HiLA66.9875.1778.12

HiLA improves over IFC-LoRA by 7.48%, 6.37%, and 7.32%, respectively.

5. Validation-set selection and training configuration

All methods use the same validation set and checkpoint-selection rule. Validation loss is evaluated every 100 optimization steps, and the checkpoint with the lowest validation loss among all evaluated checkpoints is retained at the end of training. Each baseline retains the parameterization and initialization specified by its method.

With these additions, mathematical reasoning expands from one random seed to three independent training seeds, code generation includes repeated evaluation with three independent training seeds, and commonsense reasoning and open-domain dialogue generation report means and standard deviations over five random seeds.

(W2, Q2) Inference configuration and weight merging

LoRA, DoRA, and MoRA in Table 7 all retain their adapter branches and do not merge the weights. Following the branch-preserving measurement protocol used by GAWS[1] in Appendix E.3, we measure online overhead while the adapter branch of each method participates in the forward pass. We do not pre-merge updates that can be represented statically into the backbone weights. Under this setting, HiLA increases end-to-end latency by 88.0% (1.88 times) relative to unmerged LoRA and reduces token throughput by about 49.0%. We discuss this cost as a practical limitation of HiLA in the main text and identify reducing historical-retrieval computation and inference latency as directions for future optimization.

AuthorsOpenReview ↗

Response by Authors, Part 3

(Q3) Independently trained parameter-matched current-layer nonlinear adapter

To address the concern about additional adapter capacity, we add a parameter-matched current-layer nonlinear adapter, CNA-PM (Current-layer Nonlinear Adapter with Parameter Matching), to directly test whether adding only current-layer nonlinear mapping capacity can achieve performance close to HiLA.

CNA-PM retains the standard LoRA branch. It feeds the current LoRA bottleneck representation through an RMSNorm without affine parameters and a two-layer GELU network without biases, injects the output into the original bottleneck through a residual connection, and then applies the LoRA up-projection. The network is shared across layers and across the four modules corresponding to HiLA's historical branch. It does not read information from any historical layer.

Its additional parameter count is 2×32×1,730=110,7202\times32\times1{,}730=110{,}720. CNA-PM and HiLA have the same total number of trainable parameters, 33,140,864, and the same trainable-parameter ratio, 1.0315%. They use the same LoRA initialization, training configuration, and random seed 42 for independent training.

The results on eight commonsense reasoning tasks with Llama-3.2-3B are as follows:

MethodTrainable parameters (%)BoolQPIQASIQAHSWGARC-cARC-eOBQAAverage accuracy
HiLA1.031570.5586.5180.8190.8085.0076.9689.0684.0082.96
CNA-PM1.031562.0885.5378.0581.9282.4072.7084.1880.2078.38

Under the same parameter budget, HiLA outperforms CNA-PM on all eight tasks and improves the average accuracy by 4.58%. This shows that adding only current-layer nonlinear mapping capacity is insufficient to match the performance of historical-difference retrieval.

This result complements the historical-intervention experiment in Section 4.3, Table 5 of the original manuscript. Full HiLA reaches 68.84% accuracy on GSM8K, which decreases to 59.14% when historical information is removed and to 41.93% when historical information is shuffled. CNA-PM independently trains a parameter-matched adapter that uses only current-layer information, providing further evidence that historical information contributes beyond additional adapter capacity.

References

[1] Alsuradi et al. Grouped Adaptive Weight Sharing for Parameter-Efficient Fine-tuning. Findings of ACL 2026.

[2] Zhang and Pilanci. Spectral Adapter: Fine-Tuning in Spectral Space. NeurIPS 2024.

[3] Peng et al. Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection. arXiv:2605.08177, 2026.

[4] Zhang et al. IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning. arXiv:2607.22251, 2026.

Reviewer_8kBoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates LoRA-like parameter‑efficient fine‑tuning for large language models. This work has found that previous LoRA and LoRA‑style PEFT may underutilize cross‑layer task‑related representations. Thus, this work proposes HiLA, a historical‑layer attention LoRA method. It computes cross‑layer state differences to filter redundant layer‑shared signals. It dynamically retrieves useful historical information via attention and injects it into LoRA’s low‑rank bottleneck.

Experiments are conducted on reasoning and dialogue datasets across four backbones. HiLA outperforms strong PEFT baselines and even exceeds full fine‑tuning under certain settings in some cases.

Critical strengths
  1. This work proposes a novel cross-layer LoRA paradigm working in the representation space, which can bring more potential for LoRA cross-layer collaboration.

  2. Solid mechanistic evidence via activation patching, linear probing, and CKA analysis.

  3. Strong experimental results. HiLA can consistently outperform the naive LoRA and other enhanced LoRA methods on four backbones and five metrics.

  4. Compared to the improvement, the additional parameter overhead is acceptable.

Critical weaknesses
  1. The inference latency overhead is non-negligible versus the vanilla LoRA. Meanwhile, this should be discussed in the main text instead of the appendix.

  2. HiLA is designed as a universal PEFT method. However, the evaluation only covered a limited set of backbones and tasks. The absence of diverse model-task tests weakens evidence for its general scalability.

  3. HiLA can even outperform the FFT in Table 1. The authors should explain the reason in more detail.

  4. It seems like that HiLA relies heavily on empirical hyper-parameter tuning.

Minor

  1. Lacks enough background information about the cross-layer connection in the Introduction section.
Questions for authors
  1. How sensitive is performance to $ r_{h} $ across different model sizes?

  2. More experiments on other backbone LLMs.

8 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

We sincerely appreciate your recognition of HiLA's representation mechanism, experimental results, and research value. Following your suggestions, we add an analysis of inference cost, code-generation experiments, multi-seed evaluations, and discussion of related cross-layer work in the revised manuscript. We also clarify the scope of applicability and the design rationale of the method. We respond to each point below.


(W1) Discussion of inference cost in the main text

Following your suggestion, we move the cost analysis from the appendix to Section 5, Limitations, and add full training time to present HiLA's actual computational cost more completely. The following results are measured on Llama-3.2-3B for open-domain dialogue generation on ConvAI2.

MethodTrainable parameters (%)Full training time (h)Peak memory (GiB)Latency (ms)Request throughput (req/s)Token throughput (tokens/s)
LoRA1.02810.436.19242.334.1340.81
MoRA1.02620.566.23329.553.0329.94
DoRA1.04240.696.35566.101.7716.65
HiLA1.03150.846.30455.662.1920.83

HiLA has an end-to-end latency of 455.66 ms, which is 88.0% higher than LoRA and approximately 1.88 times its latency. Its peak memory is 6.30 GiB, an increase of only about 1.8%. Full training takes 0.84 h, nearly twice the 0.43 h required by LoRA. Token throughput decreases from 40.81 tokens/s to 20.83 tokens/s, a reduction of about 49.0%.

We agree that this inference overhead is a practical limitation of HiLA relative to standard LoRA. This overhead arises because HiLA retains historical-layer states for the current token and performs input-dependent cross-layer attention retrieval during inference.

However, HiLA's additional parameter ratio remains at the same scale as those of standard LoRA, MoRA, and DoRA, and its latency is lower than DoRA's (455.66 ms vs. 566.10 ms). HiLA is therefore not intended to replace standard LoRA in every setting. It provides another adaptation choice for applications that place greater value on downstream accuracy and complex reasoning and can accept additional inference cost. Standard LoRA remains preferable in deployment settings that require extremely low latency and high throughput.

The revised manuscript now states this performance-cost trade-off explicitly and does not evaluate HiLA's practical benefit solely through parameter efficiency.

AuthorsOpenReview ↗

Response by Authors, Part 2

(W2, Q2) Validation across models, scales, and tasks

We understand the concern about HiLA's generalization. The original manuscript evaluates four backbones with different scales and architectures: Qwen2.5-1.5B, Qwen3-8B, Llama-3.2-3B, and Llama-3.1-8B. They cover the Qwen and Llama families and three parameter scales, 1.5B, 3B, and 8B. Commonsense reasoning is evaluated on all four backbones, mathematical reasoning on the two Llama backbones, and dialogue generation on Llama-3.2-3B.

Following your suggestion to broaden the task coverage, the revised manuscript adds a code-generation task. We fine-tune Llama-3.1-8B on Magicoder-Evol-Instruct-110K and evaluate the functional correctness of generated code on HumanEval+. The main results are reported in Sections 4.1–4.2 and Table 2.

MethodPass@1Pass@5Pass@10
LoRA57.7667.1869.18
MoRA58.0267.6869.40
RaSA58.2467.4569.90
GraLoRA58.9067.9669.65
Spectral Adapter59.1868.6370.55
CKA-based LoRA58.5168.2470.16
TensLoRA57.4966.7268.75
HiLA66.9875.1778.12

HiLA achieves 66.98%, 75.17%, and 78.12% on Pass@1, Pass@5, and Pass@10, respectively. Compared with the strongest baseline in the table, Spectral Adapter, HiLA improves by 7.80%, 6.54%, and 7.57%, respectively. This shows that HiLA's advantage extends beyond mathematical and commonsense reasoning to code generation, which requires complex logical organization and program construction.

In addition, Appendix A.7, Table 13 reports results from three independent training seeds to evaluate the stability of the code-generation advantage:

MetricHiLA mean ±\pm std. (%)Mean 95% CIStrongest baseline (%)Gain (%)Two-sided pp value
Pass@164.86±1.8464.86\pm1.84[60.29,69.43][60.29,69.43]59.18+5.680.0332
Pass@573.28±1.6773.28\pm1.67[69.13,77.43][69.13,77.43]68.63+4.650.0404
Pass@1076.15±1.7876.15\pm1.78[71.73,80.57][71.73,80.57]70.55+5.600.0321

The HiLA mean over the three seeds exceeds the single-seed value of the strongest baseline for all three metrics, and the lower bound of each 95% confidence interval also exceeds the corresponding baseline value. The two-sided one-sample tt tests against the fixed reported value of Spectral Adapter all give p<0.05p<0.05, which supports the stability of HiLA's advantage on code generation.

To further strengthen the comparison with related cross-layer methods, we also implement Echo-LoRA and IFC-LoRA according to their paper descriptions because the corresponding papers do not release official code, and evaluate them under the same training protocol. The results are reported in Appendix A.8, Table 14:

MethodPass@1Pass@5Pass@10
LoRA57.7667.1869.18
Echo-LoRA59.4068.6070.50
IFC-LoRA59.5068.8070.80
HiLA66.9875.1778.12

HiLA achieves the best result on all three metrics and improves over IFC-LoRA by 7.48%, 6.37%, and 7.32%, respectively.

AuthorsOpenReview ↗

Response by Authors, Part 3

(W3) Why HiLA surpasses FFT in some settings

We believe that HiLA's advantage over FFT on some mathematical-reasoning tasks may be related to the adaptation constraint created by freezing the pretrained backbone and to its explicit use of cross-layer historical representations.

Greater freedom to update parameters does not necessarily lead to better downstream generalization. In domain-specific instruction tuning, directly updating all backbone parameters can alter general representations formed during pretraining. When downstream supervision is insufficient to constrain these changes, it can also increase the risk of overfitting or forgetting existing knowledge. HiLA, in contrast, freezes the entire pretrained backbone and learns task-related updates only through low-rank adapters. It uses the current LoRA bottleneck as a Query, dynamically retrieves historical state differences from different depths, and injects them into the current low-rank update. This design limits direct changes to pretrained parameters while providing an additional path for using task-relevant information from intermediate layers. Such constrained adaptation may therefore be more suitable when the model must preserve existing knowledge while adapting to multi-step reasoning tasks.

This explanation is consistent with the following experimental observations:

  1. Availability of task information in intermediate layers. The activation-patching experiment in Figure 1 of Section 1 shows that shallow and intermediate representations can substantially affect the final prediction. Layer-wise linear probing further shows that these layers already contain readable task-related information, which supports retrieving historical representations in HiLA.
  2. Contribution of historical information. In Section 4.3, Table 5, the GSM8K accuracy of Llama-3.2-3B decreases from 68.84% for full HiLA to 59.14% when historical information is removed and to 41.93% when mismatched historical information is used. This shows that HiLA's gain depends on historical representations aligned with the current sample, rather than only on an additional adapter branch.
  3. Cross-seed stability of the advantage over FFT. In Appendix A.6, Table 12, the three-seed means for Llama-3.2-3B on GSM8K and MATH are 69.24±0.83%69.24\pm0.83\% and 19.35±1.34%19.35\pm1.34\%, respectively. All three seeds exceed the corresponding FFT results, indicating that this advantage is not an accidental consequence of a single random initialization.

(W4) Hyperparameter selection and sensitivity

We clarify that the main experiments do not search separately for the best structural hyperparameters for each backbone or downstream task. Instead, they use a common main configuration.

Specifically, all main experiments use LoRA rank r=32r=32, scaling coefficient α=64\alpha=64, and dropout 0.05. HiLA uses historical-attention rank rh=4r_h=4 throughout, injects historical information into the q/k/v and up branches, and retains the standard LoRA structure in the down branch. Historical projection matrices are shared across dimension-compatible modules. These structural settings remain the same across all four backbones. The number of training epochs and sequence length follow the predefined task protocols, without a separate search for structural configurations for each backbone.

For the historical-attention rank, Appendix B.5, Table 20 performs a joint ablation of rr and rhr_h on Qwen2.5-1.5B. With r=32r=32 fixed, rh=4/8/16/32r_h=4/8/16/32 gives average accuracies of 79.60%/79.41%/79.34%/79.29%, respectively, with a maximum change of only 0.31%. Under the current experimental setting, increasing the historical-attention rank does not provide a consistent performance gain and instead adds parameters.

In addition, Appendix B.2, Table 17 ablates the historical-information injection positions. The q/k/v+up configuration reaches 79.60%, higher than the 78.42% obtained by injecting only into q/k/v. Extending the injection to the o or g branch reduces accuracy to 77.84% and 77.21%, respectively, while enabling all six injectable branches reduces it to 76.63%. This shows that the effect of historical information depends on the selected modules. The q/k/v+up configuration is chosen through controlled ablation rather than tuned separately for each backbone.

Therefore, HiLA's structural design includes experimentally supported choices, but it does not rely on extensive empirical hyperparameter searches for each model and task. The main experiments use a common configuration, and the rank-sensitivity study shows that a small historical-attention rank already provides an effective gain within the tested range.

Reviewer_gQpwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HiLA, a PEFT method that dynamically integrates cross-layer representational changes into low-rank updates. HiLA constructs differences between the current and preceding-layer states and uses the current LoRA bottleneck representation as a query to retrieve relevant historical information through attention. Experiments are conducted on mathematical reasoning, commonsense reasoning, and open-domain dialogue generation. The results show HiLA improves performance on these tasks.

Critical strengths

S1. This paper studies a meaningful problem: how to use the preceding-layer state to improve PEFT.

S2. The method is clear and easy to understand. It is an interesting idea that use current representation as query to retrieve historical information.

S3. The experiments are extensive. The paper evaluates several downstream tasks across four backbones.

Critical weaknesses

W1. Related work is not fully discussed, and the paper overclaims its novelty. ResLoRA[1] has already used cross-layer connections that aggregate every previous lora's low-dimensional representation into the current layer. MRLA[2] has already used the representation of the current layer as a query to all previous layers to retrieve query-related information. Therefore, HiLA seems to mainly combine with previous existing components. The authors should add these existing works and clarify the contribution beyond this combination.

W2. The improvements might come from more parameters and extra computational cost. HiLA's number of trainable parameters is 15 times higher than BSLoRA's, which brings the performance improvement. Moreover, HILA's inference latency is 1.88 times higher than LoRA's, which has the same number of trainable parameters. I understand this is a trade-off between performance and cost. Therefore, the paper should more broaderly discuss about whether the trade-off is worth it or not.

W3. The improvements are small in many tasks. For example, HiLA only achieves a 0.21%-0.4% improvement in commonsense reasoning tasks. Although the authors demonstrate that they used five seeds, the improvements are still not convincing. More datasets and std for current datasets might be useful.

[1] Shi et al. ResLoRA: Identity Residual Mapping in Low-Rank Adaption. Findings of ACL 2024.

[2] Fang et al. Cross-Layer Retrospective Retrieving via Layer Attention. ICLR 2023.

Questions for authors

See W1~3

7 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

We are grateful for your comments on three key issues: the positioning of our contribution, additional capacity, and the magnitude of the gains. In the revised manuscript, we add discussion of eight related methods, clarify HiLA's design through comparisons with cross-layer methods, use an independently trained parameter-matched CNA-PM to test whether capacity explains the gains, add a code-generation task and additional mathematical-reasoning seeds, and provide statistical evidence from multiple seeds for commonsense reasoning and open-domain dialogue generation. We respond to each point below.


(W1) Relationship to ResLoRA and MRLA and the contribution of HiLA

HiLA's contribution is not the first introduction of cross-layer connections or Query-based retrieval of historical information. It lies in redefining how historical information is constructed, how it is retrieved, and how it contributes to the low-rank update. We add the related work discussed in the review to Section 2 of the revised manuscript and compare MRLA, ResLoRA, Echo-LoRA, and HiLA in Appendix C.3, Table 21.

MRLA and ResLoRA explore cross-layer information from different perspectives:

  • MRLA uses the current-layer representation as a Query in a vision backbone and aggregates historical-layer features with layer attention to enhance the current representation.
  • ResLoRA accumulates low-dimensional intermediate representations from preceding LoRA branches through a middle-shortcut and introduces them into the current LoRA branch through a residual path.

HiLA differs from a direct combination of these two methods in three aspects:

  1. Retrieval target: cross-layer representation changes instead of historical states or historical adapter outputs. HiLA constructs a difference between the current-layer and historical-layer input states for the same token: δℓ,k=xℓ−xk,0≤k<ℓ\delta_{\ell,k}=x_\ell-x_k,\qquad 0\leq k<\ell. Unlike direct aggregation of historical states or historical LoRA outputs, this difference explicitly represents how the token's representation changes from a historical layer to the current layer and suppresses components shared across depths.

  2. Retrieval condition: the needs of the current low-rank update instead of the current-layer representation alone. HiLA uses the current LoRA bottleneck representation zℓz_\ell to generate the Query and maps historical differences to Keys and Values. Attention weights then dynamically select historical information relevant to the current low-rank branch. Thus, the retrieval weights depend not only on the current token representation but also directly on the representation space of the current adapter.

  3. Information fusion: direct injection of the dynamically retrieved result into the current low-rank bottleneck. HiLA injects the historical-attention output mℓm_\ell into the current bottleneck through a residual connection, z~ℓ=zℓ+mℓ\widetilde z_\ell=z_\ell+m_\ell, and then uses the LoRA up-projection to generate the weight update. Historical information selection and fusion therefore directly serve the current low-rank update instead of simply accumulating outputs from preceding adapters.

These differences are also supported by our experiments. Section 4.3 and Figure 4 show that the mean CKA between the current and historical layers is 0.809, whereas the mean CKA between historical layers and cross-layer differences decreases to 0.126. This indicates that differencing effectively reduces the similarity of shared structure across layers. Figure 5 further shows that cross-layer differences retain strong task-discriminative information in the middle layers.

Section 4.4 and Table 6 compare different historical-signal constructions and aggregation strategies under the same experimental setting:

VariantAverage accuracy over eight commonsense tasks (%)
LoRA78.69
Raw historical states + mean aggregation78.37
Raw historical states + attention aggregation78.95
Adjacent-layer differences + mean aggregation77.04
Adjacent-layer differences + attention aggregation79.08
Cross-layer state differences + attention aggregation (HiLA)79.60

For raw historical states, attention aggregation improves over mean aggregation by 0.58%. With attention aggregation fixed, cross-layer differences improve over raw historical states and adjacent-layer differences by 0.65% and 0.52%, respectively. This shows that the gain does not come only from adding attention retrieval, but is directly related to the information provided by cross-layer differences.

AuthorsOpenReview ↗

Response by Authors, Part 2

In addition, Appendix B.1, Table 16 compares different sources of historical readout signals. Using the low-rank bottleneck representation zz for historical retrieval gives an average ConvAI2 score of 54.38%, whereas using the direct input hh of the target linear layer gives 52.74%. This further supports retrieving historical information in the low-rank representation space.

Finally, Appendix D.4 analyzes the role of the difference construction theoretically. Common additive interference in the current and historical layers is exactly canceled by differencing, and the appendix gives the conditions under which the signal-to-noise ratio of the difference exceeds that of the raw historical state. These conditions depend on the increment in discriminative signal and the correlation between cross-layer residuals.

Therefore, HiLA differs from MRLA and ResLoRA through a distinct mechanism for using historical information, which combines cross-layer state-difference construction, retrieval conditioned on the low-rank bottleneck, and historical-information injection. It is not simply an added attention module or residual connection.

(W2) Parameter-matched control and the performance-cost trade-off

To address the concern about additional parameter capacity, we design a parameter-matched current-layer nonlinear adapter, CNA-PM, to distinguish gains from extra capacity from gains provided by historical-information retrieval.

CNA-PM retains the original LoRA branch. It transforms the current bottleneck representation with an RMSNorm without affine parameters and a two-layer GELU network without biases, and injects the result into the LoRA bottleneck through a residual connection. It does not read information from historical layers. The network is shared across layers and the corresponding four modules, and its additional parameter count is 2×32×1,730=110,7202\times32\times1{,}730=110{,}720. CNA-PM and HiLA have the same total number of trainable parameters, 33,140,864, and use the same LoRA initialization, training configuration, and random seed for independent training.

Appendix B.4, Table 19 reports the results on eight commonsense tasks with Llama-3.2-3B:

MethodTrainable parameters (%)BoolQPIQASIQAHSWGARC-cARC-eOBQAAverage accuracy
HiLA1.031570.5586.5180.8190.8085.0076.9689.0684.0082.96
CNA-PM1.031562.0885.5378.0581.9282.4072.7084.1880.2078.38

Under the same parameter budget, HiLA outperforms CNA-PM on all eight tasks and improves the average accuracy by 4.58%. This shows that additional parameter capacity alone cannot explain the gain, while historical-difference retrieval provides useful information.

For actual computational cost, we move the resource measurements from the appendix to Section 5, Limitations, and add full training time. The following measurements are obtained on ConvAI2 with Llama-3.2-3B:

MethodTrainable parameters (%)Full training time hhPeak memory (GiB)Latency (ms)Request throughput (req/s)Token throughput (tokens/s)
LoRA1.02810.436.19242.334.1340.81
MoRA1.02620.566.23329.553.0329.94
DoRA1.04240.696.35566.101.7716.65
HiLA1.03150.846.30455.662.1920.83

Compared with LoRA, HiLA increases end-to-end latency by 88.0%, increases full training time from 0.43 h to 0.84 h, and increases peak memory by only about 1.8%. At the same time, HiLA has lower latency than DoRA, whose trainable-parameter count is similar (455.66 ms vs. 566.10 ms).

HiLA therefore trades additional inference computation for downstream performance rather than reducing inference cost. Standard LoRA is more suitable when latency and throughput are the priority. HiLA provides another adaptation choice when accuracy on complex reasoning tasks matters more and additional computation is acceptable.

AuthorsOpenReview ↗

Response by Authors, Part 3

(W3) Multi-seed evidence and the new code-generation task

To address the concern that the gain on commonsense reasoning is small, we add multi-seed statistics for the existing tasks and introduce a code-generation task. These experiments evaluate HiLA across different task types and independent training runs.

1. New code-generation task

We fine-tune Llama-3.1-8B on Magicoder-Evol-Instruct-110K and evaluate the functional correctness of generated programs on HumanEval+. Sections 4.1–4.2 and Table 2 of the revised manuscript report the main results for training seed 42:

MethodPass@1Pass@5Pass@10
LoRA57.7667.1869.18
MoRA58.0267.6869.40
RaSA58.2467.4569.90
GraLoRA58.9067.9669.65
Spectral Adapter59.1868.6370.55
CKA-based LoRA58.5168.2470.16
TensLoRA57.4966.7268.75
HiLA66.9875.1778.12

HiLA improves over the strongest baseline, Spectral Adapter, by 7.80%, 6.54%, and 7.57%, respectively.

Appendix A.7, Table 13 further reports means, standard deviations, and 95% confidence intervals across three independent training seeds:

MetricHiLA mean ±\pm std. (%)95% CIStrongest baseline (%)Gain (%)Two-sided pp value
Pass@164.86±1.8464.86\pm1.84[60.29,69.43][60.29,69.43]59.18+5.680.0332
Pass@573.28±1.6773.28\pm1.67[69.13,77.43][69.13,77.43]68.63+4.650.0404
Pass@1076.15±1.7876.15\pm1.78[71.73,80.57][71.73,80.57]70.55+5.600.0321

All three means exceed the fixed reported baseline, and the lower bounds of all three 95% confidence intervals also exceed the corresponding baseline. The two-sided one-sample tt tests give p<0.05p<0.05 for all three metrics. This supports the stability of the observed advantage.

To add further related methods, we experimentally discuss Echo-LoRA[4] and IFC-LoRA[5]. Because their official code repositories are not publicly available, we follow the method descriptions in the papers and report the reproduced results in Appendix A.8, Table 14:

MethodPass@1Pass@5Pass@10
LoRA57.7667.1869.18
Echo-LoRA (reproduction)59.4068.6070.50
IFC-LoRA (reproduction)59.5068.8070.80
HiLA66.9875.1778.12

HiLA improves over IFC-LoRA by 7.48%, 6.37%, and 7.32% on Pass@1, Pass@5, and Pass@10, respectively, further supporting its advantage over related cross-layer methods.

2. Multi-seed results for mathematical reasoning

The original main table reports results for training seed 42. To evaluate stability, Appendix A.6, Table 12 of the revised manuscript adds results from three different training seeds:

BackboneDatasetHiLA mean ±\pm std. (%)95% CIstrongest baselineGain (%)Two-sided pp value
Llama-3.2-3BGSM8K69.24±0.8369.24\pm0.83[67.18,71.31][67.18,71.31]RaSA: 53.15+16.090.00089
Llama-3.2-3BMATH19.35±1.3419.35\pm1.34[16.02,22.68][16.02,22.68]RaSA: 12.56+6.790.01273
Llama-3.1-8BGSM8K79.93±0.6879.93\pm0.68[78.25,81.62][78.25,81.62]DoRA: 69.14+10.790.00132
Llama-3.1-8BMATH33.06±0.1933.06\pm0.19[32.59,33.53][32.59,33.53]PiSSA: 30.12+2.940.00139

All four two-sided one-sample tt tests reach p<0.05p<0.05, and the lowest HiLA result among the three seeds in each group is higher than the corresponding low-rank baseline.

For FFT, the three Llama-3.2-3B seeds all exceed the corresponding FFT results on GSM8K and MATH, which are 65.13% and 16.84%, respectively. The three Llama-3.1-8B seeds exceed FFT on GSM8K, whose FFT result is 73.39%, but remain below FFT on MATH, whose result is 36.36%.

Reviewer_nuzhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HiLA, a parameter-efficient fine-tuning (PEFT) method that augments LoRA-style adapters with attention over cross-layer representation differences. Concretely, at each layer ℓ, HiLA forms token-wise differences δℓ,k = xℓ − xk to preceding-layer states, projects them to keys/values, and uses the current LoRA bottleneck as a query to attend over depth, producing a history-conditioned vector that augments the LoRA bottleneck before the up-projection. Across math, commonsense, and dialogue tasks on four LLM backbones, HiLA yields consistent gains over strong PEFT baselines and even surpasses full fine-tuning in some math settings; mechanistic analyses (activation patching, linear probes, CKA) aim to support the claim that preceding-layer differences carry task-relevant signals.

Critical strengths
  • Introduces an explicit representation-level cross-layer pathway for PEFT by attending over same-token cross-layer differences conditioned on the current LoRA bottleneck.
  • The “difference” construction (xℓ − xk) is well-motivated by CKA analysis to suppress shared/redundant components and to isolate evolving, task-discriminative signals.
  • The integration point is simple and compatible with standard LoRA (z̃ℓ = zℓ + mℓ), keeping the LoRA structure intact and enabling parameter sharing across modules.
  • Broad evaluation across three task families (math, commonsense, dialogue) and four heterogeneous backbones, with consistent improvements over LoRA, DoRA, PiSSA, MoRA, and RaSA.
Critical weaknesses
  • Computational/memory overhead and scalability are under-specified in the main text: attending over all preceding layers implies O(L^2) depth interactions unless a window k is enforced. It is unclear how many historical layers are used at inference, the cost of storing xk per layer/token, and the effect on latency/throughput/KV-cache.
  • Theoretical grounding is mostly empirical/diagnostic; a more formal analysis of when difference-based retrieval is superior (vs. raw states or parameter sharing) would strengthen the contribution.
  • Missing comparisons to closely related representation-level cross-layer PEFTs such as Echo‑LoRA (training-time cross-layer injection with distillation) and structural variants like GraLoRA, Spectral Adapter, IFCLoRA, and recent tensorized depth-sharing methods (e.g., TensLoRA). Some are discussed tangentially in related work (parameter sharing), but explicit empirical comparisons are absent.
  • Variance reporting is limited: key math results use a single seed; significance/CI across seeds would increase confidence, especially for headline gains over FFT.
  • Fairness of the FFT baseline is debatable: using a smaller LR but otherwise matching PEFT hyperparameters may not be optimal for FFT; stronger FFT or QLoRA baselines could contextualize the “surpass FFT” claim.
  • Echo‑LoRA should be discussed and compared experimentally, as it also leverages deeper/shallow cross-layer signals albeit only at training; this is a particularly relevant baseline.
  • Additional modern PEFT variants (GraLoRA, Spectral Adapter, IFCLoRA, CKA-based selective LoRA) warrant discussion and, where feasible, empirical comparison or rationale for exclusion.
Questions for authors
  • How many historical layers (k) are used by default at inference time? Do you attend over all previous layers or a fixed sliding window? Please report accuracy vs. k and the associated latency/memory trade-offs.
  • What is the exact inference-time memory overhead? Do you store all intermediate xk to compute δℓ,k, and how does this interact with the KV cache and batch size in long-context decoding?
  • Can you provide wall-clock training and inference throughput numbers for HiLA vs. LoRA across at least one backbone/sequence length, in the main text? Appendices mention measurements but deployment-facing numbers are crucial for PEFT claims.
  • How sensitive is performance to the historical attention rank rh and the choice of modules receiving historical attention (e.g., only Wq/Wv vs. also MLP projections)? Please add ablations or summarize from the appendix.
  • Could you compare against Echo‑LoRA (or similar training-only cross-layer injection with distillation)? This would clarify the trade-off between training-only cross-layer paths and HiLA’s inference-time historical attention.
  • For the FFT baseline, did you tune learning rates, batch sizes, and training steps specifically for full fine-tuning? If not, can you provide a better-tuned FFT or a QLoRA baseline to contextualize the “surpass FFT” claim?
  • Do attention weights concentrate on a few shallow/middle layers across tasks? Plotting depth-wise attention distributions would help interpretability and might suggest sparse retrieval strategies.
  • Are δℓ,k computed per token across the sequence? How are padding/masked positions handled, and does history attention apply at every token and layer during autoregressive decoding?
8 comments
AuthorsOpenReview ↗

Response by Authors, Part 1

We appreciate your careful comments on computational and memory overhead, the theoretical basis of the difference construction, comparisons with related methods, and the statistical reliability of the experiments. In response, we add historical-window and inference-resource experiments, a signal-to-noise analysis of the difference construction, empirical comparisons with related PEFT methods, and independent-seed experiments for mathematical reasoning and code generation. We also improve the statistical reporting for commonsense reasoning and dialogue generation. We respond to each point below.


(W1, Q1) Historical-layer range, computational complexity, and window sensitivity

By default, HiLA reads all historical blocks preceding the current layer and does not use a fixed sliding window. For layer ℓ\ell, the historical set is

\{\delta_{\ell,0},\delta_{\ell,1},\ldots, \delta_{\ell,\ell-1}\}, \qquad \delta_{\ell,k}=x_\ell-x_k.$$ For one token, the main projection cost of reading $k$ historical states at one layer is $O\big(kd(r+r_h)\big)$, where $d$ is the hidden dimension, and $r$ and $r_h$ are the LoRA rank and historical-attention rank, respectively. When every Transformer block reads all preceding layers, summing the historical-retrieval cost over layers gives $O\big(L^2d(r+r_h)\big)\approx O(L^2dr)$, where $L$ is the number of Transformer blocks. The approximation holds under the default configuration $r_h\ll r$. If the historical range is restricted to the most recent $k$ layers, the overall main projection cost becomes $O\big(Lkd(r+r_h)\big)\approx O(Lkdr)$. Thus, full-history retrieval has quadratic depth complexity, whereas a fixed window reduces the depth-wise computational complexity to linear. To evaluate how the historical range affects task performance and actual inference cost, we restrict the number of most recent historical layers accessible during inference on Llama-3.2-3B and evaluate the average accuracy over eight commonsense tasks. The results are reported in **Appendix E.4, Table 24**: | Historical window $k$ | Average accuracy (%) | Accuracy change (%) | Latency (s) | Throughput (tokens/s) | Peak memory (GiB) | | --- | ---: | ---: | ---: | ---: | ---: | | 1 | 58.79 | −24.23 | 2.78 | 21.97 | 6.12 | | 2 | 68.95 | −14.07 | 2.81 | 20.52 | 6.13 | | 4 | 80.53 | −2.49 | 2.68 | 21.07 | 6.13 | | 8 | 81.93 | −1.09 | 2.63 | 21.02 | 6.16 | | 16 | 82.80 | −0.22 | 2.79 | 20.79 | 6.22 | | Full | **83.02** | — | 2.80 | 20.80 | 6.30 | The results show that the historical range has a clear effect on performance. Retaining only the most recent one or two layers reduces accuracy by **24.23% and 14.07%**, respectively. With windows of eight and sixteen layers, accuracy increases to 81.93% and 82.80%, approaching the full-history result of 83.02%. Reducing the window, however, does not provide a comparable efficiency gain. Compared with full history, window 16 reduces peak memory by only **0.08 GiB**, while its latency and throughput are 2.79 s and 20.79 tokens/s, almost identical to the full-history values of 2.80 s and 20.80 tokens/s. Although window 8 reduces latency to 2.63 s, its accuracy remains 1.09% lower. We therefore use full-history retrieval by default as a trade-off between historical-information coverage and downstream performance, rather than because a fixed window cannot be implemented.
AuthorsOpenReview ↗

Response by Authors, Part 2

(W1, Q2) Inference memory overhead and KV cache

During the current forward pass, HiLA reuses historical hidden states that have already been computed and retains references to them for later layers. It does not create additional copies of the full hidden-state tensors.

Let each state element occupy bb bytes, let the batch size be BB, let SS be the sequence length processed in the current forward pass, and let ℓ\ell be the number of historical layers accessible from the current layer. The storage required for the raw historical states is Mhistory=bBSℓdM_{\mathrm{history}}=bBS\ell d. Thus, retaining all historical states requires O(BSLd)O(BSLd) memory. During prefill, SS is the input-prompt length. During token-by-token decoding with the backbone KV cache, S=1S=1, so the historical-state storage for one forward pass is O(BLd)O(BLd).

HiLA's historical-state buffer is independent of the backbone KV cache. The backbone continues to maintain its attention Key/Value cache according to the standard autoregressive mechanism. HiLA's historical-state list and readout cache are valid only during the current forward pass and are cleared when that pass ends. They are not appended to the KV cache across generation steps. HiLA therefore does not maintain an additional cross-layer historical cache that grows with the generated sequence length.

For a fixed window kk, if states outside the window are also released, raw-state storage can be reduced to O(BSkd)O(BSkd). Restricting the attention-access range without releasing states outside the window, however, does not guarantee a proportional reduction in memory.

This is also reflected in the measurements. On Llama-3.2-3B, the peak memory for windows 1, 16, and Full is 6.12, 6.22, and 6.30 GiB, respectively. Because the implementation reuses references to existing hidden states, the additional memory mainly comes from extending their lifetime and from temporary tensors for differences, Key/Value projections, and attention, rather than from copying all historical states.

(W1, Q3) Training and inference efficiency

We report HiLA's actual training and inference costs in Section 5, Limitations, Table 7. The following measurements use Llama-3.2-3B on the ConvAI2 open-domain dialogue-generation task:

MethodTrainable parameters (%)Full training time (h)Peak memory (GiB)Latency (ms)Request throughput (req/s)Token throughput (tokens/s)
LoRA1.02810.436.19242.334.1340.81
MoRA1.02620.566.23329.553.0329.94
DoRA1.04240.696.35566.101.7716.65
HiLA1.03150.846.30455.662.1920.83

Compared with LoRA, HiLA increases latency by 88.0% and peak memory by 0.11 GiB, or approximately 1.8%. However, request throughput is one of the key directions for future HiLA optimization. Under this configuration, the additional cost is mainly reflected in latency and throughput, while the memory increase is relatively small. Parameter sharing controls the number of additional trainable weights.

To further examine training cost across backbone scales, Appendix E.3, Table 23 reports full training time, time per optimization step, and throughput for four backbones on Commonsense-170K. The task uses two training epochs, a maximum sequence length of 256, and an effective batch size of 32:

BackboneMethodTime per optimization step (s)Full training time (h)Training throughput (tokens/s)
Qwen2.5-1.5BLoRA0.75592.245,267.32
Qwen2.5-1.5BHiLA1.48814.402,675.63
Llama-3.2-3BLoRA1.06613.153,630.20
Llama-3.2-3BHiLA1.79505.312,156.10
Qwen3-8BLoRA2.30026.811,671.27
Qwen3-8BHiLA3.666710.851,048.41
Llama-3.1-8BLoRA1.97055.831,964.10
Llama-3.1-8BHiLA3.467810.261,116.06
AuthorsOpenReview ↗

Response by Authors, Part 3

(W2) Theoretical basis and mechanistic evidence for the difference construction

The difference construction removes common additive components between the current and historical layers. Its discriminative benefit depends on the retained task-signal increment and the correlation between cross-layer residuals. We add the corresponding formal analysis in Appendix D.4.

The analysis shows that the benefit of differencing depends on the increment in the cross-layer discriminative signal and on the correlation between the residuals at the two layers. For residuals with fixed variance, stronger positive correlation helps reduce difference noise. If the two layers contain the same additive interference, that interference is exactly canceled.

These theoretical conditions correspond to the mechanistic experiments in the paper. Section 4.3 and Figures 4–5 examine this motivation at the representation level. The mean cross-layer CKA decreases from 0.809 to 0.126, indicating that differencing reduces shared structure. In the linear-probe task for commonsense reasoning, middle-layer differences also retain strong task-discriminative information, substantially outperform the reconstructed shared component, and approach or exceed the complete current state.

Section 4.4 and Table 6 provide further empirical evidence through component ablations. On Qwen2.5-1.5B, we compare different constructions of the historical signal:

VariantAverage accuracy over eight commonsense tasks (%)
LoRA78.69
Raw historical states + mean aggregation78.37
Raw historical states + attention aggregation78.95
Adjacent-layer differences + mean aggregation77.04
Adjacent-layer differences + attention aggregation79.08
Cross-layer state differences + attention aggregation (HiLA)79.60

With the attention aggregation method fixed, cross-layer differences improve over raw historical states and adjacent-layer differences by 0.65% and 0.52%, respectively. Thus, the theoretical analysis gives the conditions under which differences can outperform raw historical states, while CKA, linear probing, and component ablation provide empirical support from the perspectives of structural redundancy, task-discriminative information, and downstream performance.

Comments on the paper

2 comments

Dear reviewers,

The rebuttal period is ending soon. Please read the authors rebuttal and check if they addressed your concerns. Please acknowledge the rebuttal and adjust your scores accordingly.

Thanks.

-AC

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank all reviewers for their thorough evaluations, constructive critiques, and insightful suggestions. Their comments highlighted important aspects of HiLA, including statistical reliability, methodological contributions, empirical generalization, and practical applicability. In response, we conducted additional experiments, extended theoretical and mechanistic analyses, and further clarified key aspects of the manuscript.

Specifically, we have made the following additions and clarifications:

1. Strengthened statistical reliability and broadened downstream task evaluation

  • Introduced a new code-generation task to further demonstrate HiLA's advantages across a broader range of downstream tasks.
  • Conducted multi-seed experiments on code generation and mathematical reasoning, reporting means, standard deviations, and 95% confidence intervals, alongside two-sided t-tests to evaluate the statistical significance of the observed performance gains.
  • Supplemented the commonsense reasoning and open-domain dialogue generation results with standard deviations to provide a more comprehensive statistical report.

2. Expanded related-work discussion and comparative analysis

  • Conducted empirical comparisons with additional PEFT methods, including GraLoRA, Spectral Adapter, CKA-LoRA, TensLoRA, Echo-LoRA, and IFC-LoRA.
  • Expanded the related-work discussion in Section 2 and Appendix C, covering low-rank adaptation variants, cross-layer parameter sharing, tensorized adaptation, adapter placement and rank allocation, and cross-layer representation interaction.
  • Further clarified the methodological distinctions between HiLA and closely related methods (e.g., MRLA, ResLoRA, and Echo-LoRA) in historical-information construction, retrieval conditioning, and integration into low-rank updates.

3. Clarified HiLA's limitations and practical deployment trade-offs

  • Added historical-window sensitivity experiments and computational complexity analysis, together with further analysis of inference-time memory overhead and its relationship with KV cache memory consumption.
  • Expanded training and inference resource reporting, including full training time, inference latency, memory consumption, and throughput.
  • Introduced a dedicated Limitations section in the main text to explicitly acknowledge inference overhead and discuss future directions for more efficient historical retrieval and inference.

4. Extended theoretical and mechanistic analyses

  • Added a theoretical signal-to-noise analysis to characterize the conditions under which cross-layer differences can provide more discriminative information than raw historical representations.
  • Added historical-attention depth-distribution experiments and visualizations to investigate how historical information is retrieved across different layers.

5. Clarified experimental configurations and implementation protocols

  • Clarified the independent learning-rate search specifically conducted for full fine-tuning (FFT) and further specified the shared structural hyperparameter configurations.
  • Clarified inference-time weight-merging settings.

We believe these additions directly address the major concerns raised in the reviews, further strengthening the theoretical foundations, methodological justification, and empirical evidence of HiLA, while providing a more transparent characterization of its practical limitations and future research directions.

We once again sincerely thank all reviewers for their time, careful evaluations, and constructive feedback. Their thoughtful suggestions have been invaluable in strengthening this work, and we greatly appreciate their engagement throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.