- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces HiLA, a parameter-efficient fine-tuning method that augments LoRA with a token-wise attention mechanism applied to the differences between current and historical block representations. The LoRA bottleneck serves as a query, while shared projections generate historical keys and values; the retrieved representation is then injected before the LoRA up-projection. The evaluation spans mathematical reasoning, commonsense reasoning, and dialogue generation, supported by component ablations, representation diagnostics, and inference-resource measurements. The most pronounced improvements are observed in mathematical reasoning tasks.
Critical strengths
-
The methodological design is concrete and well-supported by targeted ablations. By operating on historical representation differences, the approach grants adapters access to preceding layers with minimal parameter overhead. The systematic comparisons—spanning raw states versus adjacent/cross-layer differences, various aggregation strategies, normalization schemes, and injection configurations—provide strong empirical grounding for the architectural choices, moving beyond mere aggregate benchmark reporting.
-
The mathematical-reasoning gains are substantial. On Llama-3.1-8B, HiLA improves over the reported low-rank baselines on both mathematical datasets, although full fine-tuning remains stronger on MATH. These results support a useful adapter contribution despite the qualifications below.
Critical weaknesses
-
Statistical reliability is insufficiently characterized. Table 7 lists five seeds for commonsense/dialogue and only seed 42 for mathematics, but the main results do not report dispersion or clearly explain seed aggregation. Some improvements are small, including 0.26 percentage points in the Qwen3-8B commonsense average and 0.02 points in dialogue BERTScore-F1. The large mathematical gains are more compelling, but their robustness across seeds and the comparability of baseline tuning remain important uncertainties. These are limitations of the empirical support, rather than evidence of a flawed construction.
-
Inference overhead materially limits the practical efficiency advantage. Table 16 reports an end-to-end latency of 455.66 ms and a throughput of 20.83 tokens/s for HiLA, compared to 242.33 ms and 40.81 tokens/s for standard LoRA. This represents an approximate 1.88× increase in latency and a 50% reduction in throughput. While the authors transparently disclose this trade-off, it significantly curtails the practical appeal of the parameter savings. Crucially, because the input-dependent historical attention mechanism cannot be statically merged into the base model weights (unlike standard LoRA), this inference penalty is permanent. The manuscript must explicitly clarify whether the baselines in Table 16 were evaluated with merged weights, as this fundamentally dictates how the deployment trade-off should be interpreted.
Questions for authors
-
Clarification and focused validation: Are the commonsense/dialogue results averages over seeds 41–45, and what is their dispersion? Do the main mathematical gains persist across additional seeds, and what validation-based tuning was used for HiLA and the strongest low-rank baselines?
-
Inference configuration and weight merging: For the latency and throughput measurements in Table 16, were the mergeable baselines (e.g., standard LoRA) evaluated with their adapter weights merged into the base model? Clarifying the exact inference configuration is essential for a fair assessment of the deployment overhead.
-
The component ablations and zero/shuffled-history interventions provide useful evidence about the historical branch, but inference-time disruption also changes the inputs expected by the trained model. Is there a comparison with a separately trained, parameter-matched nonlinear adapter using only current-layer information? Such a control would help distinguish the benefit of historical information from that of additional adaptation capacity.
6 comments
Response by Authors, Part 2
4. Code generation: new task and three-seed evaluation
In addition to the three original task categories, we add a code-generation task. We fine-tune Llama-3.1-8B on Magicoder-Evol-Instruct-110K and evaluate the functional correctness of generated programs on HumanEval+. Main Table 2 reports the results for training seed 42:
| Method | Pass@1 | Pass@5 | Pass@10 |
|---|---|---|---|
| LoRA | 57.76 | 67.18 | 69.18 |
| MoRA | 58.02 | 67.68 | 69.40 |
| RaSA | 58.24 | 67.45 | 69.90 |
| GraLoRA | 58.90 | 67.96 | 69.65 |
| Spectral Adapter | 59.18 | 68.63 | 70.55 |
| CKA-based LoRA | 58.51 | 68.24 | 70.16 |
| TensLoRA | 57.49 | 66.72 | 68.75 |
| HiLA | 66.98 | 75.17 | 78.12 |
HiLA improves over the strongest baseline, Spectral Adapter, by 7.80%, 6.54%, and 7.57%, respectively.
To evaluate the stability of this result, we independently train three models with three random seeds and generate 50 candidate programs for each problem under the same evaluation configuration. Appendix A.7, Table 13 reports the means, standard deviations, and 95% confidence intervals over the three seeds:
| Metric | HiLA mean std. (%) | 95% CI | Strongest baseline (%) | Gain (%) | Two-sided value |
|---|---|---|---|---|---|
| Pass@1 | 59.18 | +5.68 | 0.0332 | ||
| Pass@5 | 68.63 | +4.65 | 0.0404 | ||
| Pass@10 | 70.55 | +5.60 | 0.0321 |
All three HiLA means exceed the reported value of the best baseline, Spectral Adapter[2], and the lower bound of each 95% confidence interval also exceeds the corresponding baseline. The two-sided one-sample tests against the fixed baseline values all reach .
In addition, Appendix A.8, Table 14 adds a code-generation comparison with Echo-LoRA[3] and IFC-LoRA[4]. Because their official code repositories are unavailable, we implement both methods according to their paper descriptions and evaluate them with the same training configuration:
| Method | Pass@1 | Pass@5 | Pass@10 |
|---|---|---|---|
| LoRA | 57.76 | 67.18 | 69.18 |
| Echo-LoRA | 59.40 | 68.60 | 70.50 |
| IFC-LoRA | 59.50 | 68.80 | 70.80 |
| HiLA | 66.98 | 75.17 | 78.12 |
HiLA improves over IFC-LoRA by 7.48%, 6.37%, and 7.32%, respectively.
5. Validation-set selection and training configuration
All methods use the same validation set and checkpoint-selection rule. Validation loss is evaluated every 100 optimization steps, and the checkpoint with the lowest validation loss among all evaluated checkpoints is retained at the end of training. Each baseline retains the parameterization and initialization specified by its method.
With these additions, mathematical reasoning expands from one random seed to three independent training seeds, code generation includes repeated evaluation with three independent training seeds, and commonsense reasoning and open-domain dialogue generation report means and standard deviations over five random seeds.
(W2, Q2) Inference configuration and weight merging
LoRA, DoRA, and MoRA in Table 7 all retain their adapter branches and do not merge the weights. Following the branch-preserving measurement protocol used by GAWS[1] in Appendix E.3, we measure online overhead while the adapter branch of each method participates in the forward pass. We do not pre-merge updates that can be represented statically into the backbone weights. Under this setting, HiLA increases end-to-end latency by 88.0% (1.88 times) relative to unmerged LoRA and reduces token throughput by about 49.0%. We discuss this cost as a practical limitation of HiLA in the main text and identify reducing historical-retrieval computation and inference latency as directions for future optimization.
Response by Authors, Part 3
(Q3) Independently trained parameter-matched current-layer nonlinear adapter
To address the concern about additional adapter capacity, we add a parameter-matched current-layer nonlinear adapter, CNA-PM (Current-layer Nonlinear Adapter with Parameter Matching), to directly test whether adding only current-layer nonlinear mapping capacity can achieve performance close to HiLA.
CNA-PM retains the standard LoRA branch. It feeds the current LoRA bottleneck representation through an RMSNorm without affine parameters and a two-layer GELU network without biases, injects the output into the original bottleneck through a residual connection, and then applies the LoRA up-projection. The network is shared across layers and across the four modules corresponding to HiLA's historical branch. It does not read information from any historical layer.
Its additional parameter count is . CNA-PM and HiLA have the same total number of trainable parameters, 33,140,864, and the same trainable-parameter ratio, 1.0315%. They use the same LoRA initialization, training configuration, and random seed 42 for independent training.
The results on eight commonsense reasoning tasks with Llama-3.2-3B are as follows:
| Method | Trainable parameters (%) | BoolQ | PIQA | SIQA | HS | WG | ARC-c | ARC-e | OBQA | Average accuracy |
|---|---|---|---|---|---|---|---|---|---|---|
| HiLA | 1.0315 | 70.55 | 86.51 | 80.81 | 90.80 | 85.00 | 76.96 | 89.06 | 84.00 | 82.96 |
| CNA-PM | 1.0315 | 62.08 | 85.53 | 78.05 | 81.92 | 82.40 | 72.70 | 84.18 | 80.20 | 78.38 |
Under the same parameter budget, HiLA outperforms CNA-PM on all eight tasks and improves the average accuracy by 4.58%. This shows that adding only current-layer nonlinear mapping capacity is insufficient to match the performance of historical-difference retrieval.
This result complements the historical-intervention experiment in Section 4.3, Table 5 of the original manuscript. Full HiLA reaches 68.84% accuracy on GSM8K, which decreases to 59.14% when historical information is removed and to 41.93% when historical information is shuffled. CNA-PM independently trains a parameter-matched adapter that uses only current-layer information, providing further evidence that historical information contributes beyond additional adapter capacity.
References
[1] Alsuradi et al. Grouped Adaptive Weight Sharing for Parameter-Efficient Fine-tuning. Findings of ACL 2026.
[2] Zhang and Pilanci. Spectral Adapter: Fine-Tuning in Spectral Space. NeurIPS 2024.
[3] Peng et al. Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection. arXiv:2605.08177, 2026.
[4] Zhang et al. IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning. arXiv:2607.22251, 2026.
Response by Authors, Part 1
Your comments on statistical reliability, the inference measurement protocol, and additional adapter capacity are greatly appreciated. In response, we add two independent training seeds for mathematical reasoning, a code-generation task with multi-seed evaluation, multi-seed statistics for commonsense reasoning and dialogue generation, and an independently trained parameter-matched current-layer nonlinear adapter as a control. We respond to each point below.
(W1, Q1) Multi-seed statistics, confidence intervals, and validation-set selection
We add experimental statistics for four task categories to test whether HiLA's gains depend on particular conditions and clarify how each method selects its validation set.
1. Commonsense reasoning: five-seed results
Main Table 3 of the revised manuscript reports the eight-task average accuracy and standard deviation over five random seeds for four backbones. The comparison between HiLA and the strongest baseline on each backbone is as follows:
HiLA achieves the highest eight-task average accuracy on all four backbones, with standard deviations ranging from 0.28 to 0.32. Its gains over the strongest baselines range from 0.21% to 0.40%.
2. Open-domain dialogue generation: five-seed results
Main Table 4 of the revised manuscript reports the means and standard deviations of six generation metrics for Llama-3.2-3B on ConvAI2 over five random seeds. The comparison with the strongest baseline, DoRA, is as follows:
HiLA has the highest mean on five of the six metrics, while its METEOR score is 0.11 lower than DoRA's. For the small BERTScore-F1 gain, the difference is 0.02, and both methods have a standard deviation of 0.02. We therefore report both means and standard deviations to show the size of this gap relative to random variation.
3. Mathematical reasoning: two additional independent training seeds
The original main table reports results for only training seed 42. Following your suggestion, we conduct two additional independent training runs with separate random seeds. Together with the original seed 42, they form three independent training results. Appendix A.6, Table 12 of the revised manuscript reports the mean, sample standard deviation, and 95% confidence interval over the three seeds:
All four two-sided one-sample t tests reach p<0.05. For each of the four model-dataset combinations, the lowest HiLA result among the three seeds is higher than the fixed reported value of the corresponding low-rank baseline.
For FFT, all three Llama-3.2-3B seeds exceed FFT on GSM8K and MATH, whose FFT results are 65.13% and 16.84%, respectively. The three Llama-3.1-8B seeds also remain above FFT on GSM8K, whose FFT result is 73.39%, but remain below FFT on MATH, whose result is 36.36%. These cross-seed results further support the stability of the mathematical-reasoning advantage across repeated training runs.