- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose ShadowPEFT, a novel PEFT method that couples the frozen backbone to a compact shadow network via layer-wise bottleneck injection and gated residual updates. The shadow design supports both attached refinement and shadow-only inference through separate training, enabling edge-cloud collaboration. They empirically evaluate their method against LoRA, DoRA, and other PEFT baseline methods on language benchmarks, DreamBooth, and a robotic routing task, showing competitive accuracy.
Critical strengths
- The paper is well written and is easy to follow
- The detachable design of the adapter can be attached to the full model to improve accuracy or detached as a standalone edge predictor
- Their method consistently achieves better performance than the baselines at lower parameter counts
- The ablation study highlights the importance of the inject and update modules and shows that both are required for improved accuracy.
Critical weaknesses
- Unlike LoRA, where the weights can be merged into the backbone model for zero inference latency and memory overhead, ShadowPEFT cannot be merged with the backbone
- Table 11 reports the throughput and memory usage for a single synthetic optimizer step, but the total wall-clock training time for the actual benchmarked models is missing.
- For budget-matched setups, detached shadow performance completely collapses (such as 1.9% on MMLU) unless initialized with a heavy 455M pretrained checkpoint that destroys parameter efficiency.
Questions for authors
- What is the actual wall-clock fine-tuning time for all the methods on the main benchmarks?
- Can you show the decoding throughput and peak KV-Cache memory for the 4B and 8B models?
Response to Reviewer y3qv [1/2]
We would like to thank Reviewer y3qv for your helpful comments. Below are our answers to your concerns.
W1: Experimental results gain is marginal.
Answer:
We agree that the per-cell margins are modest. We would argue that this is characteristic of progress in PEFT, not a weakness specific to ShadowPEFT. DoRA, a widely used LoRA variant, improves over LoRA by only +0.27/+0.30 average points at 0.6B/4B and falls 0.52 points below LoRA at 8B (Table 1); even its positive gains are smaller than ShadowPEFT's relative improvement.
Against that backdrop, we restate the contribution. ShadowPEFT is not a new branch of the LoRA family, and we do not claim state-of-the-art accuracy. It is a different PEFT paradigm: the trainable adaptation is concentrated in a shadow network that is itself a complete predictor. A single training run therefore supports two deployment modes: attached to the backbone, or detached as a standalone model.
The margins are not from seed artifacts: we ran extensive experiments without selecting favorable seeds, and the improvement direction is consistent throughout (best average at every scale, 14/15 cells, all OOD groups; see also W2). Moreover, the gains trace to the shadow pathway rather than chance: removing the state update at matched budget reduces accuracy on both task types (Table 3).
W2: The significance test is unexplained. Table 1 gives no standard deviations. The claim of “p < 5% in a paired t-test” does not say what is paired: runs, tasks or examples. A p-value from 5 cells of averages would not mean much.
Answer: Thanks for pointing out this issue. The reported p-value in Table 1 comes from a paired t-test between ShadowPEFT and DoRA computed on the five-run average scores; it was not conducted per task, and the caption did not state this. We will add it in the camera-ready version.
W3 and Q1: How does persistent shadow state make the proposed method more effective over prior side-network and representation-level methods like LST or ReFT?
Answer:
We agree that these families exist, and we have discussed the differences between LST and ReFT in Section 2. The distinction we claim is not the presence of a side network or a gating function. It is the combination of three properties:
Compared with LST. LST is compared directly in Table 2 at matched budgets and underperforms both ShadowPEFT and LoRA substantially. Its ladder consumes frozen features in one direction; ShadowPEFT's coupling is bidirectional, and its shadow is jointly supervised as a standalone predictor.
Compared with ReFT. ReFT-style interventions are layer-local and independent across depth: successive interventions cannot accumulate task information through a shared state. ShadowPEFT's injections, by contrast, are connected by the evolving shadow state through the update module.
Compared with recurrent mechanisms. We acknowledge that the gated update is GRU-like (cited as Cho et al., 2014, in Section 3.3) and we do not claim novelty for the gate itself. Novelty is at the method level, not the component level: gated recurrent updates are an established building block. The method-level contribution here is the persistent cross-layer state together with its detachable, jointly trained predictor — properties (i)–(iii) above, which the LST comparison and the Table 3 ablation demonstrate empirically.
Q2: A clear statistical significance analysis for the gains in Table 5.
Answer:
Thanks for your suggestions. All methods in Table 5 follow the same evaluation protocol:
With both the trained checkpoint and the seed fixed, inference is almost deterministic: rerunning the evaluation reproduces nearly identical scores. There is no run-to-run variation left to estimate, so we did not attach a significance test to Table 5. We will state this protocol explicitly in the Table 5 caption in the final version.