- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a finite decision-validity audit for PEFT resource selection rules. Experiments across multiple models and tasks reveal rule failures under certain settings and identify low-resource configurations that meet specified performance tolerances.
Critical strengths
- The work systematically examines three simplified resource selection rules in PEFT from a decision-making perspective, providing practical insights into the limitations of conventional resource allocation assumptions.
- Authors provide empirical evidence across multiple models and tasks (Sections 4-7), demonstrating failures of these resource selection rules under certain settings.
Critical weaknesses
- The decision-validity audit seems to offer limited methodological novelty beyond formalizing finite pass/fail comparisons, with limited explanations for the observed rule failures. In Section 4.3, local learning-rate selection removes the rank-threshold violation for T5/QNLI at the 3-pp tolerance among the tested ranks, raising concerns about the robustness of the observed rank-threshold violations across training settings.
- Although some configurations reduce adapter parameters and distinct training examples visited, the proposed audit does not establish computational savings or provide a cost-efficient selection procedure. This may limit its practical applicability to resource selection under constrained computational budgets.
Questions for authors
- Given that local learning-rate selection removes the rank-threshold violation for T5/QNLI at the 3-pp tolerance among the tested ranks (Section 4.3), how should the observed resource-rule failures be interpreted across different training configurations?
- The proposed audit does not establish computational savings. How can it guide practical resource selection when exhaustive configuration evaluation is computationally prohibitive?
2 comments
Thank you for the detailed response and clarifications, which partially address my concerns about learning-rate sensitivity and clarify the scope of the proposed audit. However, my concerns about methodological novelty and practical computational efficiency remain. I also encourage the authors to further improve the clarity and readability of the manuscript, as highlighted by Reviewer 4kDo. Overall, I will maintain my original rating.
Response to Reviewer MYPD
Thank you for the questions about LR sensitivity and evaluation cost. We address them separately below. Section references refer to the prepared revision.
Q1. What does the removal of T5/QNLI fragmentation after LR selection mean?
The T5 result shows that the default recipe caused failures that local LR selection could repair. Section 5 now states this directly: those failures do not establish intrinsic rank demand.
On the common T5 ranks {1,2,4,8}, shared rank-specific LR selection changes the 3-pp set from {1,8} to all four ranks. The rank-eight-anchored stopping replay consequently changes its recommendation from rank eight to the measured minimum, rank one. At 1 pp, the selected set is {1,4,8}, so the same replay returns four and misses one. A more permissive retrospective per-rank/per-seed oracle makes all four pass at both tolerances. This oracle result is also retained in the revision.
On Pythia/MRPC at 3 pp, shared selection changes the return from eight to six, while rank one remains a measured pass. This result is seed-sensitive: rank four's losses are 0, 0, and 4.167 pp, while an additional previously inspected seed passes. We therefore do not claim recipe- or seed-invariant rank fragmentation.
ARC provides a separate validation/test comparison: one LR per rank is selected by mean validation accuracy, then the selected checkpoints are evaluated on the sealed test. Rank two's losses are 1.621, 0.768, and 0.939 pp, whereas ranks one and four each have worst loss 0.853 pp. The 1-pp set is {1,4,8}; all ranks pass at 3 pp. The seeds are unchanged and all ranks select the largest candidate LR, so this is held-out-question evidence within a local LR range, not independent-seed confirmation.
Q2. What can ReDA provide without exhaustive evaluation?
The required coverage depends on the claim. To verify a proposed configuration's usability, one needs its paired measurements against the reference. To refute a domain-wide rule, one decisive counterexample suffices. To claim a tested minimum, one must additionally exclude every cheaper candidate in the declared domain. Missing comparisons remain explicit rather than being filled in by an unverified ordering assumption (Section 3, Table 1).
For example, in the Llama half-pool comparison (Section 2), rank-two/half-pool passes at 3 pp and the only cheaper tested rank at that pool size fails. Those comparisons settle that restricted choice without requiring every point in the two-dimensional grid. Our study already measured the grid; this example explains the required evidence and is not a measured search-speed improvement.
Methodological contribution and resource scope.
ReDA specifies the evidence needed for a resource claim and checks whether the available measurements supply it. It makes claim coverage explicit, separates rule replay from rule-class representability, and turns disagreements into auditable selection consequences. The Qwen operating points use one eighth of the adapter parameters and fewer distinct examples visited, but retain fixed training steps and exposures. These results do not establish compute savings. With a limited evaluation budget, the user can report a verified operating point while leaving minimality unresolved. We have not demonstrated a faster search procedure.