acceptodds
Under review as a conference paper at ICLR 2027

BEYOND SCALAR BUDGETS: AUDITING RESOURCE SUFFICIENCY DECISIONS IN PEFT

Abstract

Parameter-efficient fine-tuning (PEFT) seeks lower-resource configurations that meet a specified quality tolerance. We study whether simple resource rules preserve the sufficiency decisions obtained from directly trained configurations. Specifically, we audit rank thresholds, composition of separately tested parameter and data reductions, and interchangeability at equal parameter count. Across the tested settings, all three rules can disagree with measured decisions: passing ranks need not admit a threshold, separately passing reductions can fail jointly, and equal-count configurations can receive different all-seed labels. We formulate a finite decision-validity audit that compares rule-implied decisions with same-seed paired measurements and quantifies departures from monotonicity on complete rank–data grids. Offline replays translate these violations into concrete selection errors: missed lower-resource passing configurations and false-sufficient predictions. On prospectively frozen GSM8K grids, Qwen 1.5B and 7B configurations incur worst-seed losses below 1 pp relative to their rank-eight/full-pool references while using one eighth of the adapter parameters and 3.85-fold or 1.93-fold fewer distinct examples visited. These results support validating resource rules against measured configurations rather than assuming that scalar budgets preserve sufficiency.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Rank Selection, Data Efficiency, Resource Sufficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MYPDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a finite decision-validity audit for PEFT resource selection rules. Experiments across multiple models and tasks reveal rule failures under certain settings and identify low-resource configurations that meet specified performance tolerances.

Critical strengths
  1. The work systematically examines three simplified resource selection rules in PEFT from a decision-making perspective, providing practical insights into the limitations of conventional resource allocation assumptions.
  2. Authors provide empirical evidence across multiple models and tasks (Sections 4-7), demonstrating failures of these resource selection rules under certain settings.
Critical weaknesses
  1. The decision-validity audit seems to offer limited methodological novelty beyond formalizing finite pass/fail comparisons, with limited explanations for the observed rule failures. In Section 4.3, local learning-rate selection removes the rank-threshold violation for T5/QNLI at the 3-pp tolerance among the tested ranks, raising concerns about the robustness of the observed rank-threshold violations across training settings.
  2. Although some configurations reduce adapter parameters and distinct training examples visited, the proposed audit does not establish computational savings or provide a cost-efficient selection procedure. This may limit its practical applicability to resource selection under constrained computational budgets.
Questions for authors
  1. Given that local learning-rate selection removes the rank-threshold violation for T5/QNLI at the 3-pp tolerance among the tested ranks (Section 4.3), how should the observed resource-rule failures be interpreted across different training configurations?
  2. The proposed audit does not establish computational savings. How can it guide practical resource selection when exhaustive configuration evaluation is computationally prohibitive?
2 comments
AuthorsOpenReview ↗

Response to Reviewer MYPD

Thank you for the questions about LR sensitivity and evaluation cost. We address them separately below. Section references refer to the prepared revision.

Q1. What does the removal of T5/QNLI fragmentation after LR selection mean?

The T5 result shows that the default recipe caused failures that local LR selection could repair. Section 5 now states this directly: those failures do not establish intrinsic rank demand.

On the common T5 ranks {1,2,4,8}, shared rank-specific LR selection changes the 3-pp set from {1,8} to all four ranks. The rank-eight-anchored stopping replay consequently changes its recommendation from rank eight to the measured minimum, rank one. At 1 pp, the selected set is {1,4,8}, so the same replay returns four and misses one. A more permissive retrospective per-rank/per-seed oracle makes all four pass at both tolerances. This oracle result is also retained in the revision.

On Pythia/MRPC at 3 pp, shared selection changes the return from eight to six, while rank one remains a measured pass. This result is seed-sensitive: rank four's losses are 0, 0, and 4.167 pp, while an additional previously inspected seed passes. We therefore do not claim recipe- or seed-invariant rank fragmentation.

ARC provides a separate validation/test comparison: one LR per rank is selected by mean validation accuracy, then the selected checkpoints are evaluated on the sealed test. Rank two's losses are 1.621, 0.768, and 0.939 pp, whereas ranks one and four each have worst loss 0.853 pp. The 1-pp set is {1,4,8}; all ranks pass at 3 pp. The seeds are unchanged and all ranks select the largest candidate LR, so this is held-out-question evidence within a local LR range, not independent-seed confirmation.

Q2. What can ReDA provide without exhaustive evaluation?

The required coverage depends on the claim. To verify a proposed configuration's usability, one needs its paired measurements against the reference. To refute a domain-wide rule, one decisive counterexample suffices. To claim a tested minimum, one must additionally exclude every cheaper candidate in the declared domain. Missing comparisons remain explicit rather than being filled in by an unverified ordering assumption (Section 3, Table 1).

For example, in the Llama half-pool comparison (Section 2), rank-two/half-pool passes at 3 pp and the only cheaper tested rank at that pool size fails. Those comparisons settle that restricted choice without requiring every point in the two-dimensional grid. Our study already measured the grid; this example explains the required evidence and is not a measured search-speed improvement.

Methodological contribution and resource scope.

ReDA specifies the evidence needed for a resource claim and checks whether the available measurements supply it. It makes claim coverage explicit, separates rule replay from rule-class representability, and turns disagreements into auditable selection consequences. The Qwen operating points use one eighth of the adapter parameters and fewer distinct examples visited, but retain fixed training steps and exposures. These results do not establish compute savings. With a limited evaluation budget, the user can report a verified operating point while leaving minimality unresolved. We have not demonstrated a faster search procedure.

Thank you for the detailed response and clarifications, which partially address my concerns about learning-rate sensitivity and clarify the scope of the proposed audit. However, my concerns about methodological novelty and practical computational efficiency remain. I also encourage the authors to further improve the clarity and readability of the manuscript, as highlighted by Reviewer 4kDo. Overall, I will maintain my original rating.

Reviewer_9jWLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines whether simplified resource rules preserve sufficiency decisions in parameter-efficient fine-tuning. A configuration passes when its performance loss relative to a fixed, same-seed rank-eight/full-pool reference remains within a declared tolerance for all three observed seeds. The authors audit rank thresholds, composition of separately acceptable parameter and data reductions, and interchangeability of equal-parameter-count configurations. They provide finite decision-validity criteria, measure departures from upward-closed passing sets, and use offline replays to quantify disagreements introduced by specified selection rules.

Experiments combine historical classification results, directly trained joint-resource and matched-count controls, validation-selected ARC-Challenge evaluations, and prospective decoder grids. The reported results include a rank-threshold violation on ARC at 1 percentage point, marginal-to-joint failures on BERT/QNLI and Llama/GSM8K, and different sufficiency labels among equal-count T5 configurations. Selected Qwen GSM8K configurations remain within 1 percentage point of their references using one eighth of the adapter parameters and fewer distinct training examples. The conclusions are explicitly limited to the measured configurations, training contracts, tolerances, and seeds.

Critical strengths

1.A clear and useful separation of distinct resource-selection assumptions. Sections 2.2–2.3 distinguish when a rank threshold exists, when a resource summary preserves labels, and whether marginal reductions compose. The Llama GSM8K example makes this distinction concrete: its passing map is upward closed at both declared tolerances, yet separately passing parameter and data reductions can fail jointly (§5; Table A9). This contributes a useful framework for interpreting resource-efficiency claims and avoids conflating different logical requirements.

2.Direct experimental counterexamples, supported by a meaningful prospective component and careful resource accounting. The BERT/QNLI experiment directly trains the joint configuration: both marginal reductions pass at 1 percentage point, while their intersection fails on all three seeds (§5; Table A5). The ARC study selects learning rates on validation data before evaluating the selected checkpoints on the sealed test set (§3; Appendix C). The decoder experiments also identify concrete lower-resource operating points, while Appendix O correctly distinguishes fewer adapter parameters and distinct examples from unchanged training steps and exposures. These are substantive empirical strengths.

Critical weaknesses

1.The incremental scientific contribution and practical relevance of the audited shortcuts are not sufficiently established.

As acknowledged in §8, prior work already documents weak or inverse gains from increasing PEFT parameters and sensitivity to parameter allocation, scaling, and optimization. Propositions 1–2 provide useful organization but are elementary consequences of the definitions. The main selection replays evaluate deliberately specified diagnostic rules: stop-after-first-failure in §4.2 and Appendix P, and upward closure in §7.1. The paper does not establish that these exact rules describe an important existing selection workflow or quantify their practical prevalence.

The composition finding also requires careful interpretation. Two individually acceptable reductions can exceed a common tolerance when combined even under an additive, monotone loss model; this does not require a distinctive PEFT interaction. The authors correctly avoid claiming such an interaction, but the observation alone offers limited additional insight.

These points matter because the acceptance case for this empirical study depends on the new decision-relevant knowledge it provides. The framework is useful, but the current evidence does not yet make a sufficiently strong case for its broader significance.

2.The robustness of the central decision failures remains unresolved under sampling variability and reasonable recipe changes.

Several influential labels lie close to the 1-percentage-point boundary. Two Qwen 1.5B failing cells have worst-seed losses of 1.061 pp, while Qwen 7B rank-one/full-pool passes at 0.986 pp (Figure 4; Table A8; Appendix N). One GSM8K example contributes approximately 0.0758 pp to an accuracy difference. With three seeds and no paired uncertainty analysis, the stability of these boundary decisions is unclear.

The quantitative conclusions are also sensitive to the tolerance and training recipe. Qwen 1.5B’s order defect falls from (3/15) to (1/15) when the tolerance changes from 1.0 to 1.1 pp (Table A11). The remaining violation persists, so the evidence should not be dismissed as entirely a boundary effect. Separately, local learning-rate selection changes T5/QNLI’s 3-pp passing set from ({1,8}) to ({1,2,4,8}) and removes its corresponding search miss on the common grid (§4.3; Appendices F and Q).

The paper explicitly acknowledges its finite scope and recipe dependence. Nevertheless, these unresolved issues limit how much reliable guidance the observed failures provide beyond the particular recorded runs. The retained GSM8K records also lack raw generations, restricting paired uncertainty and parsing/truncation checks (Appendix M.1).

Questions for authors

1.Which concrete PEFT resource-selection practice is represented by the audited threshold, binary marginal-composition, or equal-count rules? Can you provide evidence that the identified disagreements change a meaningful selection decision beyond the diagnostic replays? This would directly affect the assessment of significance.

2.Which central counterexamples, if any, have been confirmed with independent seeds or validation-selected recipe checks? Please provide the relevant paired results and clarify whether per-example predictions can be supplied or regenerated. Evidence that the important violations persist with adequate margins would materially strengthen the experimental assessment.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9jWL

Thank you for the detailed assessment. We agree with your additive-loss objection and with the distinction between tolerance sensitivity and sampling stability. Below we clarify what the existing evidence establishes and where it remains incomplete. Section references refer to the prepared revision.

Q1. Which practice is represented, and what meaningful decision changes?

The stopping and upward-closure procedures are specified diagnostic rules. We do not establish their prevalence in deployed PEFT workflows or attribute them to adaptive-rank methods. We have made that boundary explicit in Related Work and the discussion. We use them to examine a specific question: whether available measurements justify accepting a combined cut, skipping a cheaper candidate, or substituting an equal-count design.

Section 2 now follows a choice with the training pool fixed at half size. On Llama/GSM8K at 3 pp, rank-one/full-pool and rank-eight/half-pool separately pass, with worst losses 2.199 and 0.834 pp. Their joint configuration fails at 3.184 pp; rank-two/half-pool passes at 1.820 pp. These measured comparisons change the acceptable choice from the proposed rank one to rank two, and establish two as the smallest tested rank at that pool size. The map is still ordered at 3 pp. This is a decision consequence on the measured grid, not a prospective deployment study or an estimate of real-world error frequency.

We also agree with the additive-loss objection. On BERT/QNLI, the seed-wise sums of the marginal losses are -0.098, 0.977, and 1.660 pp. The last already predicts an all-seed failure at 1 pp, although all sums understate the measured joint losses (1.953, 1.758, 1.855 pp). The revision uses this comparison directly: two binary PASS labels discard how much of a common loss allowance each cut consumes. This finding does not establish a nonlinear or PEFT-specific interaction. Its role is to distinguish binary composition from ordering and from direct joint verification.

The contribution we claim is the audit procedure together with the measured decision consequences. ReDA organizes the comparisons needed for a claim, distinguishes observed support from missing coverage, and separates a particular rule's errors from rule-class representability. The experiments identify cases where an ordered map fails to support a joint cut and where LR selection changes the minimum returned by a stopping rule.

Q2. Independent seeds, recipe checks, and paired evidence.

We have not established independent-seed confirmation of the central counterexamples. ARC provides validation-selected LR followed by sealed-test evaluation on different questions, with the same three training seeds. Its paired test losses are:

RankSeed 42Seed 73Seed 101
10.2560.853-0.768
21.6210.7680.939
40.6830.853-0.085
8000

All values are pp relative to the same-seed rank-eight reference. Thus the 1-pp failure is driven by rank two/seed 42; all ranks pass at 3 pp. All selected LRs are the largest tested candidate. We do not interpret this as robustness to a wider tuning range or new seeds.

Several Qwen labels are indeed close to the cutoff. Two 1.5B cells at 1.061 pp pass at 1.1 pp, reducing the defect from 3/15 to 1/15. The remaining rank-four/full-pool cell loses 2.274 pp and still fails at 2 pp. Qwen 7B rank-one/full-pool passes at 0.986 pp; its failing rank-four/full-pool cell loses 1.440 pp. These margins characterize the recorded outcomes, not confidence bounds.

Shared LR selection repairs T5's 3-pp search miss on the common ranks. Pythia's selected rank-four failure is seed-dependent: its losses are 0, 0, and 4.167 pp, and an additional previously inspected seed passes at -1.225 pp. The revision reports this additional passing seed alongside the three-seed failure.

Prediction availability and uncertainty.

Matched item predictions are available for two BERT/RTE checkpoint pairs. Paired bootstrap 95% intervals are [-6.137,3.249] and [-3.610,5.776] pp; both cross 3 pp despite point-estimate passes. This does not establish uncertainty bounds for the central ARC/GSM8K cases. Their retained counts and prediction digests cannot recover paired item disagreements, and raw GSM8K generations were not retained. Recovering those predictions would require reevaluating the corresponding checkpoints under the recorded protocol; we have not done so and do not claim that regeneration is guaranteed from the distributed artifact.

The revised appendices also provide all 90 existing T5/Pythia LR endpoints and deterministic shared-LR recomputation. These are existing measurements organized for recomputation, not new training or independent validation. Our conclusions remain about the measured configurations and specified rules; sampling stability and deployment prevalence remain unresolved.

Reviewer_9jWLOpenReview ↗

Follow-up on the rebuttal and revised draft

Thanks for the clear response and for updating the draft.

That said, your response basically confirms my two main concerns. First, on the composition point, adding the two marginal drops already goes over the 1% cap on BERT, so the joint failure is really just what happens when two small drops add up past a shared threshold. And as you noted, the stopping and upward-closure rules are diagnostic baselines rather than how people actually tune PEFT in practice. Second, with only 3 seeds and a strict worst-seed cutoff, several of the main counterexamples still come down to a single bad seed (like on ARC and Pythia) or points sitting right on the 1% boundary without uncertainty intervals. Because of that, I still find the empirical takeaway too narrow and noisy, so I am keeping my overall score unchanged.

Reviewer_4kDoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether simple rules can be applied in determining how much resources can be saved during PEFT, without reducing quality too much. The authors test three common rules, which are 1) If one LoRA rank works, then other larger ranks would also work. 2) If both using fewer parameters and using less data work, then applying both jointly would work. 3) Using the same number of trainable parameters would yield similar performance. Their experiments demonstrate that these rules often fail, leading to rejection of cheaper setup that would actually work, and choosing a worse setup than expected. Based on these observations, the authors argue that one cannot safely choose PEFT setup using only a simple resource counts such as LoRA ranks, training data size, or trainable parameter count.

Critical strengths
  • The paper targets an important question: Is it possible to use simple rules to decide which low-resource configuration will be able to meet the expected model performance via PEFT.
  • The authors show real counterexamples where simple assumptions fail to demonstrate expected results.
Critical weaknesses
  • The paper is very difficult to read and understand in its current form, and requires substantial effort in decoding its meaning. There are many words that are abstract, or should be defined before usage. Some examples are: rank threshold, endpoint, missed passing configurations, false-sufficient decisions. These terms can be interpreted in many ways and thus needs more details to fully understand. Lots of sentences demonstrate similar readability issues. For example, what does "quantifies depatures from monotonicity on complete rank-data grids" mean? This problem persists throughout the manuscript, obscuring the motivation, methodology, and interpretation of the results. Substantial revision using precise definitions, concrete examples, and more direct explanations would improve its readability, facilitating its technical assessment.
  • The paper’s central claim is difficult to identify, partly because of its organization. The authors show that simple rules cannot reliably predict the resource savings in PEFT and cannot guide decisions in the training configurations. However, it remains unclear whether the primary contribution is to establish the limitations of these simple rules, propose direct testing as alternative, or providing empirical guidance for selecting these configurations. Explicitly stating the central claim and organizing the analysis around it would clarify the paper's contribution.
Questions for authors

See weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4kDo

Thank you for pointing out where the terminology and organization made the paper difficult to follow. We have revised the introduction, added a worked example before the definitions, and clarified the audit outputs. The main text remains nine pages. All section references below refer to this prepared revision.

1. Central question and contribution.

The question is: what evidence justifies accepting, skipping, or substituting a PEFT configuration at a specified quality tolerance? Our contribution is a measurement methodology, Resource Decision Audit (ReDA), that links these claims to their required comparisons and reports support, a counterexample, or missing evidence. It also distinguishes the errors of a specified procedure from whether any rule in an allowed class can represent the observed labels. The experiments show where these requirements differ and how those differences change a resource choice. We report the directly evaluated resource reductions separately; ReDA did not discover them through a new search algorithm.

2. A concrete example now precedes the definitions (Section 2).

For Llama/GSM8K at a 3-pp allowance, rank one/full pool and rank eight/half pool separately pass, with worst-seed losses of 2.199 and 0.834 pp. Their combination, rank one/half pool, loses 3.184 pp and fails. Rank two/half pool loses 1.820 pp and passes. At that pool size, these last two measurements establish rank two as the smallest tested passing rank. The complete map is nevertheless ordered at 3 pp: moving upward from a pass preserves a pass. Thus ordering does not justify combining downward resource cuts. The example gives a specific choice that the audit can resolve.

3. Definitions and outputs are now explicit (Section 3).

An endpoint is one configuration's score for one training seed. A configuration passes when its largest same-seed loss over the three recorded seeds is at most the stated tolerance. A rank threshold predicts that every rank at or above a cutoff passes. An incorrect acceptance is a configuration that a replay accepts but the measurements label FAIL; a missed passing candidate is a measured PASS that the replay skips. An unmeasured configuration is left unlabeled, not called FAIL.

For the monotonicity measure you asked about, a defect of 3/15 means that every map in which increasing rank and pool size preserves PASS disagrees with at least three of the fifteen measured labels. This is different from counting errors made by a particular selection rule.

4. Organization and interpretation.

Sections 4 to 6 now separate rule failures, recipe/tolerance sensitivity, and directly verified reductions. Figure 1 presents evidence requirements and outputs; Figure 2 shows measured losses and labels. The distinction concerns the rule used with a summary: a rank lookup can represent labels even when a rank threshold cannot. A measured pass supports using that configuration; calling it the tested minimum requires excluding cheaper candidates.

Reviewer_4kDoOpenReview ↗

Response to Author Rebuttal

Thank you to the authors for providing detailed responses to my concerns and for making corresponding changes to the revised manuscript.

I appreciate the revisions and think that they help improve the presentation of the paper. However, I believe more substantial work is needed to improve the overall readability of the manuscript and the clarity with which the paper’s main contributions are communicated. These concerns also appear to be shared by other reviewers. For this reason, I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.