acceptodds
Under review as a conference paper at ICLR 2027

CAMS: RESIDUAL CAPABILITY ALLOCATION FOR CONTINUED TOOL-USE SFT

Abstract

Data selection for tool-use supervised fine-tuning commonly ranks records by global quality, relevance, or estimated utility. Staged adaptation raises a more specific question: once a model has completed initial adaptation, which remaining training records should form the next batch for continued training under a fixed budget? We define this setting as marginal continuation selection and introduce Capability-Aware Marginal Selection (CAMS). CAMS matches what the current model still needs with what supervision each candidate record provides. A frozen train-side probe supplies capability accuracies , yielding as a residual-headroom proxy; gold tool calls and TRAIN statistics supply observable supervision demands . CAMS ranks candidates by and admits intact records under the continuation budget. The current instantiation uses two auditable demand axes: multi-binding structure and API exposure scarcity. We evaluate on the frozen API-Bank train-side confirmation set (FCS): 256 converted API-call records disjoint from training, the capability probe, and selector-construction records. On Qwen3-8B under the frozen API-Bank continuation protocol, CAMS achieves 35.29% exact API-call correctness on FCS, the highest mean among five baselines fixed before confirmation evaluation. It exceeds the strongest pre-frozen baseline by 1.83 percentage points and same-state Random by 2.65 points, with positive CAMS–Random differences in all three training states. Post-hoc descriptive CAMS–Random differences are 10.0 points on multi-argument calls and 13.43 points on rare-API calls. CAMS frames continued tool-use SFT as model-state-aware residual capability allocation: matching current needs to observable supervision in the remaining data, rather than repeating a static global ranking.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, supervised fine-tuning, tool use, large language models, continual adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_HGnfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates marginal continuation selection: after a model has undergone an initial stage of tool-use supervised fine-tuning (SFT), select a second-stage subset from the remaining records under a fixed serialized-token budget.

Critical strengths

None

Critical weaknesses
  • Fatal for the main contribution: no experiment demonstrates model-state-aware residual allocation.
  • The empirical advantage is small, uncertain, intervention-dependent, and very narrow.
Questions for authors

Evidence request: Can the authors provide a fully frozen evaluation on a new, untouched confirmation surface in a regime where state-specific deficit magnitudes actually produce different selected memberships? A decisive test should report the selected IDs per state and compare CAMS against an unweighted both-demand/static-intersection selector, shuffled demands, and multiple hash tie-break seeds under the same supervised-label count, update count, sequence/context exposure, and scheduler endpoint. Paired per-example uncertainty for CAMS–Random and CAMS–static should also be reported. Without such evidence, what empirical result in the submission isolates any benefit from model-state-aware weighting?

Reviewer_2q51OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies data selection for continued supervised fine-tuning of tool-use models after an initial adaptation stage. It proposes Capability-Aware Marginal Selection (CAMS), which measures capability accuracies on a fixed probe set, defines residual deficits as Dk=1−CkD_k=1-C_k, and ranks candidate training records using S(τ)=∑kDkVk(τ)S(\tau)=\sum_kD_kV_k(\tau), where VkV_k indicates observable supervision demands. The implementation uses two binary demand dimensions: multi-argument calls and low-frequency APIs.

Experiments use Qwen3-8B and API-Bank. On a 256-record confirmation set, CAMS achieves 35.29% API-call correctness, exceeding Random by 2.65 percentage points and the strongest pre-frozen baseline by 1.83 points. Additional analyses examine demand definitions, deficit calibration, selection saturation, mechanism controls, training-budget sensitivity, and external transfer.

Critical strengths
  1. The method is transparent and its selection behavior is directly analyzable. CAMS combines explicitly defined deficits, observable demand indicators, and deterministic budget allocation. This makes individual scores and selection decisions straightforward to inspect. The formulation also enables a precise characterization of when state-dependent weights affect selection and when candidate-pool saturation renders them inactive. This provides an analyzable example of capability-oriented data selection.

  2. The diagnostic experiments provide informative negative evidence. The paper reports demand-shuffling, coverage-matching, supervision-budget controls, and negative external results alongside the positive result under the original protocol. These analyses help distinguish residual-capability coverage, training exposure, and downstream performance, prov

Critical weaknesses
  1. The main experiment does not test the practical value of capability-deficit weighting. Section 6.2 and Appendix F show that all three model states select exactly the same 32 records. Because both deficits are positive, records satisfying both demand indicators always rank first, and this category contains 4.05 times the available token budget. Membership is therefore determined entirely by the fixed hash ordering and budget cutoff. Setting both weights to 1 would produce the same selected subset; measuring model capabilities does not change the training data in this setting. Furthermore, Table 10 reports 36.59% for both ShuffledDemand and CoverageMatchedRandom, compared with 35.29% for CAMS. The evidence therefore does not establish a benefit from accurately matching measured deficits to candidate demands over static selection or shuffled labels. The saturation analysis explains this behavior clearly, but does not provide the missing validation of the central mechanism.

  2. The reported advantage is sensitive to the training protocol and lacks robust empirical support. The main experiment matches total serialized tokens, while supervised-token counts, optimizer updates, and learning-rate endpoints differ. Table 2 reports 935 supervised tokens for CAMS versus 747–826 for Random. Under approximately matched supervised-token budgets, Table 12 reports 33.46% for CAMS versus 34.46% for Random. When supervised labels and update counts are matched, CAMS underperforms Random in eight of the nine comparisons in Table 13. These results do not invalidate the numerical improvement under the original protocol, but they leave its attribution to more effective residual-capability allocation unsupported. The principal positive result comes from one small internal confirmation set, and the available budget controls do not corroborate the proposed explanation.

Questions for authors
  1. Please confirm whether fixing Dmulti=Drare=1D_{\text{multi}}=D_{\text{rare}}=1, while retaining the same hash ordering and budget, produces exactly the same 32 training records in the main experiment. If so, which existing experiment isolates a benefit from measuring the model’s capability deficits?

  2. In each paired comparison in Table 13, do CAMS and Random restore the same Stage-1 optimizer and scheduler states and use the same supervised-loss normalization? Are there implementation differences beyond the selected data and its composition that could explain the reversal after matching supervised labels and update counts?

  3. Using the existing per-record predictions, can the authors report uncertainty for the paired CAMS–Random difference in the main experiment, explaining how shared evaluation records and training randomness are handled? The reported across-model standard deviations do not directly characterize uncertainty in the primary improvement.

Reviewer_w53AOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies "marginal continuation selection": after an initial tool-use SFT stage, which remaining records should be selected for continued training under a fixed token budget. It proposes CAMS, which scores each candidate as ∑kDkVk(τ)\sum_k D_k V_k(\tau), where Dk=1−CkD_k = 1 - C_k is a deficit from a frozen train-side probe and VkV_k are two binary demand indicators (≥2\geq 2 arguments; API in the bottom half of TRAIN frequency). Candidates are ranked by score with a hash tie-break and admitted until a serialized-token budget is met. On a 256-record API-Bank train-side confirmation set (FCS) with Qwen3-8B + LoRA, CAMS reaches 35.29% exact API-call correctness vs. 32.64% for Random and 33.46% for the best pre-frozen baseline. Appendices report construct audits, mechanism controls, budget-matched variants, and negative external (AppWorld / STATE-derived) results.

Critical strengths

The method is simple, deterministic and fully auditable, and the paper is unusually transparent: all negative and null results (saturation, mechanism controls, exposure-matched runs, external transfer) are reported rather than omitted. This makes the submission easy to evaluate accurately.

Critical weaknesses
  1. The central claim ("model-state-aware allocation") is not tested by the experiments. Section 6.2 and Appendix F show that the both-demand pool (134k tokens) is about 4x the Stage-2 budget (33k), so all three model states select the identical 32 records. The probe-derived deficits DkD_k change score values but never change membership. In the only setting evaluated, CAMS reduces to a hash-ordered prefix of records that satisfy both static demand labels; the model-state component of the method is inert. The paper therefore provides no evidence for the framing in the title, abstract and Section 1.

  2. The paper's own controls show that the demand labels are not responsible for the gain. In Appendix G.1, ShuffledDemand (demand labels randomly permuted across candidates) and CoverageMatchedRandom both outperform CAMS on FCS (0.3659 vs. 0.3529). Under supervised-token-matched budgets (Table 12) and at 8k/16k budgets (Table 14), CAMS is at or below Random. Taken together with (1), the remaining positive result -- +2.65 points over Random under one specific admission contract, on one dataset and one model -- does not support the conclusion that CAMS is an effective or principled selection rule.

Questions for authors
  1. Is there any setting (e.g., a larger budget or a smaller both-demand pool) in which the three states select different subsets, and if so, does CAMS still outperform Random there? Without such a result, the state-aware framing should be removed.

  2. Given that ShuffledDemand matches or exceeds CAMS, what evidence remains that the two demand axes carry information beyond API diversity? Please address this directly rather than deferring to future work.

Reviewer_Z35wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which records to keep in the next continuation SFT batch under a fixed budget, and proposes CAMS — a parameter-free selector that ranks candidates by a frozen probe's residual-deficit score times an observable demand indicator. On a frozen 256-record FCS slice CAMS posts the best mean among five pre-frozen baselines, but the gain is small and does not survive matched-supervision, smaller-budget, or external-portability checks; outside the tested setup, evidence for the selector's added value is limited.

Critical strengths
  1. Baselines, splits, tie-break hashes and token cutoffs are fixed before confirmation, FCS is disjoint from training/probe/selector construction (Sec 5.1, App. A), and the authors publish their own negative and mechanism-control results.
  2. The score S is auditable, so every term traces back to a frozen probe accuracy and a binary demand label (Sec 4.3, Fig. 2) — and the saturation regime (both-demand pool 4.05× budget) is diagnosed explicitly rather than hidden.
  3. DELIFT already frames selection as model-state-aware utility, LESS uses gradient-based low-rank features, and Adapt-∞ does dynamic selection for continual multimodal instruction tuning; CAMS's delta is the auditable parameter-free dot product. Tables 10–16 report mechanism controls, budget sensitivity, LESS-port and external STATE portability, including the cells where CAMS loses.
Critical weaknesses
  1. The headline +2.65 pp over Random appears under one budget contract and does not survive matched comparisons. It holds at the original serialized-token cutoff (35.29% vs 32.64%, Table 1) but flips under matched supervision tokens (33.46% vs 34.46%, Table 12), in 8/9 exact-label/update cells (Table 13), and at 8k/16k smaller budgets (Table 14).
  2. On the evaluated pool the state-aware component did not change the selected subset. The both-demand pool is 4.05× the Stage-2 budget, so all three training states pick the same 32 records by frozen hash (§6.2); the authors themselves state that state-specific deficits change score values but not membership. Whether state weights would actually move the selected subset in a less saturated candidate pool is left open.
  3. Mechanism controls score above CAMS on FCS, so the necessity of the specific components is not established. CoverageMatchedRandom (40 records/32 APIs) and ShuffledDemand (46 records/19 APIs) both reach 36.59%, above CAMS at 35.29% (Table 10); the paper concedes these do not establish that accurate demand labels or measured deficits are necessary for the gain.
  4. External portability is negative. On the STATE-derived Qwen3 surface (150 tasks, five runs each), CAMS is 10.27% vs Random 12.49% and Base 14.53%, and CAMS–Random is negative in all three domains (Table 16).
  5. The residual-headroom proxy D=1−C calibrates weakly. Within-capability Spearman ρ = 1.0 across seven proposals, but pooled raw-gain ρ is only 0.257 (Table 7, App. E), and the identity Δ = D + C_post − 1 mechanically couples deficit and gain through the shared baseline.
Questions for authors

Refer to above weaknesses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.