acceptodds
Under review as a conference paper at ICLR 2027

Target-Independent Micro-Interventions for Predicting Training Response Across Language-Model Families

Abstract

Benchmark scores describe current capability but do not determine how a checkpoint responds to further training. We introduce L-STATE, which combines current scores with capability changes from four short, target-independent training interventions. Its measured responses support direct and structure-preserving operator readouts, with a cross-family prediction bound that accounts for coordinate heterogeneity. Relative to current scores alone, both pulse readouts reduce prediction mean squared error by 39.4% across three development families. Reductions reach 78.3% for the operator on held-out GLM and 78.4% for the direct readout on held-out Granite. Fixed eight-step probes also predict longer responses: on nine GLM checkpoints under repeated-example adaptation, a post-hoc evaluation without refitting or rescaling retains 63.3% and 47.9% error reductions at 16 and 32 target steps. In a separate GLM/Yi comparison across repeated- and fresh-example training, full L-STATE reduces prediction error by 36.9–38.8% and action-selection regret by 24.8–68.8% at a common 32-step endpoint. These results show that short, reusable interventions provide information beyond static capability, improving both response prediction and training choices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Model Adaptation, Training Response Prediction, Micro-Interventions, Model State Representation, Cross-Family Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ULFcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes L-State, a 25-dimensional representation combining five current capability scores with changes induced by four target-independent, short LoRA training probes. Ridge-regression readouts use these measurements to predict the effects of candidate training actions across language-model families. The authors evaluate response prediction and action selection on several model families, including held-out GLM and Granite.

Critical strengths

The paper investigates a relevant question: whether short, reusable interventions provide information about future training responses beyond static benchmark scores. The held-out-family evaluations and explicit cost accounting are useful aspects of the experimental design.

Critical weaknesses
  1. Limited novelty and insufficient positioning against prior work. Predicting fine-tuning outcomes from probing or early-training signals has been studied extensively, including Predicting Fine-Tuning Performance with Probing (EMNLP 2022) and TuneAhead (ICML 2026). Related work such as LensLLM (ICML 2025) and COSMOS (2025) is not discussed. Given that the proposed readouts are essentially regression models over a small set of handcrafted training-response features, the substantive methodological advance and advantages over existing approaches remain unclear.

  2. Limited external validity. The 135 evaluated checkpoint states originate from only five base model families, with each family's states constructed through short synthetic LoRA training trajectories. This controlled setup does not adequately represent diverse real-world pretraining or post-training checkpoints. Moreover, the main response targets are changes in clipped token-level negative log-likelihood rather than downstream task accuracy, limiting the strength of claims about actual capability improvements.

  3. Unconvincing practical advantage over competitive alternatives. The headline improvements primarily use a static capability-only predictor as the reference. However, the paper's own cost comparison (Table 29) shows that direct eight-step target-specific screening achieves higher final utility at lower cost than the four-probe approach in three of the four evaluated settings. Thus, the practical benefit of acquiring reusable target-independent probes, relative to competitive cost-aware training-selection strategies, is not convincingly established.

  4. Disproportionate theoretical complexity and weak presentation. Much of the theoretical treatment restates standard level-set non-identifiability, linear-system identification, and conditional error propagation. The cross-family transfer guarantee depends on assumptions about unseen-family deviations that are not prospectively established, while exact coordinate sharing is empirically rejected for several actions. The extensive formalism and terminology obscure an otherwise straightforward feature-construction and regression framework, reducing the paper's clarity and substantive theoretical insight.

Questions for authors

What evidence supports the effectiveness of L-State on independently obtained, heterogeneous post-training checkpoints and on actual downstream task performance, particularly under a cost-matched comparison with target-specific screening?

Reviewer_qaaMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes L-State, which aims to predict changes in LLM capabilities after fine-tuning. The authors design four training tasks unrelated to the target tasks, perform LoRA training on each of them, and observe changes in five capabilities: mathematics, coding, science, reading, and language. This information is then used to construct L-State to predict training responses across different target tasks.

Critical strengths

The experiments evaluate multiple models and provide some research value in predicting capability changes and understanding training responses across different model families.

Critical weaknesses
  1. The selection of Probes and Capabilities in L-State lacks sufficient motivation and validation (why these dimensions can represent a model's training response, and why others cannot). Defining the four Probes as four coordinate axes does not necessarily mean they cover representative training directions. The experiments lack evaluations with different types of Probes and Capabilities.

  2. This paper aims to use four additional Probes to predict capability changes under different fine-tuning tasks and ultimately select the tasks most worth training on. Although the authors compare several methods, the results do not show that L-State consistently performs better (Table 1: Matched State achieves lower Regret than Full L-State in all three experimental settings). The paper compares multiple baselines in Table 1 but focuses on improvements over Capability-only. For the practical goal of selecting fine-tuning tasks, more direct methods such as Target-Early, Direct-Early, and Successive Halving are mainly discussed in the Appendix (and the results also do not show that L-State consistently outperforms these methods). Since the paper is very long, I would appreciate clarification from the authors if I have misunderstood anything.

Questions for authors
  1. Why were these four synthetic tasks selected as Probes, and are these five Capabilities sufficient to represent a model's training response? It would be helpful to replace the Probes with other synthetic tasks under the same training cost, or use different benchmarks to measure Capabilities, and examine whether the prediction performance remains stable.

  2. Can L-State also be applied to practical-scale fine-tuning? It would be helpful to include fine-tuning experiments closer to practical settings (using larger training datasets, more sufficient training budgets rather than repeatedly training on the same original 32 samples).

Reviewer_3dYXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

L-STATE represents a language-model checkpoint using five current capability scores and twenty score changes from four independent synthetic training probes. Direct ridge, an operator readout, and a full-state predictor estimate adaptation responses or choose among four training actions. Development holds out one family at a time among Qwen, Mistral, and OLMo; later GLM and Granite tests freeze predictions before target responses are opened. A transfer bound separates measurement, estimation, and family-coordinate errors. Further experiments examine coordinate sharing, longer training horizons, and repeated versus fresh data on GLM and Yi. Response error, direction accuracy, action regret, and acquisition cost are evaluated separately.

Critical strengths

The family-held-out tests avoid splitting related checkpoints at random. GLM and Granite predictions are frozen before target responses are opened; Granite also tests a readout-selection rule developed after GLM. Table 1 shows that the best readout differs across families.

The checks go beyond aggregate RMSE. Independent repeats test the science-action sign inversion (Table 16), and Tables 18–20 test coordinate sharing. The GLM/Yi study adds independent final outcomes and matched nonlinear capability-only predictors (Tables 39–40; Table 49). The cost results retain unfavorable cases.

Critical weaknesses

The probes often lose to cheaper target-specific screening. At the 32-step delivered-model endpoint, four probes take more time and give lower utility than both direct early-eight-step screening and successive halving in both GLM settings and Yi fresh continuation (Table 29; Figure 4). The 24-row supplementary CSV agrees. Reuse across queries at an unchanged checkpoint is proposed but not measured, nor is transfer to additional unseen target actions (Appendix K.6). The three-trajectory, short-LoRA, token-loss setting also limits conclusions beyond the favorable Yi repeated-example case.

Questions for authors

Can you include source-mean, zero-response, and constant-nonnegative-direction baselines, with uncertainty and action-wise results? Which gains remain beyond source-selected simple baselines?

Given Table 29, when is acquiring probes preferable to target-specific screening? A measured reuse case or transfer to additional unseen target actions would help establish the benefit.

Do the gains persist with larger independent evaluation sets, tokenizer-normalized scores or generated-answer accuracy, and independently varied initial checkpoints rather than short synthetic adapter trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.