acceptodds
Under review as a conference paper at ICLR 2027

Agent Capability Shapes the Value of Post-Training Data

Abstract

Effective post-training of language-model agents depends on which experiences they learn from, as well as how much data they receive. We investigate whether an agent's current knowledge and capabilities can guide data selection for supervised fine-tuning (SFT) and reinforcement learning (RL). For SFT, we distinguish KNOWS and GAP traces by how accurately the agent predicts execution outcomes from known inputs. Selecting traces with larger environment prediction gaps yields a higher observed mean reward than random selection under matched training budgets. Cross-model comparisons show that this selection advantage depends on the learner. For RL, reward prediction gaps capture errors in the agent's estimates of its own rollout rewards. After controlling reward standard deviation, selecting high reward-GAP tasks improves final mean reward by relative to random selection, showing value beyond reward variability. We also identify practical proxies based on predicted rewards: traces and tasks selected by the proxies yield strong performances without additional verification. These findings connect the value of post-training data to the agent's predictive capabilities and offer a practical way to guide data selection. Code is anonymously available at https://anonymous.4open.science/r/agent_data-6BB4/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, post-train, llm agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SKMAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether an agent’s own capability can help decide which training data to use for post-training. The authors design methods for both SFT and RL to select better training data, which also improve the post-training results.

Critical strengths
  1. They design separate methods for SFT and RL, and both work and improve the training quality.

  2. The practical selector in the RL setting does not need verified rewards during selection, which is very useful in practice.

Critical weaknesses
  1. The paper provides a code repo, but it only contains a README and is almost empty. I cannot find the supporting evidence mentioned in the paper, such as the prompts in Appendix F and the other artifacts, to verify that the results are sound. This also makes me concerned about whether the authors treat this submission seriously.

  2. Both the SFT and RL experiments use a single fixed seed with only a few runs on a single dataset, which cannot demonstrate the significance of the authors’ methods.

Questions for authors
  1. Could you please provide an accessible code repo that contains the artifacts mentioned in the paper?
  2. For the SFT experiment, only Qwen gets improved but Gemma does not. Could you add at least one more model to demonstrate the significance of your method?
  3. Both the SFT and RL experiments are tested on only a single dataset. Could you add at least one more dataset for each setting to demonstrate the significance of your method?
Reviewer_9t21OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose data selection methods based on two prediction gaps: env-GAP and reward-GAP. For env-GAP, they label action-outcome predictions as GAP or KNOWS based on their semantic correctness and select SFT traces according to the fraction of GAP-labeled steps. They also investigate observation NLL as a lower-cost proxy. For reward-GAP, they measure the mean absolute discrepancy between predicted and actual rollout rewards to select RL tasks. Additionally, they introduce a practical RL selection heuristic based on mean predicted rewards, without requiring additional reward verification. The experiments show that selecting traces with larger env-GAP improves SFT performance, while selecting tasks with larger reward-GAP improves RL performance relative to the baselines.

Critical strengths

The paper proposes a potentially useful approach to agent data selection. It identifies two types of prediction gaps and uses them to guide the selection of training data for SFT and RL.

Critical weaknesses

The motivation and novelty of the proposed selection criteria need further clarification. The relationship between the prediction gaps and their proxies is insufficiently established, and the distinctions from existing loss-based and learnability-based selection methods require a more substantive discussion.

  1. The connection between the prediction gaps and their practical proxies is insufficiently established. NLL measures the likelihood of the recorded text, while env-GAP concerns semantic correctness. Appendix C.5 only shows a modest agreement between the two. The connection between reward-GAP and the estimated reward gap is also unclear. The paper should clearly explain when and why the two proxies are expected to capture the true GAP.
  2. The novelty relative to existing difficulty-based selection methods is unclear. Loss- and perplexity-based selection seems to be some established approaches. For example, a prior study (https://arxiv.org/pdf/2405.20541) investigates how to use NLL to guide data pruning, which is a highly related topic. Moreover, the proposed estimated reward gap for RL selection has the same functional form as LILO's learnability p^(1−p^)\hat p(1-\hat p), as the authors acknowledge. The authors should discuss the differences in more details.

The empirical evaluation does not convincingly establish the robustness or practical advantages of the proposed method, which makes it difficult to assess whether the gains are reliable and computationally worthwhile.

  1. The reported improvements are not sufficiently supported by uncertainty estimates. In Sec. 4.2, the mean improvement of GAP over Random is small relative to the reported variability. The RL results report averages over two runs without standard deviations or confidence intervals. These results make it difficult to assess the reliability of the claimed improvements.
  2. Comparisons with existing selection methods are limited. Data selection is a well-established domain, and stronger baselines would help justify the benefits of the proposed methods. For SFT, existing data selection methods include influence-based methods, such as LESS. For RL, the authors cite data selection approaches like DAPO or VCRL, but none of them are compared against. Adding these additional baselines would be informative.
  3. The computational cost of data selection is not revealed. Reporting selection runtime and baseline runtime would clarify the trade-off between performance gains and selection overhead.
Questions for authors
  1. How well do the practical selection scores empirically capture the corresponding prediction gaps? Beyond the results in Appendix C.5, could you report additional evidence of the agreement between GAP and its heuristic proxies, using metrics like ranking correlations?
  2. Could the authors provide theoretical justification for the relationship between GAP and its heuristic proxies, in both SFT and RL?
  3. Could the authors show robust results using more training seeds, and report the uncertainty?
  4. What is the computational cost of selection? And how does it compare with existing methods in literature?
Reviewer_EFswOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes selecting agent post-training data based on what the model currently fails to predict. For SFT, a student model predicts the outcome of each command before seeing it, a judge labels each step as GAP (wrong) or KNOWS (right), and traces with a high GAP fraction are selected. For RL, the policy estimates the score of its own rollouts, and tasks with high estimation error (reward-GAP) are selected for GRPO; a practical version ranks tasks by p̂(1−p̂) of predicted scores. Main results: on TB-dev, GAP-selected SFT gets 15.26 vs 13.53 for random; on ScienceWorld, the GAP pool gets 19.71 vs 14.79 for random after matching reward std.

Critical strengths

The idea of separating prediction errors caused by missing information from errors caused by poor understanding of execution is reasonable, and the paper excludes environment-discovery steps carefully.

Critical weaknesses
  1. The SFT results are within noise, and the subsets are not really comparable. GAP gets 15.26 ± 3.38 and Random gets 13.53 ± 3.88, so the 1.7-point difference is smaller than one standard deviation. Each subset was trained only once, so training randomness is not counted at all. TB-dev also has only 70 tasks. Training on the full dataset (13.17) even does worse than 500 random traces, which suggests the evaluation is very noisy. On top of that, the selected subsets differ in token count, trace length, data sources, and even which annotation snapshot they came from (Table 3).

  2. The RL gain can be explained by which task types were picked, not by reward-GAP. In Figure 4, the GAP pool does better on only 13 of 30 task types and worse on 16. The +4.92 overall gain comes mostly from a few “find” tasks (+69.4, +41.9), which have 75 test instances each and so count a lot in the average. The GAP pool also has more “find” tasks than the random pool (16 vs 11, Table 7). Each pool was only run twice, and each task is seen about twice during training (40 steps × 3 tasks over 60 tasks), so the result is fragile. reward-GAP itself is also biased: the model can only guess scores from 0 to 100, but real rewards can be negative, so any task with negative rewards automatically gets a high gap. This is why reward-GAP closely tracks reward std (r = 0.63). The practical selector (21.27) scores about the same as simply picking the tasks with the most varied rewards (20.98), and no variance is given. Finally, the two signals barely relate to each other (r = 0.09), so the “agent capability” story that ties the SFT and RL parts together is not really supported.

Questions for authors
  1. Can you compare GAP and Random pools with matched task-type distributions, or report results weighted equally across task types?
  2. Is the difference between the practical selector and top reward dispersion (21.27 vs 20.98) significant? If not, what does reward-GAP add over simple variance-based selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.