acceptodds
Under review as a conference paper at ICLR 2027

Active Agentic Learning

Abstract

Training capable large language model agents requires costly trajectory annotations that contain multi-step reasoning, tool calls, and environment interactions. Active learning can reduce this burden by prioritizing informative trajectories, but conventional uncertainty scores are poorly matched to agentic data: consequential decisions are diluted by predictable transcript tokens, while environment observations can vary independently of the model's pre-call confidence. We formulate Active Agentic Learning and propose Pseudo-Influence, a trajectory-valuation criterion that directly estimates training impact through pseudo-label perturbation rather than predictive uncertainty. Our efficient estimator provides theoretical guarantees without requiring a convex objective or inverse-Hessian computation. We evaluate Qwen3-8B and Qwen3-32B over five acquisition rounds in two complementary settings: command-line interaction on Terminal-Bench 2.0 and tool use on ACEBench-Agent. Across all 16 evaluation points before full-pool training, the best Pseudo-Influence variant ranks first and exceeds the strongest non-Pseudo-Influence baseline by 1.4–5.3 pass@1 points, with a mean gain of 3.1 points. In the first acquisition round, combining Pseudo-Influence with diversity improves over random selection by 4.8 and 5.5 points on Terminal-Bench 2.0, and by 6.7 points for both model scales on ACEBench-Agent. These results indicate that training-impact valuation and diversity are complementary signals for annotation-efficient agent learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Active Learning, Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SLThOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the high annotation cost in LLM agent training. Unlike standard QA pairs, agent data consists of multi-step trajectories with tool use and environmental interactions, making each sample much harder to label. The authors point out that standard uncertainty metrics like perplexity or BPC fail here for two reasons. First, decision token dilution: the uncertainty of critical actions gets buried under predictable formatting text. Second, exogenous environmental variance: tool outputs depend on external states rather than model confidence, meaning the same confidence can lead to totally different task outcomes. The main contribution is Pseudo-Influence. To value a sample, it estimates how much the model parameters would shift if that sample were added to the training set. This avoids convex objectives and heavy inverse Hessian calculations by using Monte Carlo sampling over a few checkpoints. The paper also includes a finite-step error bound (Theorem 3.2) covering the training path, checkpoint sampling, and gradient estimation errors. Experiments on Qwen3-8B and Qwen3-32B (using TERMINAL-BENCH 2.0 and ACEBENCH-AGENT across five active learning rounds) show that Pseudo-Influence performs well. It ranked first in all 16 evaluation points before the full dataset, beating the strongest non-PI baseline by an average of 3.1 pass@1 points. Combined with D2 diversity, it beats random selection by 4.8 to 6.7 points in the first round.

Critical strengths
  1. Solid Problem Diagnosis with Empirical Grounding The motivation for this work is anchored in concrete measurements rather than pure intuition. In Section 2, the analysis of why uncertainty-based methods fail goes beyond mere speculation; it quantifies this gap using empirical data from 9 models on TERMINAL-BENCH, linking BPC to task scores. More importantly, the authors pinpoint two specific structural causes—decision token dilution and exogenous environmental variance—instead of dismissing uncertainty methods with vague generalizations. This grounded analysis makes the core premise highly persuasive, showing that the method is driven by genuine necessity rather than novelty for novelty’s sake.

  2. Substantial Theoretical Contribution Bypassing Classical Constraints Classical influence functions (Koh & Liang, 2017)[1] rely on strong convexity assumptions and inverse Hessian computations, both of which are practically prohibitive in large language model scenarios. Derived from finite-step gradient descent, Pseudo-Influence bypasses these bottlenecks. Theorem 3.2 provides an error bound that only requires smoothness (dispensing with convexity) and explicitly decomposes the error into three interpretable sources, avoiding inverse Hessians altogether. This is not merely an engineering trick; it theoretically redefines the target objective (parameter shift rather than loss change at convergence), thereby providing formal guarantees in a non-convex setting. This contribution serves as a valuable reference for future work extending influence analysis to LLM training.

Critical weaknesses
  1. A Fundamental Gap Between Theoretical Guarantees and Practical Implementation The error bound in Theorem 3.2 is derived for full-batch gradient descent, whereas the experiments actually employ the Adam optimizer with mini-batch training. The paper dismisses this discrepancy in a single sentence ("stateful optimizers such as Adam require an augmented-state analysis and are outside the present guarantee") without any supplementary analysis. This is not a minor edge case that can be brushed aside; the momentum and adaptive second-moment estimation in Adam systematically alter the parameter update trajectory, making it difficult to quantify the deviation between the GD-based parameter shift estimate and real training dynamics. Given that the theoretical guarantee is framed as a core selling point to distinguish this work from heuristic engineering, it fails under the optimizer used in practice. At the very least, the authors should provide empirical correlations between the estimated values and the true retraining effects under Adam settings (akin to Figure 4a) to partially bridge this theoretical gap, which is currently missing.

  2. The Core Argument in Section 2 Relies on Synthetic Illustrations Rather than Measured Data Figure 2 (the scatter plots of six uncertainty metrics versus task results) is explicitly labeled in the caption as "a controlled illustration, not additional measured benchmark results," and the two subfigures in Figure 3 are similarly described as "illustrative renderings rather than logged model runs." This means the primary visual evidence used in Section 2 to support the core claim—that uncertainty methods fail in agentic scenarios—is artificially constructed rather than empirically observed. While the macro-correlation between BPC and task scores (Figure 1) uses real data, it addresses a coarse-grained, cross-model question and cannot replace sample-level resolution analysis. A rigorous motivation section should use real rollout data to demonstrate the overlap in uncertainty distributions between successful and failed trajectories. The current synthetic illustrations only show that such a scenario can theoretically exist, rather than proving that it is prevalent enough in real data to impair sample selection quality.

Questions for authors

Q1: Estimation Accuracy under the Adam Optimizer The error bound in Theorem 3.2 is established solely for full-batch gradient descent, yet the experiments employ the Adam optimizer. Please provide the Pearson/Spearman correlation between the Pseudo-Influence estimates and the true parameter shifts obtained via full retraining under the actual training setup (Adam + mini-batch). What optimizer was used for Figure 4(a)? If it was also Adam, please explicitly clarify this; if it was SGD, a corresponding experiment under Adam must be provided. This directly impacts whether the theoretical claims and the empirical results mutually support each other.

Q2: Robustness to Poor Pseudo-Label Quality in Early Rounds During the first round of active learning, the surrogate model is essentially in a cold-start state (with a warm-up pass@1 of only 2.2 and 5.2 on TERMINAL-BENCH), meaning the generated pseudo-labels should be of very low quality. What is the approximate task success rate of the trajectories generated by the surrogate model in the first round? Furthermore, has the ranking stability of Pseudo-Influence under low-quality pseudo-labels been analyzed? Is there empirical support for the statement in the Limitations regarding cases "when the quality of pseudo-labeling is low"? If you can effectively resolve my issue, I will consider increasing the points.

Reviewer_TMX5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to select tasks for SFT of LLM agents. The idea is to use active learning to choose, from a pool of tasks that do not come with demonstrations, the ones to send to an oracle, which provides the demonstrations used for SFT.

The authors propose a new score, Pseudo-Influence (PI), which works as follows:

  • Run the current model on each task and use the resulting trajectory as a pseudo-label.
  • Estimate, without retraining, the magnitude of the parameter change that training on this trajectory would cause. The change is estimated from learning-rate-weighted LM-head gradient differences (the task's gradient minus the labelled set's) at 5 checkpoints sampled from the training run on the labelled set.

The authors provide a finite-horizon error bound for this estimator. Intuitively, the score indicates how much the model would learn from a task. The authors argue that it avoids two confounding factors that affect uncertainty-based scores such as entropy or perplexity:

  • decision-token dilution: likelihood is averaged over the whole trajectory, so all tokens are given the same importance, which is not the case, since a few decisive actions matter most;
  • exogenous environment variation: outcomes depend on environment observations (tool outputs, search results) that vary independently of the model's confidence.

Results: The validation consists of five acquisition rounds with Qwen3-8B and Qwen3-32B, selecting from CLI-Gym tasks (evaluated on Terminal-Bench 2.0) and EnvScaler tasks (evaluated on ACEBench-Agent), against random, PPL, BPC and MCMC-Dropout selection. The better of the two PI variants (with or without D2 diversity) ranks first at every point before the final round, where all methods use the full pool. PI with diversity beats random selection by 4.8–6.7 pass@1 points in the first round.

Critical strengths
  • Timely problem: choosing which tasks to include in the set that will be used to gather trajectories to perform SFT on is a challenging problem. The paper evaluates on realistic scenarios (executable and verifier-checked benchmarks) across model scales and domains (N=2 for each).
  • the estimator is relatively cheap: it needs no per-candidate retraining or inverse hessian like other methods.
Critical weaknesses

The score's rationale and novelty are not established

  • Circularity and validity of the score: the pseudo-label is the model prediction itself. It depends only on the model prediction itself, hence the gradient will be 0 in expectation at the checkpoint. The gradient norm therefore measures only gradient noise at this point. PI cannot therefore be considered as an alternative to uncertainty.
  • Prior methods like BADGE, GradNd, TacIn should be tested. The paper states that "the first sample valuation criterion specifically designed for agentic scenarios" but this looks like an overstatement as nothing really pins PI to agentic training loops.
  • The paper diverges from its motivation: PI uses all valid tokens, predictable and informative ones alike. The evidence that we can disantangle the signal is thin. Fig. 1 relates each of 9 models' perplexity-style score to its benchmark score but the correlation is weak (Pearson p=0.113). That compares whole models, not training samples, which is what selection needs and what we would have expected. Fig. 2 is "a controlled illustration, not additional measured benchmark results", and Fig. 3 consists of "illustrative renderings rather than logged model runs".
  • The theory doesn't show real usefulness. Thm. 3.2 bounds a parameter-displacement proxy. It doesn't really tackle ranking quality or downstream gain. Additionally, it excludes Adam. The score is never compared with the actual gain from adding a task's oracle trajectory instead.

The experiments claim gains / savings that are not established

  • Noise. Results appear to be single runs (as there are no error bars, and the number of seeds is not stated). Given the scale of the models, one would expect multiple runs to be aggregated (although the reported values are inconsistent with a single run: on 89 Terminal-Bench tasks, a single-run pass@1 must be a multiple of 1.12%, and most reported values are not - e.g. 10.2, 17.4, 22.1, 20.5). This needs clarification, especially when some runs show a dramatic collapse.
  • Missing baselines. Besides the gradient-based methods above, the comparison lacks D2 diversity alone. We'd need that to tell how much of PI + diversity comes from PI.
  • Savings are never measured.
    • The oracle is never identified. This is a major gap in the paper. We'd need to know how much it costs to run the oracle compared with the score / training etc (see below).
    • Selection has its own cost: in rounds 1 to 4 the student runs on 100%, 80%, 60% and 40% of the pool, about 2.8 times the pool size, and each run costs 74.9 s to score.
    • If the oracle is an LLM agent plus a verifier, selection may cost more than labelling every task.
    • The setup also uses 48K freely available warm-up trajectories, which doesn't really align with the premise that trajectories are scarce.
Questions for authors

Clarification requests (the answers could change my assessment):

  • Oracle. Who or what produced the oracle trajectories for every task in the pools? Additional details like filtering, success rate, how trajectory cost compare.
  • How is pass@1 computed? How many runs or seeds were used for eval, selection, fine-tuning? How many ACEBench-Agent tasks were used?
  • Protocol. Please clarify:
    • Is the "surrogate" the evaluated Qwen3 model?
    • Does each round's fine-tune restart from the warm-started checkpoint, or continue from the previous round?
    • Please share training infra details: optimizer, learning rate, epochs, full fine-tuning vs LoRA, sampling temperature, agent harness, turn limits?

Suggestions (results that would change my assessment):

  • Please validate the score against the gain. Given that Oracle trajectories exist for all tasks, you should be able to compute PI with the true trajectory + report its rank agreement with the pseudo-label version. One could also imagine fine-tuning on the top versus bottom PI tenth of tasks and compare the gain (although I would focus first on using more seeds if relevant).
  • Add baselines: EGL, GraNd, BADGE, TracIn self-influence, D2 diversity alone. Also: comparing each PI variant on its own would bring a lot of value.
  • Please report variability and cost (>= 3 seeds, confidence intervals, plot pass@1 against total cost). Explain why round 4 beats the full pool in all four panels.
  • Show that PI handles the #2 problems: would be nice to see the the score broken down by token to see whether decision tokens dominate it, and, more broadly, measure how stable scores and rankings are across several rollouts of the same task in a systematic way.
Reviewer_ojFjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a practically relevant problem: selecting informative tasks for trajectory annotation when training LLM agents under a limited budget. It analyzes the limitations of existing uncertainty-based metrics in agentic settings and proposes Pseudo-Influence, a metric that estimates the parameter changes induced by adding a pseudo-labeled trajectory to the training set. The method uses gradients from saved training checkpoints to approximate this influence efficiently. Experiments on command-line interaction and tool use across two model scales report improvements over the evaluated selection baselines, suggesting that the proposed metric is useful for improving the annotation efficiency of agent training.

Critical strengths
  1. The paper addresses a practically relevant problem, as acquiring high-quality trajectories for LLM agents can be costly. Experiments cover both command-line interaction and tool use across two model scales, and the reported improvements across acquisition rounds provide encouraging evidence that the proposed criterion can improve annotation efficiency. The inclusion of both PI-only and PI+Diversity, alongside corresponding uncertainty-based baselines, also provides useful evidence for the contribution of the scoring criterion.

  2. The method has a clearly defined finite-horizon parameter-displacement target and a tractable approximation. Theorem 3.2 explicitly separates the errors arising from the training-path approximation, checkpoint sampling, and mini-batch gradient estimation. This provides theoretical support for the estimator under the stated assumptions and makes its limitations transparent, without requiring convexity or inverse-Hessian computation.

Critical weaknesses
  1. The experiments do not fully isolate the benefit of the proposed scoring mechanism relative to simpler gradient-based acquisition criteria. At a single checkpoint, the score reduces to a centered gradient norm, making ordinary gradient norms and single-checkpoint variants particularly relevant comparisons. The current checkpoint ablation measures correlation with retraining influence rather than downstream active-learning performance. Consequently, the incremental benefits of gradient centering and accumulation across checkpoints remain insufficiently established.
Questions for authors
  1. Clarification of the quantities plotted in Figure 4(a). Both the exact and estimated pseudo-influence in Eqs. (4) and (7) are defined as Euclidean norms and should therefore be non-negative. However, Figure 4(a) contains data points with negative values on both axes. Could the authors clarify whether the scores were centered, standardized, or otherwise transformed before plotting? If so, please specify the transformations applied to each axis and explain how the plotted quantities correspond to the definitions in the method.

  2. Practical benefit of using multiple checkpoints. Do the authors have results comparing the default multi-checkpoint PI with a single-checkpoint variant under the same annotation budget, with the rest of the selection and training pipeline held fixed? Figure 4(b) evaluates correlation with retraining-based influence, but it remains unclear whether the improved approximation translates into better downstream performance. Such a comparison would help assess the practical benefit of accumulating gradients across checkpoints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.