- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles the high annotation cost in LLM agent training. Unlike standard QA pairs, agent data consists of multi-step trajectories with tool use and environmental interactions, making each sample much harder to label. The authors point out that standard uncertainty metrics like perplexity or BPC fail here for two reasons. First, decision token dilution: the uncertainty of critical actions gets buried under predictable formatting text. Second, exogenous environmental variance: tool outputs depend on external states rather than model confidence, meaning the same confidence can lead to totally different task outcomes. The main contribution is Pseudo-Influence. To value a sample, it estimates how much the model parameters would shift if that sample were added to the training set. This avoids convex objectives and heavy inverse Hessian calculations by using Monte Carlo sampling over a few checkpoints. The paper also includes a finite-step error bound (Theorem 3.2) covering the training path, checkpoint sampling, and gradient estimation errors. Experiments on Qwen3-8B and Qwen3-32B (using TERMINAL-BENCH 2.0 and ACEBENCH-AGENT across five active learning rounds) show that Pseudo-Influence performs well. It ranked first in all 16 evaluation points before the full dataset, beating the strongest non-PI baseline by an average of 3.1 pass@1 points. Combined with D2 diversity, it beats random selection by 4.8 to 6.7 points in the first round.
Critical strengths
-
Solid Problem Diagnosis with Empirical Grounding The motivation for this work is anchored in concrete measurements rather than pure intuition. In Section 2, the analysis of why uncertainty-based methods fail goes beyond mere speculation; it quantifies this gap using empirical data from 9 models on TERMINAL-BENCH, linking BPC to task scores. More importantly, the authors pinpoint two specific structural causes—decision token dilution and exogenous environmental variance—instead of dismissing uncertainty methods with vague generalizations. This grounded analysis makes the core premise highly persuasive, showing that the method is driven by genuine necessity rather than novelty for novelty’s sake.
-
Substantial Theoretical Contribution Bypassing Classical Constraints Classical influence functions (Koh & Liang, 2017)[1] rely on strong convexity assumptions and inverse Hessian computations, both of which are practically prohibitive in large language model scenarios. Derived from finite-step gradient descent, Pseudo-Influence bypasses these bottlenecks. Theorem 3.2 provides an error bound that only requires smoothness (dispensing with convexity) and explicitly decomposes the error into three interpretable sources, avoiding inverse Hessians altogether. This is not merely an engineering trick; it theoretically redefines the target objective (parameter shift rather than loss change at convergence), thereby providing formal guarantees in a non-convex setting. This contribution serves as a valuable reference for future work extending influence analysis to LLM training.
Critical weaknesses
-
A Fundamental Gap Between Theoretical Guarantees and Practical Implementation The error bound in Theorem 3.2 is derived for full-batch gradient descent, whereas the experiments actually employ the Adam optimizer with mini-batch training. The paper dismisses this discrepancy in a single sentence ("stateful optimizers such as Adam require an augmented-state analysis and are outside the present guarantee") without any supplementary analysis. This is not a minor edge case that can be brushed aside; the momentum and adaptive second-moment estimation in Adam systematically alter the parameter update trajectory, making it difficult to quantify the deviation between the GD-based parameter shift estimate and real training dynamics. Given that the theoretical guarantee is framed as a core selling point to distinguish this work from heuristic engineering, it fails under the optimizer used in practice. At the very least, the authors should provide empirical correlations between the estimated values and the true retraining effects under Adam settings (akin to Figure 4a) to partially bridge this theoretical gap, which is currently missing.
-
The Core Argument in Section 2 Relies on Synthetic Illustrations Rather than Measured Data Figure 2 (the scatter plots of six uncertainty metrics versus task results) is explicitly labeled in the caption as "a controlled illustration, not additional measured benchmark results," and the two subfigures in Figure 3 are similarly described as "illustrative renderings rather than logged model runs." This means the primary visual evidence used in Section 2 to support the core claim—that uncertainty methods fail in agentic scenarios—is artificially constructed rather than empirically observed. While the macro-correlation between BPC and task scores (Figure 1) uses real data, it addresses a coarse-grained, cross-model question and cannot replace sample-level resolution analysis. A rigorous motivation section should use real rollout data to demonstrate the overlap in uncertainty distributions between successful and failed trajectories. The current synthetic illustrations only show that such a scenario can theoretically exist, rather than proving that it is prevalent enough in real data to impair sample selection quality.
Questions for authors
Q1: Estimation Accuracy under the Adam Optimizer The error bound in Theorem 3.2 is established solely for full-batch gradient descent, yet the experiments employ the Adam optimizer. Please provide the Pearson/Spearman correlation between the Pseudo-Influence estimates and the true parameter shifts obtained via full retraining under the actual training setup (Adam + mini-batch). What optimizer was used for Figure 4(a)? If it was also Adam, please explicitly clarify this; if it was SGD, a corresponding experiment under Adam must be provided. This directly impacts whether the theoretical claims and the empirical results mutually support each other.
Q2: Robustness to Poor Pseudo-Label Quality in Early Rounds During the first round of active learning, the surrogate model is essentially in a cold-start state (with a warm-up pass@1 of only 2.2 and 5.2 on TERMINAL-BENCH), meaning the generated pseudo-labels should be of very low quality. What is the approximate task success rate of the trajectories generated by the surrogate model in the first round? Furthermore, has the ranking stability of Pseudo-Influence under low-quality pseudo-labels been analyzed? Is there empirical support for the statement in the Limitations regarding cases "when the quality of pseudo-labeling is low"? If you can effectively resolve my issue, I will consider increasing the points.