acceptodds
Under review as a conference paper at ICLR 2027

STEP: Agent Confidence Estimation from Execution Trajectories

Abstract

Agent confidence estimation (ACE) estimates how likely a Large Language Model (LLM) agent is to complete its task and is crucial for reliable agent deployment (e.g., web agents placing orders or clinical agents assisting diagnosis, etc.). However, most existing ACE methods remain difficult to deploy in practice due to three limitations: (1) Insufficient evidence. They derive confidence from token probabilities or self-reported certainty, which reflect how confident the model is in generating its text rather than whether its actions achieved the intended effect in the environment; (2) Biased information. They impose excessive hand-crafted inductive biases, resulting in the loss of raw agent information; and (3) Late estimation. They conduct estimation only after execution ends, offering limited support for early prediction. To address these limitations, we propose STEP (Stepwise Trajectory Encoding for Prediction), a framework that encodes the task, observation, and action text of each step to ground confidence in agent-environment interactions, applies temporal modeling to these raw step representations instead of relying on hand-crafted features, and employs multi-prefix supervision to support early prediction. Extensive experiments on ScienceWorld, WebShop, and InterCode Bash demonstrate that under within-task evaluation, STEP improves AUROC over the strongest baseline for each prediction setting by 19.2% on end-of-trajectory prediction and 11.6% on the more challenging early prediction.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
agent confidence estimation, LLM agents, uncertainty estimation, trajectory encoding, early prediction, evaluation protocol

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_cGfDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for estimating LLM agent success from execution trajectories. It combines semantic step encoding, GRU-based temporal modeling, and multi-prefix supervision for early prediction. Experiments on three benchmarks show improved within-task AUROC over baselines.

Critical strengths
  1. The motivation is reasonable and clear.

  2. Experiments are comprehensive, the authors conducted evaluations on three different environments and provided ablation studies, experiments comparing different acting models, and controlled experiments examining the effects of task difficulty and trajectory length.

Critical weaknesses
  1. STEP is introduced to provide success probabilities that can guide intervention, but the reliability in real-world settings remains unclear. The datasets only include tasks with both successful and failed attempts, which may not reflect the actual outcome distribution. In Table 8, STEP also shows worse calibration metrics than HTC on WebShop, and the additional acting-model experiments retrain STEP for each model, leaving its generalization to new agents untested.

  2. Early prediction evaluation includes trajectories that have already finished before the specified prefix budget, these cases may affect the mean prefix AUROC.

Questions for authors
  1. Given that the datasets only include tasks with both successful and failed attempts, how reliable would STEP's predicted probabilities be under the original task distribution?

  2. What proportion of trajectories have already finished at each prefix budget? Could the authors report early prediction performance using only trajectories that are still running?

  3. Since STEP is retrained separately for each acting model, does its application to a new agent always require additional labeled trajectories?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cGfD

Thank you for the feedback on probability reliability, transfer, and ongoing evaluation.

Reply to W1 (outcome selection) & Q1 — Probabilities on unfiltered tasks

Mixed-outcome selection supports within-task comparisons. We evaluated unchanged predictors on 200 unfiltered tasks per environment: 1,000 ScienceWorld, 1,000 WebShop, and 600 InterCode Bash attempts, with success rates of 32.9%, 37.0%, and 31.3%. There was no target retraining, selection, or calibration; test tasks were excluded from their scoring model's training and validation. The table reports Full probability quality.

EnvironmentPredictorBrier ↓NLL ↓ECE ↓
ScienceWorldHTC.236.667.171
ScienceWorldSTEP.197.579.148
WebShopHTC.237.667.137
WebShopSTEP.261.724.167
InterCode BashHTC.243.678.199
InterCode BashSTEP.233.659.179

STEP has lower Brier, NLL, and ECE than HTC in ScienceWorld and InterCode Bash. These predictors are frozen; the separate task-prior experiment instead refits predictors. Revised Appendix H.1 (Table 23) specifies the protocol.

Reply to W1 (WebShop calibration) — Probability quality

For the main cohorts (submitted Table 8; revised Table 9, Appendix C.7), HTC has lower Brier/NLL/ECE in WebShop; STEP has lower Brier/NLL in ScienceWorld and InterCode Bash. Section 4.4 discusses these results.

Reply to W1 (new agents) & Q3 — Transfer without target labels

Not necessarily. We tested 12 cross-agent transfers in ScienceWorld with frozen source predictors and no target fitting. Test tasks were excluded from source training and validation. Each collection has 100 tasks and 500 attempts. Cells give STEP / HTC Full within-task AUROC; STEP averages five source-training seeds.

Source → TargetQwenGPT-4.1DeepSeekGemini
Qwen—.602 / .602.591 / .566.622 / NA
GPT-4.1.583 / .539—.690 / .549.610 / NA
DeepSeek.597 / .518.654 / .596—.604 / NA
Gemini.611 / NA.606 / NA.606 / NA—

NA means unavailable token probabilities. STEP matches or exceeds HTC in all six comparable transfers (revised Section 4.2.3; Appendix F.1, Table 21).

Reply to W2 & Q2 — Predictions on ongoing executions

Using the first min⁡(K,T)\min(K,T) records keeps the same attempts at every budget. Restricting to T>KT>K changes the eligible attempts and tasks as KK grows, so AUROC changes can reflect population changes as well as additional execution evidence.

Using the original models, we report ongoing-only InterCode Bash results. “Finished” is the share with T≤KT\leq K. AUROC uses only T>KT>K attempts from tasks retaining both outcomes (“Tasks”); all methods use the same eligible attempts.

InterCode Bash (195 attempts)

KFinishedTasksLength-basedMean token prob.HTCDirect PromptSTEP
10.0%59.428.470.441.595.554
213.3%50.540.605.525.523.574
324.1%40.571.548.556.469.487
431.8%35.571.521.557.515.590
540.5%30.600.517.631.556.660
645.1%29.526.489.563.445.687
750.3%26.526.506.468.439.738
851.3%26.458.542.487.516.694
955.4%22.519.583.712.477.661
10100.0%0—————

Over K=1K=1–99, STEP has the highest mean AUROC (.627), followed by HTC (.549). No attempts remain ongoing at K=10K=10, so AUROC is unavailable.

See revised Section 4.2.2 and Appendix B.2 (Table 5).

Thanks for the authors' response. The results address most of my concerns, so I decide to keep my score.

Reviewer_UzDuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on agent confidence estimation in reliable agent deployment. Specifically, this paper mentions three limitations on existing estimation strategies, including insufficient evidence, biased information, and late estimation. All these aspects show their drawbacks in performing reliable confidence estimation. To address these limitations, this paper proposes a framework, STEP, to encode tasks, observations, and action text of each step to ground confidence in agent-environment interactions.

Critical strengths
  • The problem studied in this paper is meaningful. Accurate confidence estimation plays an important role in practical agent applications.

  • The results reported in this paper are good.

Critical weaknesses
  • The insight behind the proposed STEP framework is not clear. Compared to previous works, which this paper claims their drawbacks, the main idea of the proposed framework is to just feed all available information (e.g., task text, observations, and action text) to an encoder and predict the confidence with a trained projection layer. From this perspective, the novelty and scientific insight in this paper are quite weak. Further, due to these concerns, why this framework works is also a question.

  • The details about training the predictor are not clear. After reading the paper, I do not know how the predictor is trained, what the training is based on, and why the trained predictor can provide reliable predictions regarding confidence. More precise and clear explanations are required in the paper.

  • The main "novel points" in this paper are encoding more information and training a predictor with a binary cross-entropy loss. I do not think these novelties are sufficient and convincing enough for the paper submitted to ICLR.

  • This paper does not well explain why the proposed information, including task text, observation descriptions, and action results, helps alleviate the aforementioned concerns (e.g., insufficiency, bias).

Questions for authors
  1. What are the details about training the predictor for confidence estimation?

  2. Could you please provide more explanations about the semantic step encoding in Section 3.1? Why such an encoder is used to encode the text information? Why the encoded information contributes to better performance?

  3. As mentioned in the limitations of existing works, insufficient and biased information are contained. However, why is this textual information, including task descriptions, observation descriptions, and action results, sufficient and non-biased?

  4. The dimension of utu_t in Eq. (2) is not 3d3d. You'd better check the formula in this paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UzDu (1/2)

Thank you for recognizing the importance of agent confidence estimation and for your questions about STEP's contribution, design rationale, and training procedure.

Reply to W1 & W3 — Are STEP's gains merely due to encoding more information?

Our contribution combines an evaluation finding with a model of the execution evidence that remains after task-level priors are controlled. In a separate 200-task ScienceWorld collection, a task-only predictor reaches .816 pooled AUROC, above HTC (.696), yet ties all attempts within each task (Table 2). This motivates within-task evaluation, complemented by separate length-matched controls.

STEP models the relationships among the task, observation, and action within each interaction and accumulates these records chronologically. To test whether its gains merely reflect more text, we trained an MLP on the same visible text using the same frozen MiniLM. Non-overlapping 254-token chunks are encoded as 384-dimensional vectors, concatenated in order, and right-padded with zeros before a 54-unit MLP. Dataset-wide maximum chunk counts are 19/43/8, explaining the parameter differences below. We reuse the main task folds, five seeds, multi-prefix objective, and common validation criterion. The table reports Full within-task AUROC.

EnvironmentPredictorTrainable parametersFull AUROC ↑
ScienceWorldMLP408,685.645
ScienceWorldSTEP22,401.720
WebShopMLP924,781.626
WebShopSTEP22,401.789
InterCode BashMLP172,141.620
InterCode BashSTEP22,401.680

With the text and encoder held fixed, STEP achieves higher AUROC using substantially fewer parameters. Its interaction representation relates actions to task requirements and observed conditions; its recurrent state incorporates later feedback about preceding decisions. Shuffling record order further lowers Full AUROC by 9.6 and 18.4 percentage points in ScienceWorld and WebShop. These controls support the predictive value of how execution evidence is organized and accumulated. The Introduction and Section 4.3 now summarize this evidence; revised Appendix E.5 (Table 17) gives the same-text comparison and protocol.

AuthorsOpenReview ↗

Response to Reviewer UzDu (2/2)

Reply to W2 & Q1 — Predictor training and probability quality

The environment's final binary success label supervises both the complete attempt and each training prefix. Equation (5) combines BCE on the full record with mean BCE across budgets K=1,…,10K=1,\ldots,10, using the first min⁡(K,T)\min(K,T) records at each budget. We freeze MiniLM and train the step projection (including layer normalization), GRU, and sigmoid readout together. One checkpoint per fold and seed supplies all prefix and Full predictions.

Each of the five task-grouped folds uses approximately 60%/20%/20% of tasks for training, validation, and testing. We use AdamW (learning rate 10−310^{-3}, weight decay 10−410^{-4}), batches of up to three complete tasks, and a 100-epoch cap with patience 20. Validation selects the training weight λ∈{.25,.50,.75}\lambda\in\{.25,.50,.75\} and the checkpoint using equal weights on Full and mean prefix BCE (§3.3 and Appendix C.1–C.2).

BCE supplies a probabilistic target: for a visible history HH, its expected loss is minimized by the conditional probability of eventual success, P(Y=1∣H)P(Y=1\mid H). We evaluate the learned scores on held-out tasks using both discrimination (Fig. 3) and probability quality (Brier, NLL, and ECE; Appendix C.7, Table 8).

Section 3.3 now lists the supervision, frozen/trainable modules, optimizer, and validation selection rule.

Reply to W4 & Q3 — Inductive bias and sufficiency of execution evidence

Bias. Here, “bias” refers to the inductive bias introduced by manually predefined confidence statistics. STEP learns prediction features from execution text, reducing reliance on those hand-selected statistics. The revised Introduction makes this specific meaning explicit.

Sufficiency. Task descriptions provide the requirements, observations provide environmental state and feedback from preceding actions, and action text records the agent's decisions. Their combination preserves evidence about whether decisions meet those requirements and what effects have been observed, which token-probability summaries do not explicitly capture. The field ablations support their predictive contribution: omitting actions lowers WebShop Full AUROC from .789 to .727, while omitting observations lowers InterCode Bash Full AUROC from .680 to .585.

Reply to Q2 — Semantic encoding and encoder choice

We independently encode the three fields with a shared frozen all-MiniLM-L6-v2, average the non-padding token states within each field, and normalize the resulting vectors. Concatenation preserves fixed positions for the task, observation, and action. The learned projection combines these roles into a task-conditioned step representation before temporal aggregation. The input ablations in W4 & Q3 quantify the contribution of the fields.

STEP does not depend on a particular text encoder. MiniLM is a lightweight choice with competitive empirical performance, and its embeddings can be cached. The submitted comparison (Appendix E.2) also tests BGE-small, E5-base, and Qwen3-Embedding-0.6B. No alternative consistently improves on MiniLM across environments and prediction budgets.

Reply to Q4 — Dimension in Eq. (2)

Each of E(g)E(g), E(ot)E(o_t), and E(at)E(a_t) has dd coordinates, so their concatenation utu_t has 3d3d coordinates. With MiniLM, d=384d=384 and utu_t has 1,152 coordinates. The subsequent projection ϕη\phi_\eta maps utu_t to the 16-dimensional vector xtx_t.

The revised text below Eq. (2) states these dimensions and identifies each encoder output as one pooled field vector.

Dear Reviewer UzDu,

Thank you very much for your constructive and insightful comments. We have made every effort to address your concerns in our rebuttal, particularly regarding the scientific insight and novelty of STEP. We have also incorporated the relevant clarifications and discussions into the revised version of our manuscript.

We would like to kindly ask whether you have any remaining concerns after considering our responses. Please feel free to contact us if you have any further questions or concerns. We would be more than happy to engage in further discussion and provide additional clarification.

Thank you again for your time and valuable feedback.

Reviewer_xXPnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STEP, a method for predicting task success by encoding the task description, observation, and action at each execution step into a semantic representation. A GRU aggregates these representations across successive steps, while multi-prefix supervision trains a single model to predict success at different execution stages using only final outcome labels. Experiments on three text-based environments show improved success–failure discrimination. The paper also examines task difficulty and trajectory length as potential confounds in evaluation.

Critical strengths

Semantic step representations allow the ACE model to use task goals, observations, and actions as contextual evidence for success prediction. Multi-prefix supervision enables incremental predictions using only final outcome labels. Within-task evaluation and length controls help distinguish execution-specific predictive ability from reliance on task difficulty or trajectory length.

Critical weaknesses

The paper evaluates success prediction performance but does not provide results on whether using these predictions to guide retries, recovery, or human intervention improves task success rates or reduces overall costs. An experiment measuring task success under a fixed budget, or cost at a matched success rate, would strengthen the practical contribution of the proposed method.

A STEP representation encodes the task description, pre-action observation and a selected action, but does not include the post effects after executing the action. The GRU must therefore infer action effects across compressed representations. Comparing the proposed design with explicit pre-action/post-action transition representation would clarify whether STEP adequately captures goal-relevant state changes. As such, alternate encoding schemes and comparative study would promote the contribution of the proposed semantic encoding method.

STEP compresses concatenated field embeddings from 1,152 dimensions to 16 dimensions before recurrent processing. This may discard details needed to interpret later observations, although the current results do not establish such a loss. Varying projection and hidden-state dimensions, and comparing against a predictor trained directly on the same execution text, would clarify the trade-off between compactness and predictive performance.

Pooling methods match or exceed STEP in some prefix evaluations, and multi-prefix supervision provides uneven gains across environments. These findings support the usefulness of execution content more clearly than the necessity of the specific architecture. The paper should distinguish the benefits of semantic inputs, recurrent aggregation, and the training objective more explicitly.

Questions for authors
  1. Why does STEP exclude the post-action observation from each step representation? Although this feedback enters through the next step, the model must infer action effects across compressed representations. Did you consider whether explicitly encoding pre- and post-action observations would better capture goal-relevant progress?

  2. The paper reports trade-offs across prediction stages, calibration metrics, and computational costs. Can you further characterize the execution patterns or failure types for which STEP outperforms or underperforms the baselines? Paired examples where STEP succeeds but HTC or Direct Prompt fails, and vice versa, would help clarify what its semantic and temporal representations capture.

  3. Can STEP’s predictions improve task success or execution efficiency when used to guide interventions? For example, would STEP-guided retries or recovery outperform fixed-budget strategies under the same total execution budget, after accounting for successful attempts that are incorrectly interrupted?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xXPn (1/2)

Thank you for the detailed feedback on practical decision-making, action feedback, and predictor capacity.

Reply to W1 & Q3 — Retries at a fixed attempt budget

We replayed held-out completed executions under a fixed number of attempts. Within each test fold, recorded attempts are randomly ordered without replacement for 5,000 shared replays. Each task starts with one attempt. Tasks below the attempt cap (five in ScienceWorld/WebShop, three in InterCode Bash) are eligible; the policy retries the one with the lowest current Full confidence and returns its latest attempt. Decisions use scores only; success labels and terminal rewards are excluded from the selection rule. Uniform retry distributes retries evenly. The table gives final task success (%) at twice the task count (200, 200, and 118 attempts). The total budget is apportioned by fold size.

PolicyScienceWorldWebShopInterCode Bash
Uniform retry51.150.251.7
Length-based53.754.954.5
Mean token probability49.351.256.9
Direct Prompt55.956.748.1
HTC57.360.854.7
STEP62.659.058.1

STEP improves over uniform retry in all three environments. The final success rates already account for harmful retries. Under STEP, 8.0%, 6.4%, and 15.4% of all tasks, respectively, went from an initial success to a final failure.

The replay protocol and results are in revised Appendix I.2 (Table 25).

Reply to W2 & Q1 — Encoding action feedback

STEP scores immediately after ata_t is generated, before its feedback ot+1o_{t+1} arrives. That feedback enters as the observation at the next step. Applying the same boundary at Full also excludes terminal feedback that can directly reveal the outcome (revised Appendix D, Table 13).

We also tested explicit state-transition encoding by adding a task-conditioned branch for (ot−1,at−1,ot)(o_{t-1},a_{t-1},o_t) alongside the original step representation, keeping the scoring point and supervision fixed. The branch starts at step 2 and adds no residual at step 1. The variant has 50,337 trainable parameters versus 22,401 for STEP. The matched baseline is the paired rerun for this comparison.

Variant (mean prefix / Full within-task AUROC)ScienceWorldWebShopInterCode Bash
Matched STEP.610 / .720.636 / .789.619 / .680
Explicit observed transition.615 / .704.647 / .806.624 / .686

The transition branch yields small gains in mean prefix AUROC and mixed results at Full. Revised Appendix E.7 (Table 20) reports this comparison; the method section specifies the scoring time.

AuthorsOpenReview ↗

Response to Reviewer xXPn (2/2)

Reply to W3 — Capacity and prediction from the same text

Width. We varied the projection and GRU hidden widths one at a time under multi-prefix supervision. The table gives Full within-task AUROC.

Projection / hidden widthTrainable parametersScienceWorldWebShopInterCode Bash
8 / 1612,793.687.784.704
16 / 821,385.720.751.684
16 / 16 (STEP)22,401.720.789.680
16 / 3225,585.702.782.675
16 / 6436,561.713.771.673
32 / 1641,617.716.781.651
64 / 1680,049.724.779.631

We retain the submitted 16/16 setting: widening increases the parameter count without a consistent AUROC gain across environments.

Uncompressed capacity control. We replaced the linear projection with an identity map and used a GRU with 1,152-dimensional input and hidden state; layer normalization, GELU, and dropout were retained. Both input and hidden widths change. Both use multi-prefix supervision. We compare them on all five WebShop test folds with seed 42.

PredictorGRU input / hidden widthTrainable parametersPrefix mean AUROCFull AUROC
STEP16 / 1622,401.6268.7958
Without projection1,152 / 1,1527,972,993.6436.7558

The wider predictor gains .017 in mean prefix AUROC at 356× the parameter count, while STEP has the higher Full AUROC. See revised Appendix E.6 (Tables 18–19).

A classifier on the same ordered execution text reaches Full AUROC .645/.626/.620, versus STEP's .720/.789/.680 (revised Appendix E.5, Table 17). It uses the same frozen MiniLM, task folds, five seeds, and multi-prefix protocol.

Reply to W4 — Text, recurrence, and supervision

  • Text. The largest Full AUROC drops in the field ablations come from omitting task text in ScienceWorld (.720 → .682), actions in WebShop (.789 → .727), and observations in InterCode Bash (.680 → .585).
  • Recurrence. With the inputs and supervision fixed, the recurrent model has the highest Full AUROC in each environment. On prefixes, mean pooling leads in ScienceWorld (.643 vs. .611), and attention pooling is close to STEP in WebShop (.635 vs. .636).
  • Multi-prefix supervision. With the architecture fixed, mean prefix AUROC rises from .609/.616/.594 to .611/.636/.619 across the three environments. In ScienceWorld, the Full AUROC is higher with Full-only supervision (.738 vs. .720).

Section 4.3 now distinguishes these contributions explicitly.

Reply to Q2 — Behavior patterns and paired cases

We tagged 251 executions from 55 tasks: 20 ScienceWorld, 20 WebShop, and 15 InterCode Bash tasks selected using fixed task hashes. Tags use the task and full visible trace, with outcome labels and predictor scores hidden. Groups may overlap. The table gives task-averaged ranking accuracy for same-task success–failure pairs in which either attempt shows the behavior.

BehaviorTasksHTCDirect PromptSTEP
Task constraint violation23.625.507.822
Misread feedback10.692.533.618
Precondition or order error12.701.826.783
Unproductive loops27.689.606.748
Recovery after an error17.701.489.691

STEP's clearest advantage is on task constraints. HTC leads on feedback errors and Direct Prompt leads on precondition or order errors.

  • STEP ranks correctly, HTC reverses the pair. For a WebShop request for a hair-styling mirror under USD 30, the successful attempt buys a pedestal mirror for USD 11.45 and the failed attempt buys a hair-styling kit for USD 15.99. STEP scores success/failure .570/.477, while HTC scores .500/.518.
  • HTC ranks correctly, STEP reverses the pair. Both ScienceWorld attempts read 10 °C. The task requires the purple box below 200 °C, and the failed attempt chooses yellow. HTC scores success/failure .594/.556, while STEP scores .564/.609.

Revised Appendix E.4.2 includes the behavior definitions, group results (Table 16), and paired cases.

The authors' response along with additional experimental results directly addressed all of my concerns and questions. I raise my score from 2 to 3.

Reviewer_781BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STEP, a trajectory-based method for estimating whether an LLM agent will eventually complete a task. STEP encodes the task description, observations, and actions at each interaction step using a frozen text encoder, then applies a recurrent encoder to model the execution history. It uses multi-prefix supervision so that the same predictor can estimate success from partial trajectories during execution.

The paper evaluates STEP on ScienceWorld, WebShop, and InterCode Bash. It compares STEP with length-based, token-probability-based, HTC, and prompted-LLM baselines using within-task AUROC, together with additional analyses for length matching, calibration, input ablations, and different acting models. The reported results show that STEP generally achieves higher AUROC, especially for full-trajectory prediction.

Critical strengths

1.The paper identifies important evaluation shortcuts in agent confidence estimation. The analysis shows that cross-task AUROC can be inflated by task difficulty, while trajectory length can also provide a strong shortcut. The within-task and length-matched evaluations are useful steps toward a more meaningful assessment.

2.The experimental analysis is relatively comprehensive. In addition to the main comparisons, the paper includes multiple acting models, input ablations, pooling variants, text encoder variants, calibration metrics, and computational overhead analysis. These experiments make the empirical findings easier to inspect.

Critical weaknesses
  1. The practical generalization of STEP is not sufficiently demonstrated.
STEP is trained separately for each environment and relies on outcome-labeled trajectories from the target setting. The main evaluation also uses a fixed agent collection and task-grouped splits within the same benchmark. It is therefore unclear how well STEP transfers to a new agent, a changed prompt, a new environment, or a different task distribution.
This limits the paper’s claim that STEP is broadly useful for deployable agent confidence estimation. The paper would be stronger with cross-agent or cross-environment transfer experiments, or with a clearer discussion that STEP is an environment-specific learned predictor.
  2. The paper evaluates ranking more convincingly than confidence estimation.
AUROC measures whether successful attempts are ranked above failed attempts, but it does not directly establish that the predicted values are reliable probabilities for deployment. The reported calibration results are mixed; for example, STEP has worse ECE than HTC on WebShop.
The paper should place more emphasis on Brier score, NLL, ECE, selective prediction, or risk-coverage curves. It would also be useful to evaluate whether using STEP to trigger human intervention actually reduces failure risk at a fixed intervention budget.
  3. The comparison against baselines does not fully establish the source of the improvement.
STEP uses raw task, observation, and action text, while the main baselines rely largely on token probabilities, length features, or HTC statistics. Although the paper includes several ablations, the comparison does not fully isolate the contribution of semantic text information, temporal recurrence, and the multi-prefix objective.
In particular, the mean-pooling ablation outperforms the recurrent model on prefix prediction in ScienceWorld, and attention pooling is comparable in WebShop. This suggests that recurrence may not be the main source of the reported gains. The paper should provide a clearer decomposition of the gains from:
    • text content;
    • temporal order;
    • recurrent aggregation;
    • multi-prefix supervision.
  4. The early-prediction setting may not reflect the information available after an action.
The input excludes the response to the last included action because that response may reveal the outcome directly. This avoids a form of leakage, but it also means that the predictor estimates success before observing the environment’s feedback to the current action.
The paper should clarify the intended deployment timing. Is the prediction made before executing the next action, immediately after issuing an action but before receiving environment feedback, or after receiving the next observation? These settings have different practical meanings.
Questions for authors
  1. How well does STEP transfer beyond the fixed data collection setting? Have the authors evaluated STEP on a new acting model, a changed agent prompt, or a new task distribution without retraining?
  2. What is the main source of the improvement over the baselines? Can the authors separately evaluate the contributions of:
    • task, observation, and action text;
    • temporal order;
    • recurrent aggregation;
    • multi-prefix supervision?
  3. How should STEP scores be used in an actual intervention system? For example, at a fixed coverage or human-review budget, what failure reduction does STEP provide compared with HTC or simpler baselines?
  4. What exactly is the prediction time during execution? Does STEP produce its score before the next action, after the action is generated but before environment feedback, or after receiving the next observation?
  5. How robust are the results to the choice of outcome prevalence and task composition? The reported cohorts contain selected mixed-outcome tasks and fixed numbers of attempts. Since calibration metrics depend on prevalence, it would be useful to report results under more natural task distributions or explain how the reported probabilities should be interpreted outside the constructed evaluation cohorts.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 781B (1/2)

Thank you for recognizing the evaluation design and experimental breadth.

Reply to W1 & Q1 — Transfer without target fitting

We tested 12 cross-actor transfers in ScienceWorld with frozen source predictors and encoders, without target fitting. Each target has 100 tasks and 500 attempts; test tasks are excluded from source training and validation. Cells give STEP / HTC Full within-task AUROC, averaging five source-training seeds for STEP. NA means token probabilities are unavailable.

Source → TargetQwenGPT-4.1DeepSeekGemini
Qwen—.602 / .602.591 / .566.622 / NA
GPT-4.1.583 / .539—.690 / .549.610 / NA
DeepSeek.597 / .518.654 / .596—.604 / NA
Gemini.611 / NA.606 / NA.606 / NA—

STEP matches or exceeds HTC in all six comparable transfers under the same prompt template (revised Section 4.2.3; Appendix F.1, Table 21).

Reply to W2 — Probability quality

At Full, submitted Table 8 (revised Table 9, Appendix C.7) reports lower Brier/NLL for STEP in ScienceWorld and InterCode Bash, and for HTC in WebShop. ECE compares predicted and observed success within score bins: STEP has lower ECE in ScienceWorld; HTC in WebShop and InterCode Bash. On WebShop, STEP's raw scores place slightly more probability mass near 0 or 1 than HTC. Section 4.4 summarizes probability quality; Q3 evaluates review allocation.

Reply to W3 & Q2 — Decomposing the improvement

The input, order, and aggregation controls use multi-prefix supervision. All values are within-task AUROC.

1. Text content

The task ablation also omits instruction copies in observations.

Input ablation (Full)ScienceWorldWebShopInterCode Bash
STEP.7198.7887.6797
Without actions.6917.7269.6513
Without task text.6817.7665.6983
Without observations.7160.7702.5847

The largest drops come from omitting task text in ScienceWorld, actions in WebShop, and observations in InterCode Bash.

2. Temporal order

For training and evaluation, we shuffle intact interaction records within each attempt before constructing prefixes, then apply the same multi-prefix objective. This table reports Full AUROC.

Record order (Full)ScienceWorldWebShopInterCode Bash
Original order.7198.7887.6797
Shuffled order.6243.6052.6415

Shuffling lowers Full AUROC by 9.6 and 18.4 percentage points in ScienceWorld and WebShop, supporting chronological order.

3. Recurrent aggregation

Cells give prefix mean / Full. Prefix mean averages K=1–10 using the first min(K,T) records.

AggregationScienceWorldWebShopInterCode Bash
Recurrent (STEP).611 / .720.636 / .789.619 / .680
Mean pooling.643 / .691.623 / .687.621 / .672
Attention pooling.618 / .672.635 / .708.594 / .657

With inputs and supervision fixed, recurrence has the highest Full AUROC in all three environments; prefix results vary.

4. Multi-prefix supervision

Both variants use recurrence; cells give prefix mean / Full.

SupervisionScienceWorldWebShopInterCode Bash
Multi-prefix supervision.611 / .720.636 / .789.619 / .680
Full-only BCE.609 / .738.616 / .785.594 / .669

Multi-prefix supervision raises mean prefix AUROC in all three environments, most in WebShop and InterCode Bash.

Revised Section 4.3 summarizes the controls: fields in C.9, order/pooling in E, and supervision in A.

AuthorsOpenReview ↗

Response to Reviewer 781B (2/2)

Reply to W4 & Q4 — Exact prediction time

Observe oto_t → Generate ata_t → Score ptp_t → Receive ot+1o_{t+1}.

Scoring occurs after generating the action, before incorporating its environmental response, using (g,o1,a1,...,ot,at)(g,o_1,a_1,...,o_t,a_t). Here oto_t includes visible feedback from the previous action. Full uses the same boundary at the last record.

The revised method section and Figure 2 now mark this scoring point.

Reply to Q3 — Review allocation

In an offline evaluation at 65% coverage (about 35% review budget), we set aside the lowest-confidence completed executions for review. We pool attempts across tasks and report the failure rate (%) among retained executions, averaging over random tie-breaking. Lower is better.

MethodScienceWorldWebShopInterCode Bash
Random48.8049.8048.21
Length-based46.1546.4644.88
Mean token probability49.5445.5444.09
Direct Prompt42.1542.7751.18
HTC43.0843.3844.09
STEP36.9242.0939.53

Compared with the best baseline in each environment, STEP lowers the retained failure rate by 5.2 percentage points in ScienceWorld, 0.7 in WebShop, and 4.6 in InterCode Bash. Revised Appendix I.1 includes all three risk–coverage curves (Figure 9) and the numerical comparison at this coverage (Table 24).

Reply to Q5 — Outcome prevalence and task composition

We applied the original predictors without target fitting to 200 unscreened tasks per environment. ScienceWorld/WebShop use five attempts per task (1,000 total); InterCode uses the first three (600). Success rates are 32.9%, 37.0%, and 31.3%. Each task is scored only by source models that excluded it from training and validation.

Each cell is HTC / STEP. Probability metrics use all attempts. Within-task AUROC uses the 81/50/50 mixed-outcome tasks, respectively.

EnvironmentWithin-task AUROC ↑Brier ↓NLL ↓ECE ↓
ScienceWorld.566 / .732.236 / .197.667 / .579.171 / .148
WebShop.693 / .765.237 / .261.667 / .724.137 / .167
InterCode Bash.560 / .642.243 / .233.678 / .659.199 / .179

The within-task advantage persists on unscreened collections. STEP also retains lower Brier, NLL, and ECE than HTC in ScienceWorld and InterCode Bash. Here the submitted predictors are frozen, whereas the separate task-prior experiment refits predictors within the 200-task collection. Revised Appendix H.1 (Table 23) specifies this distinction and the unfiltered evaluation protocol.

Thank you for the additional experiments. The ScienceWorld results address my main concern about transfer: STEP matched or exceeded HTC on all six comparable transfers without retraining on the target agent. I will raise my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Manuscript Revisions

We thank all reviewers for their careful feedback and for recognizing the importance of agent confidence estimation and the value of controlled evaluation. The revised manuscript includes the additional evidence and clarifications below.

Additional evidence and clarifications

  • Contribution and execution evidence (UzDu, 781B, xXPn). A task-only predictor reaches .816 pooled AUROC versus .696 for HTC in a separate ScienceWorld collection, while tying attempts within each task. This motivates within-task evaluation with separate length controls. The same-text MLP and record-order controls further test the contribution of organizing task-conditioned interactions and accumulating evidence over execution history.
  • Transfer without target fitting (781B, cGfD). We evaluated 12 cross-actor transfers in ScienceWorld. Frozen STEP predictors match or exceed HTC in all six comparable directions, without target-agent fitting.
  • Outcome selection and probability quality (781B, cGfD, UzDu). We evaluated the unchanged predictors on 200 unfiltered tasks per environment, excluding each scoring task from its model's training and validation. We also clarify how the submitted Brier, NLL, and ECE results complement discrimination metrics.
  • Practical use (781B, xXPn). We evaluated review allocation at a fixed coverage and confidence-guided retries at an identical attempt budget. The retry replay uses current scores only, shares sampled attempts across policies, and accounts for initially successful tasks whose final attempt fails.
  • Design and training details (xXPn, UzDu). We clarify the prediction time, final-outcome supervision, trainable modules, task-separated validation, and representation dimensions. Additional controls test observed-transition encoding, projection/state widths, and uncompressed predictors.
  • Ongoing executions (cGfD). We report InterCode Bash results using only attempts with T>KT>K, alongside the finished share and number of evaluable tasks at each budget, separately from the fixed-cohort evaluation.
  • Behavior patterns (xXPn). Trace-based behavior groups and paired examples identify where STEP or competing predictors rank the successful execution correctly.

Manuscript revisions

  • Introduction: clarify the dependence on predefined confidence statistics and summarize the same-text control.
  • Method: Figure 2 marks the scoring timeline; the text below Eq. (2) states the dimensions; Section 3.3 gives the training procedure.
  • Experiments: Section 4.2.3 summarizes cross-actor transfer, Section 4.3 separates the design controls, and Section 4.4 discusses the submitted probability-quality results alongside discrimination.
  • Appendix: B.2 reports ongoing-only evaluation; C.9 gives field ablations; E.3–E.7 report order, behavior, same-text, capacity, and observed-transition controls; F.1 and H.1 report transfer and unfiltered cohorts; I gives risk–coverage and fixed-attempt retry results.

Additions and replacement text are marked in red in the revised PDF. The individual replies provide the corresponding results for each weakness and question.

Area_Chair_u4WXOpenReview ↗

Please engage in the discussion

Reviewers: if you have not responded to the authors' reply, please read it and engage in the discussion. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.