- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
From my understanding, I think the paper proposed a new method, "rollout-replay" to assess how reliably AI agents perform tasks using their interaction traces, without needing outcome labels during measurement. It analysis model behavior at key interaction points, such as before taking actions and after receiving observations, using two metrics. First, Level, it captures typical behavior. Second, Dispersion, which measures variation across repeated executions. Experiments across multiple benchmarks show that these event aligned metrics rank task reliability better than traditional full trajectory measurements. The results also generalizing to a held out benhmark. This study also finds that replaying traces through the original acting model generally produces better reliability rankings than using a different model.
Critical strengths
As a reviewer with fixing the LLM calibration and hallucination issues. I think the paper contributes in 2 major points.
First, the authors created a method only uses unlabeled execution traces for estimating the reliability without requiring external outcome feedback or external model. (As we know, data labeling and training model for data labeling is always expensive. Therefore, any method that uses unlabeled trace is already very meaningful to the field, espeically it does not require additional training)
Second, as showing in the paper, the authors was able to evaluate their method across multiple benchmarks and model families. This demonstrate consistent improvements when comparing with whole-trajetory baselines.
Critical weaknesses
Overall, I think the paper is well written. However, there are still two isuess worth mentioning. I also hope the authors can discuss them in the rebuttal.
Major Weakness:
First, this method needs more compute. This rollout-replay method needs multiple executions to estimate the task level reliability more accurately. Therefore, more executions runs in theory will bring better ranking accuracy. But, in the same time, it also increases the compute cost. This may limit the scalability when there is strong resource constraint.
Minor Weakness:
Second, although the authors provide some intuition for why event aligned measurements work better than whole trajectory measurements, the underlying mechanism is not clear. Experiment only shows that event-aligned uncertainty signals correlate with task reliability. But, it does not explain why such correlation exists. (I understand that fully explaining the mechanism is hard, but I hope the author can at least share some of their insight on how they develop this method especially on the thought process)
Questions for authors
I have two questions regarding the practice and understanding of the method.
Major question:
First, the paper shows that increasing the number of scored executions improves the reliability-ranking performance, but it also increases replay costs. Is it possible to reduce the computational cost of rollout-replay while maintaining reliable rankings?
Minor question (I do not expect perfect answer. I understand fully answer this question may require a new research project):
Second, your experiments show that measuring model behavior at action boundaries and after observations produces better reliability ranks than measuring behavior across whole trajectories. What further experiments could the authors conduct to better understand why these interaction points give more useful signals for predicting reliability?
3 comments
Thanks the authors for addressing my concerns. I really like this work and hope the author can open source this work soon so people in the community can benefit from it!!
Thank you for your support and encouragement! We’re glad our response addressed your concerns. We’ll continue improving the code and aim to share it with the community soon :)
Response to Reviewer vENc
We thank the reviewer for the positive assessment and thoughtful questions.
Computational efficiency. There are two complementary directions for reducing measurement cost. First, our budget ablation (Fig. 3b and Table 8) shows that reducing the scoring budget from eight executions to four retains much of the ranking performance: stable/mixed AUCs are 0.798/0.726, compared with 0.816/0.743 for eight executions. This provides evidence for using smaller sampling budgets under resource constraints.
Second, the observation procedure has the potential to be implemented online. We use offline replay to support controlled comparisons of readouts on identical recorded trajectories. When the acting model is also the reader, the fixed event-level statistics could instead be collected during natural rollout generation, given access to the required token distributions. This would avoid separate replay forward passes for measurement. Repeated natural executions would remain necessary for task-level aggregation, and the overhead of collecting and aggregating these statistics has not yet been measured.
Motivation for event-aligned measurement. Our starting point was the repeated action–observation structure of long-horizon agent interaction. We read token distributions before action commitments and after observations because these are points where the model selects actions or responds to new environmental evidence; whole-trajectory averaging may obscure information associated with these decisions. This consideration also resonates with GiGPO [1], which combines trajectory-level and step-level credit assignment to capture both overall trajectory quality and local action effectiveness.
Sec. 5.1 discusses this intuition, and App. A.3, Table 6 compares action-boundary, observation-update, and combined readouts across four intrinsic statistics. The gains across readouts support the value of measurement location, while the controls in Sec. 5.3 and App. A.7, Table 15 examine simpler explanations involving tool errors and execution length.
To investigate why these locations provide useful signals, a concrete next experiment would compare event positions with non-event positions matched for sampling count and context length within the same trajectories.
Reference
[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS, 2025.