acceptodds
Under review as a conference paper at ICLR 2027

Probing Task-Level Agent Reliability Through Rollout and Replay

Abstract

An agent's execution record can be available before an external evaluator returns a task outcome. We ask whether the acting model's own unlabeled reasoning-and-action traces reveal how reliably it handles a fixed task. Our rollout–replay protocol collects repeated natural executions, replays their recorded histories through the original model, and reads intrinsic statistics at action and post-observation events. Discovery batch A fixes the measurement rules before evaluation on new execution seeds in batch B. Across benchmarks, event-aligned readouts rank task reliability more accurately than matched whole-trajectory statistics. Level separates groups whose sampled executions consistently succeed or fail, while Dispersion distinguishes higher- from lower-success groups with mixed outcomes. Scores from A also rank outcomes from independent B executions; comparator gains persist before explicit outcome disclosure and on held-out MCPMark. Cross-model replay shows that reader capability alone does not replace compatibility with the acting model. These results support task-level reliability measurement from unlabeled interaction traces. They do not establish calibrated success probabilities or a benefit from using the scores to change agent behavior.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
language agents, self-monitoring, trajectory-level reliability, post-training, continual learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_vENcOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

From my understanding, I think the paper proposed a new method, "rollout-replay" to assess how reliably AI agents perform tasks using their interaction traces, without needing outcome labels during measurement. It analysis model behavior at key interaction points, such as before taking actions and after receiving observations, using two metrics. First, Level, it captures typical behavior. Second, Dispersion, which measures variation across repeated executions. Experiments across multiple benchmarks show that these event aligned metrics rank task reliability better than traditional full trajectory measurements. The results also generalizing to a held out benhmark. This study also finds that replaying traces through the original acting model generally produces better reliability rankings than using a different model.

Critical strengths

As a reviewer with fixing the LLM calibration and hallucination issues. I think the paper contributes in 2 major points.

First, the authors created a method only uses unlabeled execution traces for estimating the reliability without requiring external outcome feedback or external model. (As we know, data labeling and training model for data labeling is always expensive. Therefore, any method that uses unlabeled trace is already very meaningful to the field, espeically it does not require additional training)

Second, as showing in the paper, the authors was able to evaluate their method across multiple benchmarks and model families. This demonstrate consistent improvements when comparing with whole-trajetory baselines.

Critical weaknesses

Overall, I think the paper is well written. However, there are still two isuess worth mentioning. I also hope the authors can discuss them in the rebuttal.

Major Weakness:

First, this method needs more compute. This rollout-replay method needs multiple executions to estimate the task level reliability more accurately. Therefore, more executions runs in theory will bring better ranking accuracy. But, in the same time, it also increases the compute cost. This may limit the scalability when there is strong resource constraint.

Minor Weakness:

Second, although the authors provide some intuition for why event aligned measurements work better than whole trajectory measurements, the underlying mechanism is not clear. Experiment only shows that event-aligned uncertainty signals correlate with task reliability. But, it does not explain why such correlation exists. (I understand that fully explaining the mechanism is hard, but I hope the author can at least share some of their insight on how they develop this method especially on the thought process)

Questions for authors

I have two questions regarding the practice and understanding of the method.

Major question:

First, the paper shows that increasing the number of scored executions improves the reliability-ranking performance, but it also increases replay costs. Is it possible to reduce the computational cost of rollout-replay while maintaining reliable rankings?

Minor question (I do not expect perfect answer. I understand fully answer this question may require a new research project):

Second, your experiments show that measuring model behavior at action boundaries and after observations produces better reliability ranks than measuring behavior across whole trajectories. What further experiments could the authors conduct to better understand why these interaction points give more useful signals for predicting reliability?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vENc

We thank the reviewer for the positive assessment and thoughtful questions.

Computational efficiency. There are two complementary directions for reducing measurement cost. First, our budget ablation (Fig. 3b and Table 8) shows that reducing the scoring budget from eight executions to four retains much of the ranking performance: stable/mixed AUCs are 0.798/0.726, compared with 0.816/0.743 for eight executions. This provides evidence for using smaller sampling budgets under resource constraints.

Second, the observation procedure has the potential to be implemented online. We use offline replay to support controlled comparisons of readouts on identical recorded trajectories. When the acting model is also the reader, the fixed event-level statistics could instead be collected during natural rollout generation, given access to the required token distributions. This would avoid separate replay forward passes for measurement. Repeated natural executions would remain necessary for task-level aggregation, and the overhead of collecting and aggregating these statistics has not yet been measured.

Motivation for event-aligned measurement. Our starting point was the repeated action–observation structure of long-horizon agent interaction. We read token distributions before action commitments and after observations because these are points where the model selects actions or responds to new environmental evidence; whole-trajectory averaging may obscure information associated with these decisions. This consideration also resonates with GiGPO [1], which combines trajectory-level and step-level credit assignment to capture both overall trajectory quality and local action effectiveness.

Sec. 5.1 discusses this intuition, and App. A.3, Table 6 compares action-boundary, observation-update, and combined readouts across four intrinsic statistics. The gains across readouts support the value of measurement location, while the controls in Sec. 5.3 and App. A.7, Table 15 examine simpler explanations involving tool errors and execution length.

To investigate why these locations provide useful signals, a concrete next experiment would compare event positions with non-event positions matched for sampling count and context length within the same trajectories.

Reference

[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS, 2025.

Thanks the authors for addressing my concerns. I really like this work and hope the author can open source this work soon so people in the community can benefit from it!!

Thank you for your support and encouragement! We’re glad our response addressed your concerns. We’ll continue improving the code and aim to share it with the community soon :)

Reviewer_2JHYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method to estimate an AI agent's reliability in executing tasks (task-level reliability) using the agent's execution traces, without requiring success/failure outcome labels from these traces at scoring time. The method involves running the agent multiple times on the same task and recording those executions, replaying those recorded executions through the original model and storing the token probabilities and other statistics at action and observation boundaries, and then studying whether that information can be used to determine downstream task success. The main finding is that event-aligned statistics computed from these trajectories can outperform whole-trajectory statistics when ranking groups of tasks by reliability.

Critical strengths

The paper does a good job in the experimental setup of ensuring that measurement choices in batch A are frozen before evaluating new execution seeds in batch B. This "Freeze before evaluation" setup is well-designed for the purpose of this analysis paper and reduces concern about potential overfitting to a particular type of executions or trajectories.

The cross-model replay analysis in Section 5.4 is interesting and demonstrates that self-replay consistently achieves a higher AUC compared to cross-model replay, even if the replay model is larger than the trajectory model. I think this finding could be useful for researchers thinking about relationships between replay models and trajectory models and how to evaluate trajectories.

Critical weaknesses

W1: The paper could be strengthened by having more concrete discussion of how the findings translate into a usable method that might apply to arbitrary tasks, especially longer-horizon tasks. It is not currently clear how L or D, for example, might be useful for different classes of tasks and this lack of discussion limits how useful the paper is. Providing qualitative examples could help the authors address this weakness too; there are currently no concrete examples of trajectories in the paper's discussion.

W2: The paper's claim about measuring token-distribution statistics around interaction boundaries being more effective than averaging over an entire trajectory is compared to a fairly narrow baseline. For example, it could be possible to measure the number of failed tool calls and use that as a proxy to estimate downstream success. It is not clear whether the method proposed by this paper is uncovering something truly new or summarizing existing signals related to task difficulty; the lack of novelty in the proposed method and findings is a substantial weakness here.

W3: The paper has clarity and presentation issues. For example, the dense figure on the very first page of the paper cannot be interpreted by the reader until they read the methodology section, and the Introduction combines many ideas together without clearly motivating the paper.

Questions for authors

In relation to the weaknesses stated above, it would be good for the authors to clarify the following:

Question in relation to W1: given an unseen task with an unknown success rate in practice, how should a researcher use this method and instantiate L and D? What key takeaways does the analysis in this paper have to guide researchers on how L and D should be interpreted for different types of tasks to estimate task-level reliability?

Question in relation to W2: Is measuring information about interaction boundaries in trajectories more effective than computing and using even simpler measurements over trajectories (e.g., model confidence, failed tool calls, etc)? It would be good to have more discussion and clarification about what the proposed method captures beyond simpler signals.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2JHY

We thank the reviewer for the detailed feedback and constructive suggestions.

Response to W1: Practical significance and interpretation of LL and DD.

Our paper is an empirical investigation of what a model’s own execution statistics reveal about task-level reliability in long-horizon interaction. Work on LLM reasoning has demonstrated the potential of intrinsic confidence and uncertainty signals for guiding inference and learning. This motivates a related question for agents: does useful reliability information remain observable when reasoning is embedded in repeated actions and environmental feedback, and how should that information be measured?

Our rollout–replay protocol provides a controlled way to investigate this question. The original manuscript examines measurement location through event-aligned comparisons (Sec. 5.1; App. A.3, Table 6), persistence across fresh executions through independent-batch evaluation (Sec. 5.2; App. A.5, Tables 10–11), and actor–reader compatibility through cross-model replay (Sec. 5.4; App. A.6, Tables 12–13).

For a new task, the frozen measurement rules can be applied to repeated unlabeled executions to compute LL and DD as the mean and standard deviation of rollout summaries (Secs. 4.1–4.2; Eq. (3); App. A.4, Table 7). In the additional setting-adjusted A→B analysis below, Event LL provides a fixed ranking signal across all observed success-count groups, while DD describes variation across executions and is informative in the mixed-outcome comparison.

These findings motivate future work on self-monitoring, compute allocation, and experience selection; demonstrating improvements from such interventions requires additional evaluation.

Response to W2: Comparison with simpler signals and the contribution of event alignment.

The original manuscript already compares standard intrinsic confidence readouts, including log probability, entropy, margin, and token rank (App. A.3, Table 6), and simple execution statistics (App. A.4, Table 9). Table 1 specifies that the matched whole-trajectory comparator uses margin with the same eight-execution aggregation. To extend this evidence to a tool-failure proxy and fresh outcomes, the additional analysis below uses A measurements to rank B outcomes:

ScoreB Stable AUCB Mixed AUCFull-range, setting-demeaned ρ\rho
Event LL0.7520.6630.56
Whole-trajectory LL0.6740.6130.38
Event DD0.6320.6910.18
Whole-trajectory DD0.6010.6210.08
Tool-failure rate0.6500.6240.34

Event LL has higher point estimates than tool-failure rate on both subgroup AUCs and full-range correlation. Each event-aligned aggregation also exceeds its whole-trajectory counterpart on these metrics.

As complementary evidence, Sec. 5.3 and App. A.7, Table 15 already report controls for tool errors and execution length. In the original same-batch no-tool-errors comparison, event-aligned scores achieve stable/mixed AUCs of 0.801/0.727, versus 0.676/0.625 for the matched comparator. Table 15 also reports call-, trajectory-length-, and observation-length-matched comparisons. These restrictions complement the new independent-batch failure-rate baseline above.

We use established token-distribution statistics. Our contribution concerns where to read them, how to aggregate them across executions, and whether the resulting task-level measurements anticipate outcomes on fresh executions. These performance comparisons do not establish statistical independence from failure signals or identify a unique internal mechanism.

Response to W3: Presentation and qualitative examples.

Figure 1 was intended to summarize the main evidence, but we agree that it introduces several definitions before readers encounter the method. We will simplify the opening figure, organize the Introduction around the practical question of measuring reliability while external outcomes remain unavailable, and clarify the empirical contribution and its implications in the Discussion. We will also add a worked trajectory example showing the interaction events, readouts, and aggregation, making the numerical findings easier to interpret.

Thank you again for your detailed feedback. We would very much appreciate your thoughts on our response, particularly the clarification of how to use the scores on new tasks and the additional comparisons with simpler baselines. We would welcome the opportunity to discuss any remaining concerns.

Dear Reviewer 2JHY,

As the discussion deadline approaches, we would appreciate hearing your thoughts on our response. Could you let us know whether any concerns remain? We would welcome the opportunity to discuss them before the author discussion closes.

Thank you for your time.

Reviewer_hCx7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the reliability of a fixed LLM agent on a given task can be inferred from unlabeled execution traces without external outcome feedback. The proposed rollout–replay protocol collects repeated natural executions and replays their recorded histories through the original model to extract intrinsic token-distribution statistics at action boundaries and post-observation updates. These statistics are aggregated across executions into Level (mean) and Dispersion (standard deviation), which are evaluated for distinguishing stable and mixed-outcome task groups, respectively.

Experiments span four main benchmarks (BFCL, AppWorld, ScienceWorld, and OdysseyArena), a held-out benchmark (MCPMark), and eight model checkpoints from the Gemma 3 and Qwen3.5 families. At eight scoring rollouts, the method achieves pooled AUCs of 0.816 and 0.743 for stable and mixed groups, compared with 0.685 and 0.634 for the matched whole-trajectory comparator. Independent A-to-B evaluation, pre-outcome-disclosure controls, and cross-model replay further examine signal persistence and model specificity.

Critical strengths

1. A controlled protocol for measuring reliability without outcome supervision.

The rollout–replay framework separates natural agent behavior from retrospective measurement, avoiding intervention in the execution policy. Freezing measurement choices before evaluation on independent execution seeds is a particularly valuable design choice. The A-to-B analysis reports AUCs of 0.752 and 0.691, demonstrating that the extracted signals are not solely associated with the particular executions used for scoring. Together with held-out MCPMark evaluation, this provides meaningful support for the central claim that unlabeled traces contain information about recurring task-level behavior.

2. Consistent empirical evidence for the importance of event alignment and actor–reader compatibility.

The event-aligned method improves over the matched whole-trajectory comparator by 0.131 and 0.109 AUC on the main stable and mixed comparisons, with reported paired confidence intervals excluding zero. The improvement is observed across multiple readout statistics and evaluation restrictions. Furthermore, cross-model replay demonstrates that a larger reader does not necessarily yield better reliability rankings for trajectories generated by another model. These experiments provide complementary evidence for the importance of both measurement location and model compatibility.

Critical weaknesses

1. Outcome-conditioned evaluation limits the interpretation and applicability of the reliability scores.

The main evaluations distinguish 8/8-success from 0/8-success groups for stable tasks and 5–7/8 from 1–3/8 groups for mixed tasks, excluding groups with 4/8 successes. These are selected binary discrimination problems rather than a unified assessment of reliability across the full range of task outcomes. Moreover, Level and Dispersion are evaluated on different outcome-defined subsets, making their reported AUCs not directly comparable. Since these subsets cannot be identified in advance without outcome information, the experiments do not establish how the method would rank arbitrary incoming tasks or distinguish more finely graded success probabilities. The authors acknowledge this limitation, but it substantially restricts the practical interpretation of the proposed representation.

2. The independent-batch evidence does not fully establish the method's incremental value over simpler alternatives.

Although the A-to-B experiment is an important contribution, the paper reports the proposed scores' AUCs without corresponding independent-batch results for the matched whole-trajectory comparator or simpler task-difficulty baselines. Consequently, it remains unclear whether the principal comparative gains persist when both the scores and the evaluation outcomes come from disjoint execution batches. The method also requires repeated natural executions and model replay, whereas the reported relative cost primarily counts replayed executions rather than end-to-end computational expense. Without stronger independent-batch comparisons and more complete cost accounting, the advantage over cheaper reliability heuristics remains uncertain.

Questions for authors

1. Evaluation across the full reliability spectrum. Could the authors evaluate Level and Dispersion across all observed success-count groups, including 4/8 cases, rather than separately on the stable and mixed binary subsets? In particular, can a single score or outcome-independent combination provide useful rankings across tasks with different underlying success rates? Such an analysis would materially strengthen the interpretation of the method as a general task-level reliability measure.

2. Independent-batch baselines and computational cost. For the A-to-B evaluation, could the authors report the performance of the matched whole-trajectory comparator and simple execution-statistic or task-input-only baselines, together with paired confidence intervals? Could they also quantify total inference cost, including natural rollout collection and replay, rather than only relative replay counts? These results would help establish whether the observed reliability signal provides additional value under realistic computational constraints.

3. Experimental consistency and reproducibility. Table 3 reports Qwen3.5-27B self-replay AUCs of 0.806/0.739, whereas Tables 1–2 report 0.816/0.743 for the main setting, which is also described as using Qwen3.5-27B. Could the authors clarify whether this difference arises from evaluation support, random seeds, or configuration changes? A more explicit specification of the final selected event masks, readout reductions, and missingness rules would also improve reproducibility.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hCx7

We thank the reviewer for the constructive feedback.

The original stable/mixed comparisons evaluated Level and Dispersion on different outcome-defined subsets and excluded 4/8 groups. The manuscript already discusses this limitation in Sec. 6 and reports the included and excluded group counts in App. A.1, Table 4. We now evaluate each fixed A-batch score against B outcomes across all success counts, including 4/8, without using B group membership to select the score.

A-batch scoreFull-range, setting-demeaned Spearman ρ [95% CI]
Event Level0.56 [0.49, 0.62]
Event Dispersion0.18 [0.10, 0.26]
Whole-trajectory Level0.38 [0.30, 0.45]
Whole-trajectory Dispersion0.08 [0.00, 0.16]

Event Level has the highest full-range correlation point estimate among these scores. Dispersion has a higher AUC point estimate within the mixed comparison, as shown below, but a weaker full-range correlation. The findings support ranking under the setting-adjusted evaluation, without establishing calibrated probabilities or unrestricted transfer to unseen tasks.

The original manuscript reports execution-statistic baselines in Table 1 and App. A.4, Table 9, and independent-batch persistence in Sec. 5.2 and App. A.5, Tables 10–11. To examine whether the comparative gains also persist across batches, we now add the following independent-batch baseline comparisons. Every method uses A measurements to rank disjoint B outcomes:

Batch-A scoring methodB stable AUCB mixed AUC
Event Level0.7520.663
Event Dispersion0.6320.691
Whole-trajectory Level0.6740.613
Whole-trajectory Dispersion0.6010.621
Model-call count0.6180.603
Generated token count0.6070.590
Trajectory length0.6130.598
Tool-failure rate0.6500.624
Task-input length0.5540.541

The AUC point estimates for Event Level and Dispersion exceed their matched whole-trajectory counterparts by 0.078 and 0.070 on the stable and mixed comparisons, respectively, and exceed the listed simpler baselines on those corresponding comparisons. We will add paired confidence intervals for these A→B differences in the revision, complementing the original main and held-out paired comparisons in App. A.7, Table 16.

Regarding cost, offline replay lets us compare readouts on identical trajectories; the reported offline protocol incurs both natural rollout collection and replay costs. For self-replay, an implementation with access to the required token distributions during generation could instead collect the fixed readout online without separate replay forward passes. This implementation would still require KK natural executions. We have not yet measured the overhead of collecting and aggregating these statistics. Cross-model replay still requires the alternative reader’s forward passes.

The manuscript also examines the scoring-budget trade-off in Sec. 5.1, Fig. 3b, and App. A.4, Table 8: in that original same-batch ablation, four scoring executions yield stable/mixed AUCs of 0.798/0.726, versus 0.816/0.743 for eight. Table 8 counts replayed executions; the current results do not quantify end-to-end inference cost.

Finally, the main-setting results in Tables 1–2 use the full eligible B task set, whereas Table 3 reuses the cross-model self-replay diagonal shown in Sec. 5.4, Fig. 4, and App. A.6, Tables 12–13. To clarify the support difference, for each acting model, the cross-model comparison retains only task groups with valid scores for every reader, ensuring that readers rank identical tasks and trajectories. In particular, Table 13 reports Qwen3.5-27B self-replay AUCs of 0.806/0.739 on this restricted support, versus 0.816/0.743 on the full support.

Secs. 4.1–4.2 discuss the reconstruction requirements and measurement choices. We will make the final configuration explicit in a table covering event/token masks, probability conventions, reductions, the standard-deviation convention, score directions, and missingness/exclusion rules. We will also report positive/negative counts for each cross-model column, supplementing the batch-level counts already reported in App. A.1, Table 4.

Thank you for your detailed rebuttal. It addresses my concerns, so I will keep my original score.

Thank you for following up. We’re glad our response addressed your concerns, and we appreciate your time and thoughtful feedback.

Reviewer_QJoJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether unlabeled execution traces can rank the task-specific reliability of a fixed agent. The proposed rollout–replay protocol collects independent executions and replays their recorded histories through the acting model, extracting token-distribution statistics at action boundaries and post-observation events. Rollout-level summaries are aggregated into a mean score (Level) and an across-rollout dispersion score (Dispersion). The paper reports improvements over matched whole-trajectory statistics, persistence across independent execution batches, and transfer to a held-out benchmark. Cross-model replay experiments also suggest that matching the replay model to the acting model matters. The results concern reliability ranking rather than calibrated success probabilities or demonstrated improvements in downstream decisions.

Critical strengths

The matched comparisons provide useful evidence for the value of event-aligned measurement. Table 1 compares event-aligned and whole-trajectory signals using the same eight-rollout budget and corresponding aggregation statistics. The reported AUC improvements of 0.131 on stable groups and 0.109 on mixed groups suggest that measurement location contributes information beyond simply collecting more executions.

The evaluation includes meaningful checks beyond same-batch association. Independent A-to-B evaluation, held-out MCPMark results, and pre-outcome-disclosure analysis strengthen the evidence that the signal is not explained solely by the outcomes of the scored executions or explicit outcome disclosure. The paper also appropriately limits its claims regarding calibration, causal mechanisms, and downstream utility.

Critical weaknesses

The evaluation primarily establishes discrimination within outcome-defined subsets rather than ranking across the full task population. Stable and mixed groups are evaluated separately using different statistics, and tasks with exactly four successes out of eight are excluded. Consequently, the results do not yet establish how well either fixed score ranks an arbitrary collection of tasks when group membership is unknown. The paper acknowledges this limitation, but it materially restricts the scope of the reliability-ranking evidence.

Questions for authors

I have no additional questions for the authors at this stage.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QJoJ

We thank the reviewer for the thoughtful assessment and for suggesting a broader evaluation of reliability ranking.

Our original evaluation separated two questions, as discussed in Sec. 5.1 and examined in the aggregation ablation (App. A.4, Table 7): whether the mean readout distinguishes consistently successful from consistently unsuccessful observed executions (8/8 versus 0/8), and whether across-execution dispersion distinguishes higher- from lower-success groups when both contain successes and failures (5–7/8 versus 1–3/8). We excluded 4/8 cases from the latter binary comparison because neither outcome predominates; App. A.1, Table 4 reports these excluded groups alongside the evaluated groups. Sec. 6 explicitly discusses the resulting limitation: this design characterizes the roles of Level and Dispersion within these outcome regimes, but does not by itself establish ranking across the full task population.

The original independent A→B analysis is reported in Sec. 5.2 and App. A.5, Tables 10–11. Here, we extend it to all success-count groups, including 4/8 tasks. Each A-batch score retains its fixed definition and direction; B outcome-group membership plays no role in choosing the score. The full-range correlations and additional score comparisons below are new analyses that complement the original subgroup results. We report subgroup AUCs and full-range Spearman correlations with B success rates after removing setting-level differences.

A-batch scoreB Stable AUCB Mixed AUCFull-range, setting-demeaned Spearman ρ [95% CI]
Event Level0.7520.6630.56 [0.49, 0.62]
Event Dispersion0.6320.6910.18 [0.10, 0.26]
Whole-trajectory Level0.6740.6130.38 [0.30, 0.45]
Whole-trajectory Dispersion0.6010.6210.08 [0.00, 0.16]

Event Level shows a positive full-range association with independent B outcomes, with a higher correlation point estimate than Whole-trajectory Level (0.56 versus 0.38). This supports ranking beyond the original outcome-defined subsets using a single fixed score, without prior knowledge of group membership, under the setting-adjusted evaluation. Dispersion’s mixed-group AUC point estimate is higher than Level’s (0.691 versus 0.663), but its full-range correlation is weaker (0.18 versus 0.56), indicating that its usefulness depends more on the comparison being made. These findings do not establish calibrated success probabilities.

Thank you again for your thoughtful review. If you have a chance to revisit our response, we would greatly appreciate your thoughts on the additional full-range evaluation, which includes the 4/8 cases and evaluates each fixed score without using outcome-group membership. Please let us know if any concerns remain or if further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.