acceptodds
Under review as a conference paper at ICLR 2027

Auditing LLM API Quality Beyond Task Success

Abstract

Black-box access makes it difficult to determine whether the quality delivered by an LLM API has deteriorated. For agentic workloads, task pass rates summarize outcomes but do not reveal whether decisions were grounded in observations, instructions were followed, or errors were handled appropriately. We study whether process-quality measurements provide actionable auditing information beyond final outcomes. We propose a trajectory-based auditing framework that uses generative judges to produce dimension-specific assessments with supporting evidence and compares them with reference measurements under a controlled workload. To accommodate long trajectories, the framework combines compressed views with selective access to omitted evidence. We develop an evaluation protocol using natural trajectories, targeted quality interventions, and null edits, with independent checks of the realized effects. The protocol compares outcome-only, process-only, and combined audit signals, and evaluates detection, false alarms, evidence localization, and judging cost. Comparisons across judge models and execution harnesses assess the dependence of the measurements on the evaluator. The study examines when observable process quality complements pass rates as a practical signal for black-box API auditing.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, black-box API auditing, agent evaluation, process quality, LLM-as-a-judge, trajectory analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9e4tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether trajectory-level process assessments can complement task outcomes when auditing black-box LLM services. It scores five dimensions with llm judges, compares four evidence-access workflows, and validates responses to targeted and quality-preserving edits. The main scoring study contains eight original trajectories and 1,458 repeated judgments. A separate replay study uses 1,358 trajectories across sixteen tasks to compare outcome, execution-statistic, and process signals under prompt and deployment changes. Scores generally decrease for selected degradations, but evaluator disagreement is substantial, and the monitoring results do not establish improved degradation detection at matched false-alarm rates.

Critical strengths
  • The monitoring analysis reports false alarms, exclusions, reused trajectories, and unsuccessful calibration diagnostics.

  • The controlled-edit protocol is useful.

Critical weaknesses
  • The main benefit of central auditing is still unclear. In Table 2, using local MMD improves the detection of changes in deployment, but it also leads to more false alarms. The disjoint-calibration diagnostic results in local process-only detection and false alarms of 0% by reaching threshold saturation, rather than solving the problem of balancing these two issues.

  • The process measurements have too little independent validation to support the proposed use.

  • 4.2 is single-sourced on trajectories (all 1,358 are DeepSeek) so neither the judge configuration nor the source distribution is varied in the monitoring half. It remains unclear whether this work with a different judge?

Questions for authors
  • Can the saved replay results support a comparison of process-containing and process-free detectors at matched held-out false-alarm rates?

  • Were the appended intervention instructions in Appendix E.1 visible to the scoring judge as part of the canonical record? If so, how does the rubric distinguish inefficient behavior from compliance with an explicit requirement to repeat work? If they were removed, please specify the judging input and how applicable instructions were preserved.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer 9e4t (Part 1)

W1

Online Monitor Experiments: In our original monitoring experiments, both Global and Local used normal trajectories collected in advance for each task. Global pooled these trajectories, while Local built a separate reference for each task.

Disjoint Calibration: Thank you for raising this point. The calibrated threshold was too high for any run to trigger an alarm, yielding 0% detection and 0% false alarms. Separating the calibration data also left each stage with five trajectories per task instead of ten. The experiment therefore cannot determine whether the failure came from separating the data, having fewer calibration samples, or both.

Together, these limitations motivated us to develop a monitor that starts from an initial period of normal account activity. Partial pooling estimates each task's expected scores and usual variation using both its own observations and information from other tasks. Tasks with little history rely more on the shared prior; as their own history grows, it contributes more to the estimate. We compare recent scores with their expected normal values and use CUSUM to accumulate signs of a change. We set the alarm threshold using historical data from normal service.

Single-account results. We replayed the existing DeepSeek trajectories on sixteen tasks, monitoring one account (K=1) with four concurrent requests and introducing a change after an initial normal period. Deployment changes cover model version, serving endpoint and reasoning budget. Prompt interventions cover repeated reasoning, redundant tools and delayed recovery; the two tool-dependent interventions use twelve tasks. Each family averages its three conditions equally.

The original methods use five reference and ten calibration trajectories per task and a ten-score window. The new method starts from 32 normal account scores and checks the most recent twenty scores. The original rows below cover abrupt changes; the new rows cover abrupt and gradual changes.

Entries are detection @ measured false-alarm rate (%), with a 5% calibration target for every method. A run counts as detected only if the monitor has received a changed request's score before its first alarm. A false alarm is any alarm during an entirely normal run. P denotes process scores; Vote₂ requires at least two of verifier outcomes, execution statistics and process scores.

MethodDeployment: P-onlyDeployment: Vote₂Prompt: P-onlyPrompt: Vote₂
Submitted MEWMA / Global13.00 @ 11.832.50 @ 0.503.33 @ 7.330.00 @ 0.17
Submitted MEWMA / Local43.83 @ 87.6765.50 @ 34.3350.33 @ 73.8323.33 @ 23.50
Submitted MMD / Global19.00 @ 10.671.83 @ 0.333.00 @ 10.670.00 @ 0.17
Submitted MMD / Local38.33 @ 89.1764.83 @ 35.6744.17 @ 78.6725.50 @ 24.67
Partial + window CUSUM (abrupt)26.67 @ 11.2525.83 @ 2.504.17 @ 9.170.83 @ 1.25
Partial + window CUSUM (ramped)19.17 @ 11.2522.92 @ 2.504.58 @ 9.170.83 @ 1.25

The revised method reduces reliance on precollected task-specific reference data while retaining the ability to detect deployment changes.

W2

Blinded human labels. We added human validation for six additional trajectories, one per task, covering two tasks each from AutomationBench, SWE-smith and Terminal-Bench. The human reviewer assigned scores without seeing the GLM-5.3/Codex or Astra scores and rationales.

TaskRGTUEffERIF
Multi-system offboarding4.50/4/52.33/2/22.83/1/22.17/2/22.33/2/2
Customer feedback routing10.00/7/89.83/10/98.50/9/99.83/10/108.67/6/8
Cookiecutter hook validation7.83/7/77.83/7/78.00/4/69.50/8/96.83/6/6
SQLFluff Jinja templating6.33/6/68.33/7/86.67/6/69.67/10/105.83/4/4
COBOL modernization7.83/6/77.17/7/84.50/3/58.83/10/103.00/3/5
Windows 3.11 installation7.00/6/76.33/6/76.67/6/76.33/6/77.00/6/7

Each cell shows Average/Astra/Human: the mean of six GLM-5.3/Codex judgments, the Astra AI assessment, and the human reviewer's score. RG/TU/Eff/ER/IF denote reasoning grounding, tool use, efficiency, error recovery and instruction following.

The monitoring results in Q1 also show that, for the GLM-5.3→GLM-5.3-Flash switch, process scores achieve higher detection than execution statistics alone at the same measured false-alarm rate.

W3

We ran additional experiments on using more LLMs as task solver and judge. And for details please refer to Q1.

AuthorsOpenReview ↗

Responses to Reviewer 9e4t (Part 2)

Q1

We compare V (verifier outcomes), S (execution statistics), P (process scores) and their combinations using the Partial + window CUSUM monitor described in W1. All results use one account (K=1). Vote₂ requires at least two of V/S/P.

The full DeepSeek columns summarize the deployment changes and prompt interventions described in W1. The mini panels test DeepSeek V4.1→V4, GLM-5.3→5.3-Flash and HY4→HY3 configuration switches. Gradual changes increase the share of requests using the changed condition.

AUROC: abrupt / gradual changes.

We use the highest monitoring score from each run to distinguish normal from changed runs. Higher AUROC means better separation.

SignalFull DeepSeek: deploymentFull DeepSeek: promptMini: DeepSeekMini: GLMMini: HY
V0.7363/0.71280.5972/0.58640.9533/0.91250.8275/0.77410.9502/0.9122
S0.8253/0.79790.5157/0.51000.9720/0.95440.7176/0.68310.9984/0.9920
P0.7019/0.67220.4696/0.45940.8592/0.79050.8845/0.84770.8486/0.8002
V∧S0.7386/0.72060.5610/0.54920.9411/0.90120.7406/0.69770.9478/0.8931
V∨S0.7693/0.75540.5511/0.53330.9663/0.92990.7991/0.72340.9922/0.9750
V∧P0.7146/0.69230.5383/0.52110.8429/0.79580.8364/0.77500.8217/0.7831
V∨P0.7040/0.67250.5285/0.52450.9228/0.86400.8731/0.81060.9339/0.8873
S∧P0.7274/0.69240.4898/0.48590.8644/0.80910.7682/0.73290.8306/0.7981
S∨P0.7833/0.77200.5094/0.50540.9513/0.90860.8552/0.80130.9839/0.9672
V∧S∧P0.7303/0.71060.5448/0.53300.8827/0.84230.7877/0.72940.8405/0.7997
V∨S∨P0.7784/0.76210.5362/0.52640.9688/0.93170.8861/0.82410.9920/0.9725
Vote₂0.7640/0.73530.5259/0.50850.9527/0.90370.8763/0.80880.9747/0.9342

Detection: abrupt / gradual @ measured FPR (%).

A run counts as detected only if the monitor has received a changed request's score before its first alarm. FPR is the percentage of normal runs that raise any alarm. Thresholds for this table were fitted on normal data with a 5% false-alarm target.

SignalFull DeepSeek: deploymentFull DeepSeek: promptMini: DeepSeekMini: GLMMini: HY
V30.00/25.00 @ 11.259.17/7.92 @ 9.1772.50/55.00 @ 5.0047.50/31.25 @ 13.7582.50/71.25 @ 11.25
S52.50/51.67 @ 7.927.92/8.75 @ 11.2586.25/80.00 @ 11.2537.50/35.00 @ 11.2597.50/93.75 @ 10.00
P26.67/19.17 @ 11.254.17/4.58 @ 9.1736.25/26.25 @ 7.5063.75/48.75 @ 11.2547.50/36.25 @ 8.75
V∧S16.25/14.58 @ 0.000.00/0.00 @ 0.0063.75/46.25 @ 0.0016.25/11.25 @ 1.2581.25/68.75 @ 1.25
V∨S53.33/49.58 @ 15.4214.17/10.42 @ 13.3385.00/75.00 @ 12.5058.75/47.50 @ 16.2598.75/96.25 @ 13.75
V∧P5.42/4.58 @ 2.500.42/0.42 @ 1.2528.75/12.50 @ 1.2522.50/15.00 @ 0.0042.50/31.25 @ 2.50
V∨P36.67/27.08 @ 13.3310.42/7.50 @ 10.4272.50/57.50 @ 6.2575.00/51.25 @ 18.7582.50/71.25 @ 13.75
S∧P15.00/10.83 @ 0.000.42/0.42 @ 0.0027.50/16.25 @ 1.2521.25/13.75 @ 0.0047.50/35.00 @ 2.50
S∨P50.00/48.75 @ 11.678.33/7.50 @ 10.4287.50/73.75 @ 11.2570.00/57.50 @ 13.7597.50/92.50 @ 11.25
V∧S∧P3.33/1.25 @ 0.000.00/0.00 @ 0.0025.00/11.25 @ 0.008.75/7.50 @ 0.0042.50/31.25 @ 1.25
V∨S∨P50.83/48.33 @ 15.0015.00/11.25 @ 10.8388.75/76.25 @ 10.0077.50/58.75 @ 21.2598.75/92.50 @ 16.25
Vote₂25.83/22.92 @ 2.500.83/0.83 @ 1.2568.75/51.25 @ 2.5040.00/25.00 @ 1.2585.00/72.50 @ 3.75

Detection at matched false-alarm rates.

For the GLM switch, S-only and P-only have the same measured FPR at the two operating points below.

Operating pointSignalRealized FPR (%)Abrupt detection (%)Ramped detection (%)
Low-FPR pointS3.7531.2521.25
Low-FPR pointP3.7547.5031.25
Main pointS11.2537.5035.00
Main pointP11.2563.7548.75

At 3.75% FPR, P improves abrupt and gradual detection over S by 16.25 and 10.00 percentage points. At 11.25% FPR, the gains are 26.25 and 13.75 points.

P also has the highest single-signal AUROC on the GLM mini panel, while S leads on the DeepSeek and HY mini panels. Prompt interventions remain weakly detected.

AuthorsOpenReview ↗

Responses to Reviewer 9e4t (Part 3)

Continue on Q1

Cross-judge detection (abrupt / gradual @ measured FPR, %). We also apply Deepseek-v4.1-flash-1017 x Claude Code as a judge on new trajectories. The new experiment shows consistent results, as process scores do have advantage under certain circumstances.

SignalGLM switch · GLM/CodexGLM switch · DeepSeek/Claude CodeHY switch · GLM/CodexHY switch · DeepSeek/Claude Code
V47.50/31.25 @ 13.7537.50/30.00 @ 13.7582.50/71.25 @ 11.2572.50/62.50 @ 10.00
S37.50/35.00 @ 11.2541.25/37.50 @ 11.2597.50/93.75 @ 10.0095.00/93.75 @ 12.50
P63.75/48.75 @ 11.2561.25/55.00 @ 13.7547.50/36.25 @ 8.7531.25/30.00 @ 15.00
V∧S16.25/11.25 @ 1.2511.25/10.00 @ 1.2581.25/68.75 @ 1.2568.75/58.75 @ 0.00
V∨S58.75/47.50 @ 16.2552.50/43.75 @ 16.2598.75/96.25 @ 13.7596.25/92.50 @ 12.50
V∧P22.50/15.00 @ 0.0025.00/18.75 @ 5.0042.50/31.25 @ 2.5031.25/26.25 @ 5.00
V∨P75.00/51.25 @ 18.7558.75/53.75 @ 16.2582.50/71.25 @ 13.7567.50/60.00 @ 13.75
S∧P21.25/13.75 @ 0.0020.00/17.50 @ 1.2547.50/35.00 @ 2.5030.00/27.50 @ 2.50
S∨P70.00/57.50 @ 13.7565.00/60.00 @ 21.2597.50/92.50 @ 11.2593.75/90.00 @ 16.25
V∧S∧P8.75/7.50 @ 0.007.50/6.25 @ 0.0042.50/31.25 @ 1.2530.00/25.00 @ 0.00
V∨S∨P77.50/58.75 @ 21.2562.50/56.25 @ 22.5098.75/92.50 @ 16.2596.25/92.50 @ 15.00
Vote₂40.00/25.00 @ 1.2537.50/31.25 @ 7.5085.00/72.50 @ 3.7570.00/61.25 @ 7.50

Q2

We re-examined the data-generation and judging pipeline and confirmed that the appended intervention instructions are excluded from the judging input. The judge input, including system prompt, task, agent trajectory and available tools, is preserved according to the harness. The actual form of judge input depends on which harness used, but no intervention instruction will be visible to agent judge. Additionally, intervention instructions will be preserved by controlled-edit construction pipeline, but they will not be passed on to scoring judge.

Reviewer_YRupOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes auditing black-box LLM APIs with process-quality scores rather than task outcomes alone. An LLM judge scores agent trajectories on five dimensions (reasoning grounding, tool use, efficiency, error recovery, instruction following), citing evidence from the record, under different reading setups (full record, compressed view with retrieval, or file-based inspection via Codex CLI / Claude Code).

Validation has two parts. First, eight trajectories each receive a controlled degradation and a presentation-only null edit, and are scored by three judges under up to four setups (1,458 judgments). Degradations lower target scores on average (mean ΔD = −0.821), while null edits change little (mean |ΔN| = 0.292). Second, simulated account-level monitoring with MMD and MEWMA compares verifier outcomes, execution statistics and process scores on 1,358 DeepSeek trajectories. Process scores increase sensitivity in some settings, but false alarms often exceed the nominal 5% target, and no advantage at matched false-alarm rates is established.

Critical strengths

1. Important problem with a careful, honest protocol. Auditing whether an API still performs agentic work to standard, beyond model identity, is practically relevant. The validation protocol is well designed: null edits serve as controls, a blinded LLM reviewer checks edits, failures are retained rather than zero-filled, and no score-dependent replacement is performed.

2. Unusually transparent reporting. The authors openly report unsupported edits, judge disagreement, calibration saturation and over-alarming. This makes the paper a useful map of what does and does not yet work in process-based API auditing.

Critical weaknesses

1. The judge-validation evidence is too thin. There are only eight independent cases (one per task). Degradation effects are small (−0.82 on a 0–10 scale), and only 19 of 81 triplets fully separate degraded from original scores across six repeats. Human labels are non-blind, assistant-supported, one per case, and the db-wal label uses a rubric revised after the pilot. This cannot establish that process scores reliably detect quality changes.

2. The monitoring results do not show that process scores add value. Realized false alarms far exceed the nominal 5%; local P's false-alarm rate even exceeds its detection rate (deployment changes, MMD: 41.47% vs. 47.14%). With disjoint calibration, local detectors saturate to 0/0. The prompt interventions, designed precisely to create process defects, are barely detected. Without a comparison at matched false-alarm rates, the paper's central question of whether process quality helps beyond outcomes remains unanswered.

Questions for authors

Questions that could change my assessment:

  1. Matched comparison: Can you report ROC curves or detection at matched realized false-alarm rates for V, S, P and their combinations?
  2. Offline separability: Independent of the monitoring machinery, how well do process scores separate normal from intervention or deployment-change trajectories (e.g., per-trajectory AUC)?
  3. Intervention realization: How often did the model actually comply with the prompt interventions (e.g., performing the repeated tool calls)? If compliance is low, the monitoring results say little about process scores.
  4. Scale: Can the controlled panel be extended beyond eight cases, with blinded human labels?

Clarifications:

  1. Page 6 states "compression is necessary or universally sufficient"; is "neither ... nor" intended?
  2. What is the judging cost (tokens, latency) per trajectory relative to the detection gain?

Suggestions (not affecting my score): align wording with the actual evidence (e.g., "To prove the value...", "very promising"), and include key results in the abstract.

1 comment
AuthorsOpenReview ↗

Responses to Reviewer YRup

W1

We added six trajectories with blinded human labels. GLM-5.3/Codex six-repeat means were within one point of 25/30 labels. See W2 of our response in the 9e4t review thread for all scores.

W2

Disjoint calibration saturated the threshold (0% detection and false alarms). Prompt interventions remained hard to detect. See our response to Reviewer 9e4t (W1) for the revised monitor and single-account replay.

Q1

For the GLM switch at 3.75% measured false-alarm rate, process scores improved abrupt and gradual detection over execution statistics by 16.25 and 10.00 percentage points. See our response to Reviewer 9e4t (Q1) for all signal combinations, AUROCs, and cross-judge comparisons.

Q2

We evaluated the existing per-trajectory process scores without the online detection algorithm. For each task and dimension, we enumerate normal–changed rollout pairs and report AUC = Pr(s_normal > s_changed) + 0.5 Pr(s_normal = s_changed), then average across tasks.

Panel and change conditionRGTUEffERIF
Original: DeepSeek version switch60.6 [52.4,68.8]56.9 [51.1,63.0]48.5 [38.2,58.8]60.9 [52.7,68.9]56.0 [50.4,61.3]
Original: endpoint switch44.3 [38.6,50.1]50.8 [44.0,58.0]56.5 [50.7,62.3]45.0 [39.2,50.4]45.8 [40.0,52.2]
Original: low reasoning budget48.2 [44.2,52.2]56.0 [51.3,60.9]53.3 [47.6,58.7]51.4 [45.3,57.2]53.2 [50.3,56.3]
Original: repeated reasoning47.2 [41.7,52.5]48.1 [41.5,55.1]47.0 [40.7,53.0]44.2 [39.5,49.3]46.2 [42.2,50.3]
Original: redundant tools57.7 [48.9,67.8]49.0 [41.2,57.1]58.4 [50.8,65.8]53.0 [45.6,60.9]48.3 [41.3,56.4]
Original: delayed recovery48.9 [42.5,54.8]50.4 [45.5,55.0]53.4 [46.9,59.1]49.2 [42.6,55.2]48.6 [43.6,53.7]
Mini: DeepSeek 1017→073156.1 [43.3,67.3]56.2 [41.5,70.5]49.1 [38.7,59.2]60.8 [50.3,70.9]58.3 [50.9,64.9]
Mini: GLM-5.3→5.3-Flash58.6 [52.9,64.7]56.0 [48.1,63.9]55.1 [48.5,61.6]53.8 [46.8,60.4]59.4 [52.7,66.0]
Mini: HY4→HY374.6 [64.8,83.0]51.1 [39.0,60.4]53.3 [39.1,68.1]67.6 [55.4,78.3]58.1 [49.6,66.2]

Values are percentages [95% CI] from 100,000 benchmark-stratified task bootstrap resamples. RG/TU/Eff/ER/IF denote reasoning grounding, tool use, efficiency, error recovery and instruction following. The original panel covers 16 tasks, with 12 for redundant tools, delayed recovery and TU; the mini panel covers a 12 task subset selected from 16 tasks.

The scores distinguish some switches, particularly HY4→HY3 on RG and ER, while discrimination is limited for several other conditions. Thus, single-trajectory discrimination depends on both the process dimension and the change condition, and online monitor algorithm is necessary especially when accurate detection and low false alarm rate is needed.

Q3

We examined the agent-generated edits and confirmed that all 16 reviewed variants contained the requested types of local changes. These edits are small and may have limited impact on whole-trajectory scores.

Q4

See our response to Reviewer 9e4t (W2) for the six additional trajectories and blinded ratings.

Q5

Thank you for the suggestion. On the three tasks we tested, compressed and file-based workflows yielded more consistent scores across repeated judgments than full-direct scoring.

Q6

Thank you for the suggestion. The table below reports mean input tokens, output tokens, latency, and cost per judgment for each judge–harness configuration.

JudgeHarnessJudgments / judgments with reported costMean input tokensMean output tokensMean latency (min)Mean cost (USD)
DeepSeek V4.1 FlashFD144 / 14447.5k8.0k6.040.0083
DeepSeek V4.1 FlashCA54 / 54382.8k10.2k9.960.0267
DeepSeek V4.1 FlashCC144 / 131501.6k18.7k13.510.0230
DeepSeek V4.1 FlashCX144 / 143544.0k19.4k8.080.0263
GLM-5.3FD144 / 14446.8k28.4k3.570.1503
GLM-5.3CA54 / 54554.8k60.7k6.930.5248
GLM-5.3CC144 / 140843.4k44.7k6.220.4483
GLM-5.3CX144 / 144587.4k10.4k3.800.2103
Qwen3.8-MaxFD144 / 14450.1k12.7k5.300.1032
Qwen3.8-MaxCA54 / 54433.5k27.4k10.360.4068
Qwen3.8-MaxCC144 / 144635.5k40.0k15.371.4598
Qwen3.8-MaxCX144 / 1421,091.3k36.6k15.430.6462

Q7

We have replaced “To prove the value…” with “To examine how process scores respond to changes in execution” and removed “very promising.” The abstract now also reports the main controlled-edit findings and the monitoring trade-off between detection and false alarms.

Reviewer_pQFNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether LLM-judge scores of process quality on five trajectory dimensions give a client more information than task pass rates when a black-box API may have changed. It combines a controlled-edit experiment (eight originals, degraded and null versions, several judges and harnesses) with a monitoring study that replays DeepSeek trajectories and compares detectors built on verifier outcomes, execution statistics and process scores. The authors acknowledge that the results do not establish an advantage at matched false-alarm rates or independently validated degradation detection.

Critical strengths

The question is impornant, since identity tests do not show whether the agent behind a changed API still works. The harness comparison is the most useful contribution: full-prompt judging is the least repeatable configuration for every judge on long records. The authors report limitations, including a diagnostic that weakens their own result, and the reported numbers are consistent.

Critical weaknesses
  1. The providedevidence does not answer the research question. No detector is compared at a matched false-alarm rate, and process scores alarm on unchanged accounts about as often as they detect changes. The only rule close to its target does not use process scores, and a diagnostic with disjoint calibration pools removes the effect entirely. The comparison table omits the relevant ablations. The results come from a simulated replay with a single generator and a single judge, and that judge was not validated with the revised monitoring rubric. A matched false-alarm comparison against execution statistics alone, computable from the existing data, would test the main claim.

  2. The paper does not engage with work on evaluating LLM judges of agent trajectories, although its method depends on such a judge. Zheng et al. (2023) MT-Bench is the standard reference on judge biases and agreement with humans. Who&When (ICML 2025) shows that judges struggle to locate the decisive step in long multi-agent traces. MAST (NeurIPS 2025) derives its failure taxonomy from annotated traces, whereas the five dimensions used here are not grounded in such annotation and overlap in the paper's own data. AEGIS (ICLR 2026) validates judges on a much larger set of injected errors; this paper uses eight originals. TRAIL and the concurrent Who&When Pro and AdaMAST are also relevant.

  3. Judge validation uses no standard metric and reports no uncertainty estimate, although the paired design supports per-call pairwise accuracy. Some tasks show no decrease at all. Scores cover whole trajectories while edits affect only a few steps, so the evidence-localization claim in the abstract is not tested, and human validation relies on a small set of non-blind labels.

  4. The harness setup is insufficiently specified. Codex CLI and Claude Code are designed for other model families, and the paper does not describe how they were run with the judges used, or which tools and network access the judges had. The comparison is therefore not reproducible, and a judge with web access could have retrieved the task instead of reading the record. No GPT, Claude or Gemini judge is tested.

  5. The abstract announces evaluations that do not appear in the paper, a figure is never referenced, and there is no conclusion.

Questions for authors
  1. What are the detection and false-alarm rates of execution statistics alone and of their combination with process scores, and how do process scores compare at matched false-alarm rates or by ROC AUC?

  2. Can you report per-call pairwise accuracy for each judge configuration, with uncertainty across tasks?

  3. How were Codex CLI and Claude Code run with the judges used, and which tools and network access did the judges have?

  4. In what fraction of trajectories did the requested behaviour occur under each prompt intervention?

  5. How do the five dimensions relate to taxonomies such as MAST and TRAIL, and what evidence shows that they are separable?

  6. Can the controlled-edit protocol be scaled toward error-injection pipelines such as AEGIS?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer pQFN (Part 1)

W1

Thank you for the suggestions. See W1 of our response in the 9e4t review thread for the disjoint-calibration diagnostic, revised monitor, and single-account replay. See Q1 below for matched false-alarm comparisons, ablations, and cross-judge results.

Validating new rubric: We have re-run scoring on the Terminal db-wal trajectory and its degraded/null variants using the revised monitoring rubric across nine judge–harness configurations, with six repetitions per version.

MetricOriginal rubricRevised rubric
Configurations with degraded mean below original3/92/9
Configurations with degraded mean above original4/96/9
Mean score difference (degraded − original)+0.148+0.185
Degradation pairwise accuracy28.4%24.4%
Null exact agreement46.6%41.1%
Mean within-trajectory SD0.981.18

Statistics average configurations using available successful scores. The aggregate score contrast changes by only 0.037 points. Low pairwise accuracy under both rubrics is consistent with the limited effect of the two localized edits (1.29% of trajectory text) relative to scoring variability.

W2

Thank you for pointing us to these studies. We discuss their relevance below and report an additional analysis of how judges locate and interpret scoring evidence.

  • MAST and AdaMAST have reminded us to explain how we designed the five evaluation dimensions and how they relate to these taxonomies. For details, please refer to Q5.
  • We have enriched our controlled degradation according to the method in AEGIS.
  • MT-Bench, Who&When, Who&When Pro, and TRAIL highlighted the need to analyze agent-judge trajectories, understand disagreements between human and agent judgments, and check whether scoring rationales correctly assess solver trajectories. We analyzed 27 GLM-5.3 judgments of eight original trajectories across four harnesses. In Codex, 67 of 105 tool responses were truncated, often prompting narrower follow-up reads. Also, the judges generally identified the main evidence, but their rationales sometimes misinterpreted it. Judges also sometimes reached different scores from the same evidence because they interpreted instructions differently or differed in how much weight they gave a defect.

W3

We agree that the edits are small, changing approximately 0.005–1.846% of the original trajectory's characters. Pairwise accuracy and uncertainty are reported in Q2.

For each task, we inspected the final rationale from the first of six scoring repetitions in each scheduled judge–harness configuration. The denominator is 12 (3 judges × 4 harnesses) for the two math tasks and HelpScout, and 9 (3 × 3) for other tasks, where compressed agentic was not scheduled. The numerator counts rationales explicitly identifying at least one actual edit.

TaskLocalized configurations / checked (%)
Math relations4/12 (33.3%)
Math series0/12 (0.0%)
Calendly9/9 (100.0%)
HelpScout12/12 (100.0%)
feedparser8/9 (88.9%)
dataset9/9 (100.0%)
bn-fit9/9 (100.0%)
db-wal2/9 (22.2%)

Assessing technical trajectories can require domain expertise, especially when deciding which steps in a mathematical derivation make meaningful progress. AI assistants helped interpret the trajectory evidence and provided preliminary assessments for all eight cases. The human reviewer checked the AI assessments against the trajectory evidence and used their own judgment to assign the final scores. So the final rubric scores might differ from the AI scores in three cases (Appendix B). Thus, we treat this non-blind review as groundtruth calibration evidence.

A human reviewer scored six additional trajectories blind to model scores and rationales. GLM-5.3/Codex six-repeat means were within one point of 25/30 human ratings. See our response to Reviewer 9e4t (W2) for the per-task scores.

AuthorsOpenReview ↗

Responses to Reviewer pQFN (Part 2)

W4

Harness configuration, tools, and network access: Please refer to Q3.

Additional judges: We evaluated GPT-6.1 Sol, Opus 5, and Gemini 3.8 Flash on the same eight-task O/N/D panel, with six repeated judgments per sample (432/432 scores available).

GPT-6.1 Sol / Codex

Task (dimension)OriginalNullDegraded
Math relations (Eff)3.00(0.00)3.00(0.00)3.33(0.52)
Math series (RG)3.00(0.00)3.00(0.00)3.00(0.00)
Calendly (TU)8.00(0.00)9.33(1.03)7.00(0.00)
HelpScout (RG)4.33(0.82)4.33(0.82)4.00(0.00)
feedparser (Eff)7.00(0.00)7.00(0.00)7.00(0.00)
dataset (IF)10.00(0.00)10.00(0.00)6.00(0.00)
bn-fit (TU)7.00(0.00)7.00(0.00)7.00(0.00)
db-wal (ER)7.33(1.75)7.33(0.82)5.67(1.37)

Opus 5 / Claude Code

Task (dimension)OriginalNullDegraded
Math relations (Eff)3.50(0.55)3.67(0.52)4.00(0.00)†
Math series (RG)3.50(0.84)3.50(0.55)†3.83(1.17)
Calendly (TU)9.00(0.00)9.00(0.00)7.33(0.52)†
HelpScout (RG)7.83(0.75)†8.00(0.63)†6.50(0.55)†
feedparser (Eff)8.00(0.00)†7.83(0.41)†7.83(0.41)†
dataset (IF)9.00(0.63)†8.67(0.52)5.17(1.17)
bn-fit (TU)8.67(0.52)8.00(0.00)†7.50(0.55)
db-wal (ER)3.17(0.41)†3.33(0.52)†3.00(0.63)†

Gemini 3.8 Flash / Codex

Task (dimension)OriginalNullDegraded
Math relations (Eff)4.00(0.00)4.00(0.00)4.00(0.00)
Math series (RG)3.00(0.00)3.00(0.00)3.00(0.00)
Calendly (TU)10.00(0.00)10.00(0.00)7.00(0.00)
HelpScout (RG)9.83(0.41)9.33(1.63)7.83(0.98)
feedparser (Eff)7.00(0.00)7.17(0.41)7.33(0.52)
dataset (IF)9.50(0.84)10.00(0.00)7.00(0.00)
bn-fit (TU)7.83(1.33)8.00(1.55)7.00(0.00)
db-wal (ER)1.50(1.97)2.50(1.64)†2.50(1.76)

Cells report mean (sample SD) on the Figure 3 target dimension. O/N/D = original/null/degraded. † includes offline-recovered scores; original evidence or format issues remain recorded.

Rel = Math relations (Eff); Ser = Math series (RG); Cal = Calendly (TU); Help = HelpScout (RG); Feed = feedparser (Eff); Data = dataset (IF); BN = bn-fit (TU); WAL = db-wal (ER).

W5

Thank you for the suggestions. W2 summarizes our analysis of how judges locate and interpret scoring evidence. We will also cite the controlled-edit figure in the text and add a conclusion section.

Q1

For the GLM-5.3→GLM-5.3-Flash switch at 11.25% measured false-alarm rate, execution statistics (S) detected 37.50%/35.00% of abrupt/gradual changes, and process scores (P) detected 63.75%/48.75%. S∧P detected 21.25%/13.75% at 0% measured false alarms. See our response to Reviewer 9e4t (Q1) for the full detection, AUROC, ablation, and cross-judge results.

Q2

Here we report our per-call pairwise accuracy and uncertainty, which we have also included in the latest version of the paper.

JudgeHarnessTasksDegradation accuracy [95% CI], %Null exact agreement [95% CI], %
DeepSeek V4.1 FlashFull direct857.3 [32.6, 81.2]53.5 [36.1, 69.1]
DeepSeek V4.1 FlashCompressed agentic357.4 [16.7, 100.0]61.1 [33.3, 83.3]
DeepSeek V4.1 FlashCodex CLI863.2 [38.9, 85.1]44.4 [28.8, 63.2]
DeepSeek V4.1 FlashClaude Code860.8 [35.4, 84.7]46.5 [38.5, 55.2]
GLM-5.3Full direct856.9 [30.2, 83.0]48.3 [34.7, 63.9]
GLM-5.3Compressed agentic355.6 [16.7, 83.3]52.8 [25.0, 72.2]
GLM-5.3Codex CLI839.2 [14.2, 66.7]59.0 [40.3, 79.2]
GLM-5.3Claude Code857.6 [29.9, 84.7]59.7 [46.9, 74.7]
Qwen3.8-MaxFull direct842.7 [26.4, 62.5]42.0 [29.9, 60.4]
Qwen3.8-MaxCompressed agentic341.7 [8.3, 66.7]66.7 [50.0, 83.3]
Qwen3.8-MaxCodex CLI858.0 [36.1, 80.2]50.7 [41.7, 59.4]
Qwen3.8-MaxClaude Code866.0 [46.9, 85.8]40.3 [33.7, 49.3]

Each rate compares all 6×66\times6 pairs of single-call scores on the edit's target dimension, then averages across tasks. Degradation accuracy requires sO−sD≥1s_O-s_D\geq1; null agreement requires sO=sNs_O=s_N. Brackets give 95% task-bootstrap confidence intervals (100,000 resamples).

We admit that the pairwise accuracy is not high enough for agent judge. Edits affect only 1–4 locations in a complete trajectory, while repeated agent-judge scores vary. These localized changes and scoring variability can leave the whole-trajectory score unchanged. At the distribution level, pooling 486 target scores per version gives DKL(PD∥PO)=0.455D_{\mathrm{KL}}(P_D\Vert P_O)=0.455 versus DKL(PN∥PO)=0.00466D_{\mathrm{KL}}(P_N\Vert P_O)=0.00466 nats, approximately 98 times larger for degraded edits.

As a tolerance check, allowing ∣sO−sN∣≤1|s_O-s_N|\leq1 raises null agreement to 76.9–88.9% across configurations.

AuthorsOpenReview ↗

Responses to Reviewer pQFN (Part 3)

Q3

Here is the harness setup and LLM inference configuration used in our experiments. We have already included this part in the latest version of our paper.

Table 1 — Harness setup

HarnessAPI interface formatTrajectory inputAvailable toolsWeb access
Full directChat Completions APIFull record in the promptNoneDisabled
Compressed agenticChat Completions APICompressed record with retrievable fragmentsA self-designed tool which can retrieve trajectory fragmentsDisabled
Codex CLI (0.153.4)Responses APIComplete, read-only fileLocal file inspection, search, and shellDisabled
Claude Code (2.1.251)Messages APIComplete, read-only fileLocal file inspection, search, and shellDisabled

Table 2 — LLM inference configuration

JudgeReasoning effortTemperatureOutput-token limit per request
DeepSeek V4.1 Flashhigh065,536
GLM-5.3max065,536
Qwen3.8-Maxxhigh065,536

top_p and top_k use provider defaults; no fixed request seed is supplied. CLI judging allows up to 64 model calls and 262,144 cumulative output tokens per judgment.

Q4

We examined the agent-generated edits and confirmed that all 16 reviewed variants contained the requested types of local changes. These edits are small and may have limited impact on whole-trajectory scores.

Q5

MAST and AdaMAST categorize agent failures. Our five dimensions assess observable task execution and are closer to TRAIL's trajectory-level rubric. Instruction following aligns with instruction adherence. Tool use and efficiency cover aspects of TRAIL's reliability, and efficiency and error recovery cover aspects of TRAIL's plan optimality. Reasoning grounding relates to hallucination and tool-output misinterpretation in TRAIL's error taxonomy. A dedicated security assessment is outside the scope of our rubric.

We score each dimension against its own behavioral criteria, and we believe most dimensions are clearly separate except for error recovery and efficiency. Error recovery assesses how the agent responds to specific failures, while efficiency assesses whether work advances the task throughout the trajectory given the information available at each step. An agent can recover effectively in local errors or failures and still do unnecessary work over the whole trajectory. For more details, please see Appendix C.2 for the rubric boundaries.

Q6

AEGIS’s response corruption modifies an agent’s outputs during task execution, including actions expressed as tool calls. Actually, we considered this approach when designing our controlled-edit protocol. Even with a shared prefix, the first intervention can change the tool results and decisions that follow, making the effect of the intended degradation harder to isolate. This is why we chose localized edits to completed trajectories for the sensitivity study in Section 4.1.

We added a runtime extension that applies a predefined edit before the harness processes a solver response and then lets execution continue. In a pilot on the same eight tasks and target dimensions as Section 4.1, we scored each control and intervention trajectory. Scores range from 0 to 10.

Task (GLM-5.3 × Codex, 6 repeats)TargetControl mean (SD)Runtime edit mean (SD)
Math relationsEff4.67 (0.82)4.50 (0.55)
Math seriesRG3.00 (0.00)3.00 (0.00)
CalendlyTU5.83 (1.83)6.00 (3.46)
HelpScoutRG6.00 (1.10)7.33 (1.97)
feedparserEff7.50 (0.55)7.00 (0.00)
datasetIF7.67 (1.21)7.00 (0.00)
bn-fitTU7.33 (0.82)6.83 (0.41)
db-walER3.83 (0.41)3.17 (0.98)

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Note on Rebuttal Timing and Additional Experiments

Thank you for your patience. Running and analyzing the additional experiments in response to the reviews took longer than we expected. Our replies report the new comparisons at matched false-alarm rates, evaluations with additional judges, and blinded human ratings for six additional trajectories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.