- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether trajectory-level process assessments can complement task outcomes when auditing black-box LLM services. It scores five dimensions with llm judges, compares four evidence-access workflows, and validates responses to targeted and quality-preserving edits. The main scoring study contains eight original trajectories and 1,458 repeated judgments. A separate replay study uses 1,358 trajectories across sixteen tasks to compare outcome, execution-statistic, and process signals under prompt and deployment changes. Scores generally decrease for selected degradations, but evaluator disagreement is substantial, and the monitoring results do not establish improved degradation detection at matched false-alarm rates.
Critical strengths
-
The monitoring analysis reports false alarms, exclusions, reused trajectories, and unsuccessful calibration diagnostics.
-
The controlled-edit protocol is useful.
Critical weaknesses
-
The main benefit of central auditing is still unclear. In Table 2, using local MMD improves the detection of changes in deployment, but it also leads to more false alarms. The disjoint-calibration diagnostic results in local process-only detection and false alarms of 0% by reaching threshold saturation, rather than solving the problem of balancing these two issues.
-
The process measurements have too little independent validation to support the proposed use.
-
4.2 is single-sourced on trajectories (all 1,358 are DeepSeek) so neither the judge configuration nor the source distribution is varied in the monitoring half. It remains unclear whether this work with a different judge?
Questions for authors
-
Can the saved replay results support a comparison of process-containing and process-free detectors at matched held-out false-alarm rates?
-
Were the appended intervention instructions in Appendix E.1 visible to the scoring judge as part of the canonical record? If so, how does the rubric distinguish inefficient behavior from compliance with an explicit requirement to repeat work? If they were removed, please specify the judging input and how applicable instructions were preserved.
3 comments
Responses to Reviewer 9e4t (Part 2)
Q1
We compare V (verifier outcomes), S (execution statistics), P (process scores) and their combinations using the Partial + window CUSUM monitor described in W1. All results use one account (K=1). Vote₂ requires at least two of V/S/P.
The full DeepSeek columns summarize the deployment changes and prompt interventions described in W1. The mini panels test DeepSeek V4.1→V4, GLM-5.3→5.3-Flash and HY4→HY3 configuration switches. Gradual changes increase the share of requests using the changed condition.
AUROC: abrupt / gradual changes.
We use the highest monitoring score from each run to distinguish normal from changed runs. Higher AUROC means better separation.
| Signal | Full DeepSeek: deployment | Full DeepSeek: prompt | Mini: DeepSeek | Mini: GLM | Mini: HY |
|---|---|---|---|---|---|
| V | 0.7363/0.7128 | 0.5972/0.5864 | 0.9533/0.9125 | 0.8275/0.7741 | 0.9502/0.9122 |
| S | 0.8253/0.7979 | 0.5157/0.5100 | 0.9720/0.9544 | 0.7176/0.6831 | 0.9984/0.9920 |
| P | 0.7019/0.6722 | 0.4696/0.4594 | 0.8592/0.7905 | 0.8845/0.8477 | 0.8486/0.8002 |
| V∧S | 0.7386/0.7206 | 0.5610/0.5492 | 0.9411/0.9012 | 0.7406/0.6977 | 0.9478/0.8931 |
| V∨S | 0.7693/0.7554 | 0.5511/0.5333 | 0.9663/0.9299 | 0.7991/0.7234 | 0.9922/0.9750 |
| V∧P | 0.7146/0.6923 | 0.5383/0.5211 | 0.8429/0.7958 | 0.8364/0.7750 | 0.8217/0.7831 |
| V∨P | 0.7040/0.6725 | 0.5285/0.5245 | 0.9228/0.8640 | 0.8731/0.8106 | 0.9339/0.8873 |
| S∧P | 0.7274/0.6924 | 0.4898/0.4859 | 0.8644/0.8091 | 0.7682/0.7329 | 0.8306/0.7981 |
| S∨P | 0.7833/0.7720 | 0.5094/0.5054 | 0.9513/0.9086 | 0.8552/0.8013 | 0.9839/0.9672 |
| V∧S∧P | 0.7303/0.7106 | 0.5448/0.5330 | 0.8827/0.8423 | 0.7877/0.7294 | 0.8405/0.7997 |
| V∨S∨P | 0.7784/0.7621 | 0.5362/0.5264 | 0.9688/0.9317 | 0.8861/0.8241 | 0.9920/0.9725 |
| Vote₂ | 0.7640/0.7353 | 0.5259/0.5085 | 0.9527/0.9037 | 0.8763/0.8088 | 0.9747/0.9342 |
Detection: abrupt / gradual @ measured FPR (%).
A run counts as detected only if the monitor has received a changed request's score before its first alarm. FPR is the percentage of normal runs that raise any alarm. Thresholds for this table were fitted on normal data with a 5% false-alarm target.
| Signal | Full DeepSeek: deployment | Full DeepSeek: prompt | Mini: DeepSeek | Mini: GLM | Mini: HY |
|---|---|---|---|---|---|
| V | 30.00/25.00 @ 11.25 | 9.17/7.92 @ 9.17 | 72.50/55.00 @ 5.00 | 47.50/31.25 @ 13.75 | 82.50/71.25 @ 11.25 |
| S | 52.50/51.67 @ 7.92 | 7.92/8.75 @ 11.25 | 86.25/80.00 @ 11.25 | 37.50/35.00 @ 11.25 | 97.50/93.75 @ 10.00 |
| P | 26.67/19.17 @ 11.25 | 4.17/4.58 @ 9.17 | 36.25/26.25 @ 7.50 | 63.75/48.75 @ 11.25 | 47.50/36.25 @ 8.75 |
| V∧S | 16.25/14.58 @ 0.00 | 0.00/0.00 @ 0.00 | 63.75/46.25 @ 0.00 | 16.25/11.25 @ 1.25 | 81.25/68.75 @ 1.25 |
| V∨S | 53.33/49.58 @ 15.42 | 14.17/10.42 @ 13.33 | 85.00/75.00 @ 12.50 | 58.75/47.50 @ 16.25 | 98.75/96.25 @ 13.75 |
| V∧P | 5.42/4.58 @ 2.50 | 0.42/0.42 @ 1.25 | 28.75/12.50 @ 1.25 | 22.50/15.00 @ 0.00 | 42.50/31.25 @ 2.50 |
| V∨P | 36.67/27.08 @ 13.33 | 10.42/7.50 @ 10.42 | 72.50/57.50 @ 6.25 | 75.00/51.25 @ 18.75 | 82.50/71.25 @ 13.75 |
| S∧P | 15.00/10.83 @ 0.00 | 0.42/0.42 @ 0.00 | 27.50/16.25 @ 1.25 | 21.25/13.75 @ 0.00 | 47.50/35.00 @ 2.50 |
| S∨P | 50.00/48.75 @ 11.67 | 8.33/7.50 @ 10.42 | 87.50/73.75 @ 11.25 | 70.00/57.50 @ 13.75 | 97.50/92.50 @ 11.25 |
| V∧S∧P | 3.33/1.25 @ 0.00 | 0.00/0.00 @ 0.00 | 25.00/11.25 @ 0.00 | 8.75/7.50 @ 0.00 | 42.50/31.25 @ 1.25 |
| V∨S∨P | 50.83/48.33 @ 15.00 | 15.00/11.25 @ 10.83 | 88.75/76.25 @ 10.00 | 77.50/58.75 @ 21.25 | 98.75/92.50 @ 16.25 |
| Vote₂ | 25.83/22.92 @ 2.50 | 0.83/0.83 @ 1.25 | 68.75/51.25 @ 2.50 | 40.00/25.00 @ 1.25 | 85.00/72.50 @ 3.75 |
Detection at matched false-alarm rates.
For the GLM switch, S-only and P-only have the same measured FPR at the two operating points below.
| Operating point | Signal | Realized FPR (%) | Abrupt detection (%) | Ramped detection (%) |
|---|---|---|---|---|
| Low-FPR point | S | 3.75 | 31.25 | 21.25 |
| Low-FPR point | P | 3.75 | 47.50 | 31.25 |
| Main point | S | 11.25 | 37.50 | 35.00 |
| Main point | P | 11.25 | 63.75 | 48.75 |
At 3.75% FPR, P improves abrupt and gradual detection over S by 16.25 and 10.00 percentage points. At 11.25% FPR, the gains are 26.25 and 13.75 points.
P also has the highest single-signal AUROC on the GLM mini panel, while S leads on the DeepSeek and HY mini panels. Prompt interventions remain weakly detected.
Responses to Reviewer 9e4t (Part 3)
Continue on Q1
Cross-judge detection (abrupt / gradual @ measured FPR, %). We also apply Deepseek-v4.1-flash-1017 x Claude Code as a judge on new trajectories. The new experiment shows consistent results, as process scores do have advantage under certain circumstances.
| Signal | GLM switch · GLM/Codex | GLM switch · DeepSeek/Claude Code | HY switch · GLM/Codex | HY switch · DeepSeek/Claude Code |
|---|---|---|---|---|
| V | 47.50/31.25 @ 13.75 | 37.50/30.00 @ 13.75 | 82.50/71.25 @ 11.25 | 72.50/62.50 @ 10.00 |
| S | 37.50/35.00 @ 11.25 | 41.25/37.50 @ 11.25 | 97.50/93.75 @ 10.00 | 95.00/93.75 @ 12.50 |
| P | 63.75/48.75 @ 11.25 | 61.25/55.00 @ 13.75 | 47.50/36.25 @ 8.75 | 31.25/30.00 @ 15.00 |
| V∧S | 16.25/11.25 @ 1.25 | 11.25/10.00 @ 1.25 | 81.25/68.75 @ 1.25 | 68.75/58.75 @ 0.00 |
| V∨S | 58.75/47.50 @ 16.25 | 52.50/43.75 @ 16.25 | 98.75/96.25 @ 13.75 | 96.25/92.50 @ 12.50 |
| V∧P | 22.50/15.00 @ 0.00 | 25.00/18.75 @ 5.00 | 42.50/31.25 @ 2.50 | 31.25/26.25 @ 5.00 |
| V∨P | 75.00/51.25 @ 18.75 | 58.75/53.75 @ 16.25 | 82.50/71.25 @ 13.75 | 67.50/60.00 @ 13.75 |
| S∧P | 21.25/13.75 @ 0.00 | 20.00/17.50 @ 1.25 | 47.50/35.00 @ 2.50 | 30.00/27.50 @ 2.50 |
| S∨P | 70.00/57.50 @ 13.75 | 65.00/60.00 @ 21.25 | 97.50/92.50 @ 11.25 | 93.75/90.00 @ 16.25 |
| V∧S∧P | 8.75/7.50 @ 0.00 | 7.50/6.25 @ 0.00 | 42.50/31.25 @ 1.25 | 30.00/25.00 @ 0.00 |
| V∨S∨P | 77.50/58.75 @ 21.25 | 62.50/56.25 @ 22.50 | 98.75/92.50 @ 16.25 | 96.25/92.50 @ 15.00 |
| Vote₂ | 40.00/25.00 @ 1.25 | 37.50/31.25 @ 7.50 | 85.00/72.50 @ 3.75 | 70.00/61.25 @ 7.50 |
Q2
We re-examined the data-generation and judging pipeline and confirmed that the appended intervention instructions are excluded from the judging input. The judge input, including system prompt, task, agent trajectory and available tools, is preserved according to the harness. The actual form of judge input depends on which harness used, but no intervention instruction will be visible to agent judge. Additionally, intervention instructions will be preserved by controlled-edit construction pipeline, but they will not be passed on to scoring judge.
Responses to Reviewer 9e4t (Part 1)
W1
Online Monitor Experiments: In our original monitoring experiments, both Global and Local used normal trajectories collected in advance for each task. Global pooled these trajectories, while Local built a separate reference for each task.
Disjoint Calibration: Thank you for raising this point. The calibrated threshold was too high for any run to trigger an alarm, yielding 0% detection and 0% false alarms. Separating the calibration data also left each stage with five trajectories per task instead of ten. The experiment therefore cannot determine whether the failure came from separating the data, having fewer calibration samples, or both.
Together, these limitations motivated us to develop a monitor that starts from an initial period of normal account activity. Partial pooling estimates each task's expected scores and usual variation using both its own observations and information from other tasks. Tasks with little history rely more on the shared prior; as their own history grows, it contributes more to the estimate. We compare recent scores with their expected normal values and use CUSUM to accumulate signs of a change. We set the alarm threshold using historical data from normal service.
Single-account results. We replayed the existing DeepSeek trajectories on sixteen tasks, monitoring one account (K=1) with four concurrent requests and introducing a change after an initial normal period. Deployment changes cover model version, serving endpoint and reasoning budget. Prompt interventions cover repeated reasoning, redundant tools and delayed recovery; the two tool-dependent interventions use twelve tasks. Each family averages its three conditions equally.
The original methods use five reference and ten calibration trajectories per task and a ten-score window. The new method starts from 32 normal account scores and checks the most recent twenty scores. The original rows below cover abrupt changes; the new rows cover abrupt and gradual changes.
Entries are detection @ measured false-alarm rate (%), with a 5% calibration target for every method. A run counts as detected only if the monitor has received a changed request's score before its first alarm. A false alarm is any alarm during an entirely normal run. P denotes process scores; Vote₂ requires at least two of verifier outcomes, execution statistics and process scores.
The revised method reduces reliance on precollected task-specific reference data while retaining the ability to detect deployment changes.
W2
Blinded human labels. We added human validation for six additional trajectories, one per task, covering two tasks each from AutomationBench, SWE-smith and Terminal-Bench. The human reviewer assigned scores without seeing the GLM-5.3/Codex or Astra scores and rationales.
Each cell shows Average/Astra/Human: the mean of six GLM-5.3/Codex judgments, the Astra AI assessment, and the human reviewer's score. RG/TU/Eff/ER/IF denote reasoning grounding, tool use, efficiency, error recovery and instruction following.
The monitoring results in Q1 also show that, for the GLM-5.3→GLM-5.3-Flash switch, process scores achieve higher detection than execution statistics alone at the same measured false-alarm rate.
W3
We ran additional experiments on using more LLMs as task solver and judge. And for details please refer to Q1.