- Rating
- 1
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper presents a new benchmark for judging mid-trajectory interventions. The authors are particularly interested in which choices preserve final consequences and outcomes of a trajectory. The goal of the benchmark is to provide a controlled test of supervision during long-horizon agentic work. The benchmark is comprised of 70 checkpoints from AppWorld and is balanced across domains and intervention types.
The paper is often incomprehensible and fails to define or justify many of the experimental settings. Moreover, the benchmark itself is not especially novel and the lack of readability and context behind the choices in the paper make the results hard to trust, as well as to interpret meaningfully. Therefore, the paper does not have any strong scientific findings, although it does attempt to address a real gap in the literature.
Critical strengths
- The benchmark is balanced, includes controls, and budget matched, reflecting good experimental hygiene.
- A wide range of models are tested, including both open weight and closed models.
Critical weaknesses
- Issues with writing; particularly the use of terms that are not defined (seven actions, “Outcomes”, scenario-seed realizations, “Expanded”, “D/S/P strata”). I don’t know what these are, and for those that have explanations later, the explanations are vague, confusing, or non-explanations.
- The terminology is mixed between judges and supervisors, which muddles the presentation. There is of course overlap, but the distinction is not made as clear as it could be.
- It’s not clear what “State” really is in Table 1.
- The paper does not explain what source-sibling and pilot-exposes cases mean.
- Table 4 and 5 don’t show uncertainty, so we cannot conclude that any of these differences are necessarily significant.
- The discussion on why action accuracy (AA) and task success (TS) are the most interesting metrics is insufficient, and leaves out many of the interesting questions about how different models might achieve success differently under the different inventions.
Questions for authors
Recommendations:
- An example from the benchmark would be more useful than the current figures.
- Rewrite the paper using less jargon, more straightforward explanations, and more cohesive organization. Section 3 mostly does not justify the choices, only stating what was done.
Nitpicks:
- The sentence "Closest work uses same-prefix branching to study intervention advantage” is not grammatically correct (what is the closest work? Zhang et al., 2026a? And why?)
Thank you for the questions. The benchmark asks: which intervention best preserves completion at an unfinished checkpoint?
1. Protocol and definitions.
Protocol: freeze a checkpoint → run matched interventions → save outcomes → obtain a judge's pre-outcome choice → score by lookup. The LLM judge answers one prompt without tools or exploration. The executor constructs the table by applying interventions and continuing tasks; the evaluator scores choices. Supervision is the broader process of monitoring and intervention.
The seven actions have distinct operational effects:
A scenario–seed realization fixes scenario variables and a random seed; an outcome is one binary completion result. The 70 cases × 7 actions × 5 realizations produce 2,450 outcomes and 490 mean action values, while each judge makes 70 decisions per interface.
Within each realization, candidates share the starting state, executor, permissions, budget, continuation contract and checker. Matching and equal budgets make branches comparable; frozen outcomes separate judgment from execution variability. Five realizations provide repeated observations, and deleting each in turn checks reference stability.
Original provides full history and concise descriptions. Declared specifies targets and effects without changing actions, outcomes or reference. Expanded also adds alternatives in selected cases and recomputes the reference, so it does not isolate menu size.
Provenance: source-disjoint (D), 42 cases from 21 source tasks held out by source; source-sibling (S), 14 new checkpoints from seven source tasks used in benchmark development; pilot-exposed (P), 14 pilot cases revalidated under the final pipeline. Each source contributes two cases; history/position controls use D. Table 1's State means environment/workspace evidence beyond the trace, not the hidden execution snapshot.
2. Example: why TS and AA both matter.
A calendar update sends an entry code to staff event S17. Original says “Revise the pending request”; Declared specifies redirecting the payload to public event P06. Staff-only authorization occurs at event 3 of 36 and is omitted by Recent-32. The action itself is unchanged.
Continue, Advise and Remedy succeed in 5/5 realizations, averaging 3, 4 and 5 interactions; Revise fails in all five. Continue is the reference because completion is maximized first and lower interaction cost breaks ties.
TS credits retained completion value; AA credits exact reference selection, including the cost tie-break. Advise preserves full TS but gets zero AA; Revise loses both. AA alone treats these choices identically; TS alone misses avoidable cost. Regret measures lost completion value; Invalid Rate identifies invalid outputs.
3. Consequential errors versus reference mismatches.
GPT-5.5's Original choices comprise 38 reference matches, seven equal-value alternatives, 23 lower-value choices and two invalid responses. Under Declared, 68/70 match the reference; TS rises from 69.14% to 98.29% against the unchanged table. The seven equal-value corrections improve only AA; invalid and lower-value corrections account for the TS gain. Aggregate TS–AA gaps cannot recover this decomposition.
On ten Continue-reference cases, Original→Declared hits are 2→4 for Qwen3-Coder, 4→9 for GPT-5.5 and 5→7 for Opus 4.6: distinct observed selection behavior on the same cases.
4. Uncertainty in Tables 4–5.
The three-model mean Middle–Far differences are 3.65 percentage points for TS and 9.52 for AA. Each model gains a net four reference matches among 42 cases, not necessarily four changed decisions. Without paired intervals, these descriptive estimates cannot establish significance or reject a monotonic account. Relocation changes ordering and neighboring context, preventing causal attribution to distance. Our choice-level findings do not depend on position significance.
5. Novelty and related-work wording.
Zhang et al. (2026a) studies intervention value and learned routing. LongJudge-Bench evaluates LLM intervention judgment through shared executable outcomes and controlled comparisons of action descriptions and historical evidence.
If these clarifications and results address your concerns, we would be grateful if you could reconsider your assessment and score. Otherwise, we would welcome specific feedback on the remaining concerns and what would help resolve them.