- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that Agent-as-a-Judge methods treat evaluation as reading: either the whole trajectory is fed to one model, or the spans most similar to each criterion are retrieved. This fails for absence claims, buried evidence, and unrecorded state changes.
The proposed framework, ATTEST, has three parts:
- A Criterion Typer classifies each criterion as existential, universal-prohibition, or temporal.
- A Budgeted Investigator gathers evidence through the channel matching that type: targeted retrieval, coverage scans, or read-only environment probes. It allocates a capped action budget using an uncertainty heuristic.
- An Evidence Adjudicator issues SAT/UNSAT/INSUF verdicts based on provenance-ranked, cited evidence.
The authors also introduce EviProbe, a 1,000-instance diagnostic benchmark: 200 base trajectories, each paired with four controlled variants (narration manipulation, distractor insertion, criterion negation, evidence deletion). Criterion-level evidence annotations enable a “grounded accuracy” metric. ATTEST is validated on AJ-Bench, RuVerBench, and EviProbe with DeepSeek and GLM backbones, against an LLM judge, a per-criterion judge, and an agentic judge baseline.
Critical strengths
- Principled framing of evidence requirements. The observation that a criterion’s logical form determines what evidence can settle it is clear and well motivated. One witness suffices for an existential claim, while absence requires coverage. Abstention and provenance checks follow naturally from it. This matters because it gives the community a vocabulary for why judges fail, beyond final accuracy. The case studies (e.g., resisting a fabricated confession in narration) show the failure modes are real.
- EviProbe’s diagnostic design. Pairing each instance with minimally edited counterfactuals and evidence-slot annotations lets evaluators separate “correct” from “correctly supported.” It also tests abstention when evidence is deleted. This is consequential because it could outlast the method: it targets a gap that existing judge benchmarks do not measure. The authors are also quite candid about its limitations.
Critical weaknesses
- The headline claims are not supported by the results. The abstract and introduction claim consistent improvement under matched interaction budgets, however, in Table 1, ATTEST underperforms the agentic baseline on AJ-Bench with DeepSeek (48.5 vs 56.9), including PostgreSQL (49.4 vs 67.2, line 419). RuVerBench margins are within a point without significance testing, and with GLM ATTEST ranks second on the authors’ own EviProbe. Also, ATTEST uses roughly 3× the agentic judge’s tokens (156.6K vs 52.2K, line 457). Furthermore, Figure 1b averages metrics that Section 6.1 says should not be averaged. The paper’s core empirical message is not borne out by its own evidence.
- The central thesis is untested. There are no ablations of typing, individual channels, the scheduler, or the provenance gate. The global trajectory view gives ATTEST much of the same input as the full-trace baselines. The only agentic baseline is the authors’ own implementation. As a result, it cannot be determined whether any gain comes from typed acquisition rather than added compute or the overall pipeline. This is consequential because typed acquisition is the paper’s main conceptual claim, and the abstract’s conclusion that acquisition is “the primary lever” is likewise unsupported (oracle gap 0.5, CI [−2.1, 3.7]).
Questions for authors
- Compute-matched comparison. ATTEST uses about 3× the tokens of the agentic judge (156.6K vs 52.2K per instance). Can you report results with matched total tokens or model calls? Either give the agentic baseline an equivalent budget, or constrain ATTEST to about 52K.
- Component ablations. Can you report results with each of the following removed or replaced?
- Typing: all criteria routed to one untyped channel.
- The coverage channel.
- The uncertainty scheduler: uniform allocation instead.
- The provenance gate.
- The global trajectory view.
Without it, the claim that typed acquisition drives the gains is untested.
- Agentic baseline specification. Is the “Agent-as-a-Judge” baseline an implementation of Zhuge et al. (2024) or a custom agent? I’d also like to know which tools it has: does it have access to the same filesystem and PostgreSQL connectors and reconstructed environments as ATTEST? Can you add at least one published agentic judge? If ATTEST is weaker than published methods, or has different tool access, the comparisons are harder to interpret.
- PostgreSQL underperformance. ATTEST scores 49.4 F1 on PostgreSQL vs 67.2 for the agentic judge, despite having a dedicated probe channel. What causes this? For example, probe failure rates, replay fidelity, query errors, or misclassified criteria. This bears directly on whether environment probing works as claimed.