acceptodds
Under review as a conference paper at ICLR 2027

Judging by Evidence: Typed Evidence Acquisition for Agent-as-a-Judge

Abstract

Agent-as-a-Judge, which actively interacts with environments and tools to gather verifiable evidence, has emerged as a promising alternative to rule-based verifiers and LLM-as-a-Judge for evaluating long-horizon agents and for providing rewards in agentic RL. Existing methods treat judging as a reading task, either feeding an entire trajectory to a single model or retrieving the spans most similar to each criterion. However, both break down on long trajectories, where decisive evidence is buried among tens of thousands of tokens, absence can never be established by retrieval, and state changes that the trajectory never records remain unverifiable. To address this, we propose **ATTEST**, a framework that recasts Agent-as-a-Judge from a reading task into a budgeted investigation, acquiring evidence separately for each criterion according to its logical type. Specifically, a Criterion Typer (CT) first determines what evidence each criterion requires, and a Budgeted Investigator (BI) acquires it through targeted retrieval for existential criteria, coverage scans that establish absence for universal ones, or active environment testing for stateful ones, allocating the budget by uncertainty. In turn, an Evidence Adjudicator (EA) grounds each verdict in provenance-ranked evidence rather than the agent's own narration, and abstains when evidence is insufficient. To diagnose where judges actually fail, we further construct **EviProbe**, a diagnostic benchmark that annotates the evidence supporting each criterion and pairs every instance with minimally edited counterfactuals, separating evidence-retrieval failures from verdict failures. Across three benchmarks, ATTEST outperforms both LLM-as-a-Judge and prior agentic judges under matched interaction budgets, suggesting that how evidence is acquired, rather than how much a judge reasons, is the primary lever for reliable agentic evaluation. Code is available at: https://anonymous.4open.science/status/Judging-by-Evidence-EF27.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent-as-a-Judge, Agent Evaluation, Evidence Acquisition, Long-Horizon Agents, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CSFsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that Agent-as-a-Judge methods treat evaluation as reading: either the whole trajectory is fed to one model, or the spans most similar to each criterion are retrieved. This fails for absence claims, buried evidence, and unrecorded state changes.

The proposed framework, ATTEST, has three parts:

  • A Criterion Typer classifies each criterion as existential, universal-prohibition, or temporal.
  • A Budgeted Investigator gathers evidence through the channel matching that type: targeted retrieval, coverage scans, or read-only environment probes. It allocates a capped action budget using an uncertainty heuristic.
  • An Evidence Adjudicator issues SAT/UNSAT/INSUF verdicts based on provenance-ranked, cited evidence.

The authors also introduce EviProbe, a 1,000-instance diagnostic benchmark: 200 base trajectories, each paired with four controlled variants (narration manipulation, distractor insertion, criterion negation, evidence deletion). Criterion-level evidence annotations enable a “grounded accuracy” metric. ATTEST is validated on AJ-Bench, RuVerBench, and EviProbe with DeepSeek and GLM backbones, against an LLM judge, a per-criterion judge, and an agentic judge baseline.

Critical strengths
  1. Principled framing of evidence requirements. The observation that a criterion’s logical form determines what evidence can settle it is clear and well motivated. One witness suffices for an existential claim, while absence requires coverage. Abstention and provenance checks follow naturally from it. This matters because it gives the community a vocabulary for why judges fail, beyond final accuracy. The case studies (e.g., resisting a fabricated confession in narration) show the failure modes are real.
  2. EviProbe’s diagnostic design. Pairing each instance with minimally edited counterfactuals and evidence-slot annotations lets evaluators separate “correct” from “correctly supported.” It also tests abstention when evidence is deleted. This is consequential because it could outlast the method: it targets a gap that existing judge benchmarks do not measure. The authors are also quite candid about its limitations.
Critical weaknesses
  1. The headline claims are not supported by the results. The abstract and introduction claim consistent improvement under matched interaction budgets, however, in Table 1, ATTEST underperforms the agentic baseline on AJ-Bench with DeepSeek (48.5 vs 56.9), including PostgreSQL (49.4 vs 67.2, line 419). RuVerBench margins are within a point without significance testing, and with GLM ATTEST ranks second on the authors’ own EviProbe. Also, ATTEST uses roughly 3× the agentic judge’s tokens (156.6K vs 52.2K, line 457). Furthermore, Figure 1b averages metrics that Section 6.1 says should not be averaged. The paper’s core empirical message is not borne out by its own evidence.
  2. The central thesis is untested. There are no ablations of typing, individual channels, the scheduler, or the provenance gate. The global trajectory view gives ATTEST much of the same input as the full-trace baselines. The only agentic baseline is the authors’ own implementation. As a result, it cannot be determined whether any gain comes from typed acquisition rather than added compute or the overall pipeline. This is consequential because typed acquisition is the paper’s main conceptual claim, and the abstract’s conclusion that acquisition is “the primary lever” is likewise unsupported (oracle gap 0.5, CI [−2.1, 3.7]).
Questions for authors
  1. Compute-matched comparison. ATTEST uses about 3× the tokens of the agentic judge (156.6K vs 52.2K per instance). Can you report results with matched total tokens or model calls? Either give the agentic baseline an equivalent budget, or constrain ATTEST to about 52K.
  2. Component ablations. Can you report results with each of the following removed or replaced?
  • Typing: all criteria routed to one untyped channel.
  • The coverage channel.
  • The uncertainty scheduler: uniform allocation instead.
  • The provenance gate.
  • The global trajectory view.
    Without it, the claim that typed acquisition drives the gains is untested.
  1. Agentic baseline specification. Is the “Agent-as-a-Judge” baseline an implementation of Zhuge et al. (2024) or a custom agent? I’d also like to know which tools it has: does it have access to the same filesystem and PostgreSQL connectors and reconstructed environments as ATTEST? Can you add at least one published agentic judge? If ATTEST is weaker than published methods, or has different tool access, the comparisons are harder to interpret.
  2. PostgreSQL underperformance. ATTEST scores 49.4 F1 on PostgreSQL vs 67.2 for the agentic judge, despite having a dedicated probe channel. What causes this? For example, probe failure rates, replay fidelity, query errors, or misclassified criteria. This bears directly on whether environment probing works as claimed.
Reviewer_6jm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Agent-as-a-Judge evaluation of long agent trajectories. It argues that different criteria require different kinds of evidence: a single witness for an occurrence, full coverage for an absence, and direct inspection for an environment state. It proposes ATTEST, a training-free framework with three components:

A Criterion Typer classifies each criterion by logical type. A Budgeted Investigator acquires evidence through targeted retrieval, coverage scans, or read-only environment probes, under an uncertainty-guided action budget. An Evidence Adjudicator issues provenance-gated SAT/UNSAT/INSUF verdicts and abstains when evidence is insufficient.

The paper also introduces EviProbe, a diagnostic benchmark, annotated with criterion-level evidence slots to assess robustness and evidence support beyond verdict accuracy. Experiments on AJ-Bench, RuVerBench, and EviProbe with two LLM backbones report improvements over LLM-as-a-Judge and agentic baselines in several settings. The paper also notes that gains vary across backbones and metrics.

Critical strengths

S1: Clear and well-motivated problem framing

The paper addresses a relevant problem: different criteria require different kinds of evidence (Sec 1, 3.2). The distinction between confirming an event with a single witness, establishing absence through full coverage, and inspecting environment state is clear and practically important. The paper is also careful about what a scan can and cannot establish (Sec 4.2, App A.2). Modeling insufficient evidence explicitly, through a three-way verdict with abstention (Eq. 2), is a reasonable design choice for reliable evaluation.

S2: Diverse evaluation settings and informative perturbations

The evaluation spans two public benchmarks, AJ-Bench and RuVerBench, and the proposed EviProbe. Together they cover search, deep research, coding, data systems, and GUI tasks (Sec 6.1, App B.1). EviProbe’s controlled perturbations probe judge behavior beyond verdict accuracy (Sec 5.2): narration manipulation, distractors, criterion negation, and evidence removal. They already yield useful findings; for example, distractors are cited in 10.5% of eligible cases despite no verdict flips (Sec 6.3).

Critical weaknesses

W1: Limited technical novelty of the heuristic composition

ATTEST combines standard retrieval, filter-based scanning, and read-only probes with a hand-designed budget heuristic (Eq. 9, Table 5), mapping three familiar logical types to predefined channels (Sec 4.1, 4.3). No ablation isolates the benefit of typing, and the global view given to EA confounds acquisition (Sec 4.4, 7). ATTEST also trails the untyped agentic judge where typing should help most: on prohibitions (92.3% vs. 96.2%, Table 7) and on PostgreSQL (49.4 vs. 67.2 F1, Table 6). An untyped-routing ablation and the typer's accuracy are needed (essential).

W2: Validity of EviProbe's evidence-support metric

Grounded checks citations against model-annotated slots (Eq. 10, 13), not evidence sufficiency. Three problems follow:

  • Valid alternative evidence goes uncredited; even the annotators reached only Jaccard 0.64 (App B.2).
  • Slot coverage does not imply entailment (App E).
  • Absence checks are scored by verdict alone (Sec 5.3).

The claimed separation of acquisition from judgment failures is therefore unsupported. A limited human audit of evidence sufficiency is needed (essential); otherwise, the claim should be qualified.

W3: Insufficient comparison with closely related work

SkillTV-Evolve (Han et al., 2026) and Skill-RM (Chen et al., 2026) already explore criterion-guided verification and adaptive selection of evaluation resources. Sec 2 offers only a generic distinction from the former and omits the latter. A detailed methodological comparison and an experimental baseline are needed.

W4: Experimental rigor insufficient for the stated claims

Results come from single runs, with CIs only for DeepSeek on EviProbe (Table 1, Sec 6.3), and rankings flip across backbones. Compute is not matched: 156.6K vs. 52.2K tokens per instance for ATTEST and the agentic judge (Sec 6.4). The abstract's claims of outperforming baselines "under matched interaction budgets" and of acquisition as "the primary lever" are thus unsupported; the gold-evidence gap is only 0.5 [−2.1, 3.7] (Sec 6.4). Aligning these claims with Sec 7 is essential, and compute-matched, repeated runs are desirable.

W5: Anonymized code and data are not accessible

The anonymous repository link could not be accessed at the time of review. Given the many heuristic defaults (Table 5) and EviProbe being a claimed contribution, a working release is needed for verification and reuse.

References

  • Han, Z., et al. (2026). SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution. arXiv:2608.05573
  • Chen, T., et al. (2026). Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill. arXiv:2606.03980
Questions for authors

please see the weakness section

Reviewer_XYusOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ATTEST selects retrieval, coverage scans, or environment probes according to each evaluation criterion’s logical type, combining provenance checks with abstention. The authors also introduce EviProbe, comprising 200 base trajectories and four controlled variants per trajectory, to assess verdict correctness, evidence support, and recognition of insufficient evidence. ATTEST improves performance in some settings, particularly DeepSeek’s insufficiency detection, but the gains are not consistent across backbones, benchmarks, and metrics.

Critical strengths
  1. EviProbe combines evidence annotations, three-way verdicts, and four controlled perturbations to distinguish failures that final accuracy cannot separate. The primary evaluation uses verifier-derived criteria and includes human annotation audits, providing useful quality checks for the diagnostic resource.

  2. DeepSeek’s insufficiency detection shows a supported improvement over per-criterion judging: INSUF F1 increases by 8.2 points, with a paired 95% interval of [2.9, 13.6]. This benefit is directly relevant to reliable evaluation.

Critical weaknesses
  1. The contribution of typed acquisition remains unisolated. Equal action-budget formulas do not imply equal compute, and rubric decomposition changes the number of criteria used in the budget calculation. With DeepSeek, ATTEST averages 156.6K tokens per instance versus 52.2K for the agentic baseline. Without comparisons and component ablations controlling these differences, the gains may reflect additional compute or rubric processing rather than typed acquisition itself.

  2. Broad superiority and error-attribution claims are insufficiently supported. DeepSeek scores 48.5 on AJ-Bench versus 56.9 for the agentic baseline; GLM scores 83.0 on EviProbe versus 85.3. The accuracy gap between the gold-evidence experiment and ATTEST is only 0.5 points, with an interval crossing zero. These findings support narrower, setting-dependent benefits but do not establish evidence acquisition as the main source of improvement.

Questions for authors

Continue from Weakness,

  1. Do existing comparisons hold criteria, actual compute, and global-context access constant to isolate the benefit of typed acquisition?

  2. How were the development and reported evaluation subsets separated? Can existing records provide results specifically for examples unused in tuning?

  3. Do the saved logs distinguish fully exposed, outline-only, and cited evidence? Can these records support the proposed decomposition of acquisition and adjudication failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.