- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a verification gap in partially observable agent settings. When the policy chooses which evidence to acquire, a verifier that checks a structured certificate against the acquired ledger certifies record-relative validity, not complete-record correctness (Eq. 2). To study this, the authors build LegalEvidenceGym. It is a synthetic environment with five stylized employment-termination rule programs, 1,500 deterministic episodes, and an exact public engine that maps a ledger to element states. It also includes Evidence-Twin pairs that differ in one source condition.
The paper makes three empirical contributions. First, a static GPT-4o experiment shows a decision–certificate gap (92.8% DA vs. 84.2% PVS). Second, the paper names a pattern “Evidence Collapse”: across 3 seeds × 2 cost-penalized GRPO objectives on Qwen2.5-7B, reward, CV, and PVS rise while DER, ETC, and DA fall. Third, BEATS is a test-time procedure that works as follows:
It generates N=4 trajectories with forced public probe prefixes of depth 0–3. It replays each one without answers. It ranks them with a lexicographic key over recoverability, error counts, evidence completion , and cost. It canonically re-serializes the certificate from the selected ledger.
On 150 held-out episodes × 6 checkpoints, BEATS@4 reaches 92.3% PVS, versus 86.9% for deepest-prefix Max-Probe@4 with identical recovery (+5.44, pair-clustered CI [2.44, 9.00]). It does so at 23.3% lower environment cost, close to an evaluator-only pool oracle (93.0%).
Critical strengths
S1. The conceptual boundary is stated precisely and operationalized well. Eq. 2 cleanly separates record-relative validity from complete-record correctness. The environment exposes each layer (latent world, available sources, acquired ledger, entailed decision, certificate) as a separately measurable object. Evidence-Twins combined with the joint ETC metric extend contrast sets from output sensitivity to acquisition behavior, and they refuse credit for repeating one label across a pair. This is the most reusable artifact in the paper. It gives anyone training tool-using agents against terminal verifiers a controlled test of whether decisions track decision-relevant evidence.
S2. Experimental hygiene and claim calibration are well above typical. The paper does several things right:
The selector is frozen on validation before test. All selectors draw from the same candidate pool and get identical recovery. Bootstrap resampling is clustered by twin pair, so 900 correlated observations are not treated as independent. An evaluator-only oracle measures headroom. Per-seed results are reported. The authors explicitly decline to claim reward hacking.
This makes the numbers trustworthy. It also makes it possible to run the decompositions below from the reported tables, which is to the authors’ credit even where those decompositions cut against the framing.
Critical weaknesses
W1. “Evidence Collapse” is the optimum of the stated objective, and the control that could make it more than that is missing.
Table 1 decomposes the reward gain almost exactly. With :
The cost reduction () is worth . Decision accuracy, the only correctness term, falls by . The net is , consistent with the rise in Fig. 3A.
For Certificate-GRPO, , consistent with Fig. 3C. GRPO is doing what the reward pays for. At , five units of cost are worth more than eight points of decision accuracy, so investigating less is optimal, and lower DER and ETC follow directly. An expert shown the reward function would predict this outcome.
The paper’s thesis is stronger: terminal reward and record-relative verification do not, by themselves, reward adequate investigation. But at the outcome term does reward acquiring decisive evidence, since on twin pairs that is the only way to be right on both. The thesis and the cost-tradeoff account therefore make different predictions. The zero-cost or fixed-cost control that the authors acknowledge is missing (Sec. 4.3) is exactly the experiment that separates them. For Certificate-GRPO in particular, if the validity bonus alone drives smaller ledgers at , that would be genuine support for the thesis. Without this control, or a sweep, the second contribution is a name for the behavior the reward specifies.
The cross-checkpoint Spearman correlations do not close this gap. Reward and DER both trend monotonically with update number, so their rank correlation largely reflects the shared time trend.
W2. BEATS’s gain over the strongest baseline is a validity gain, not an evidence-sufficiency gain, and the method relies on an exact public engine that the named application domains lack.
The +49 PVS gain is smaller than the +55 gain in valid outputs. Conditional on a valid certificate, decision accuracy is 95.3% for BEATS versus 95.7% for Max-Probe, and Max-Probe also has higher DER (0.942 vs. 0.885). So the ledgers BEATS selects are not more decision-sufficient. BEATS avoids more trajectories that end invalid.
Canonical serialization yields a valid certificate for any ledger. The residual invalid outputs must therefore come from non-recoverable trajectories (budget, horizon, action, or parse failures). Deep prefixes make these more likely under the 12-unit evaluation budget. As this account predicts, the gap is largest in the high-cost stratum (0.773 vs. 0.640). The leading terms of ( and the error counts) capture this without or cost. On the evidence shown, the PVS gain does not support the framing that the relevant selection object is the information state. The deciding criterion is terminal recoverability.
Separately, the pipeline leans on throughout:
Recovery submits . is read off the engine. A single sample with recovery already reaches 0.784 PVS, versus 0.418 for greedy decoding without it.
“Answer-free” is accurate, but it says little when the decision function is public and exact. In medical diagnosis, auditing, or web search, which the conclusion names, estimating element resolution without such an engine is the hard problem (cf. SeekBench, S2G-RAG), and BEATS assumes it away.
Finally, the 23.3% cost reduction is measured against the selector that, by construction, picks the most expensive prefix. Self-Consistency (5.20) and Certificate-only (4.92) are both cheaper than BEATS (5.42), and N=4 uses about 2× the model tokens of greedy decoding.
Questions for authors
Q1 (could change my assessment). Please report a “deepest recoverable prefix” baseline. It would filter candidates by the first four terms of ( and the three error counts), then take the deepest remaining prefix. Please also report a leave-one-term-out ablation of . If this baseline reaches ≈0.92 PVS, then and cost contribute savings but not correctness, and the contribution should be reframed accordingly.
Q2 (clarification). Please break down the invalid outputs (≈83 for Max-Probe, 28 for BEATS) by cause: budget exhaustion, horizon, invalid action, or parse error. Please also report BEATS vs. Max-Probe at 20- and 30-unit evaluation budgets. Does the gap survive when the budget no longer truncates deep prefixes?
Q3 (could change my assessment). Please evaluate a scripted, non-LLM acquisition baseline. It would acquire public sources in cost order until resolves every element or fixes the burden-based decision, then submit the canonical certificate. What PVS and cost does it reach? Recovery already computes the decision from the ledger, so this isolates what the LLM contributes.
Q4 (could change my assessment). Please run one (or fixed-cost) GRPO run per objective, even single-seed, and ideally a small sweep. Does DER still decline? If it does, Evidence Collapse is a finding about terminal verification. If it does not, it is a cost tradeoff and is better presented as a DER-vs- curve.
Q5 (clarification). Outcome-GRPO does not reward validity, yet CV rises by 0.28. Is this mechanical? Smaller ledgers leave more elements UNRESOLVED and create fewer opportunities to overclaim. If so, CV↑ and DER↓ are one effect rather than two co-moving signals. Please report CV conditioned on ledger size.
Q6 (clarification). How many key orderings and N values were tried on validation before the freeze? Please also report DA for every row of Table 2.
Suggestions (do not affect my score):
The strongest version of this paper would lead with the environment and twin diagnostic, and turn the cost tradeoff into a predictive DER-vs- law. Sec. 4.2 is a weak fit. It uses a single unrecorded GPT-4o snapshot, and it tests a different failure mode (overclaiming on a fixed record, not record construction). Consider moving it to the appendix. The Gemma replication belongs in the main text. The final paragraph of the conclusion prescribes training-time fixes that the paper does not test.