- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that flat, per-span context attribution cannot tell evidence that is jointly necessary from evidence that is interchangeable. It proposes representing attribution as a monotone CNF (“AND-of-OR”) evidence formula over prompt spans. Under a clause-matched Formula F1, a flat span set read as an AND of singletons is bounded by , which is 0.83 for . The evaluation uses synthetic Easy (four phenomena), Hard (25 spans), and Long-50/100, plus a 500-example Natural set. The Natural set was curated from HoVer/FEVER/SciFact by keeping examples whose GPT-4o-mini behaviour matches BRIDGE ∧ (EV$_1$ ∨ EV$_2$). All runs use a registry-only answering instruction.
Critical strengths
-
Flat scores really cannot express that two spans are substitutes rather than joint requirements. The monotone CNF representation fits this naturally: each clause is a requirement, and its members are interchangeable alternatives. It also needs only text-API access. The flat-output ceiling is simple, but it states exactly why flat outputs lose under a structure-aware metric. This framing is the main reason I think the direction is worth pursuing.
-
Unusually candid diagnostic reporting. The paper includes: a seed-beam oracle audit, an arithmetic split of each gap around the flat ceiling, a negative cross-target result, etc.
Critical weaknesses
-
The benchmarks match the method’s built-in structure, so the gains do not show the value of query-based formula recovery. Every benchmark except Easy uses one gold template: for Hard and Long-50/100, or its relabelling BRIDGE ∧ (EV$_1$ ∨ EV$_2$) for Natural. LEAF is built around exactly this shape. The cartographer’s role vocabulary names it (REDUNDANT_DATA is defined as “an alternative wording of the same fact”), and the first beam element is . Three results suggest the hand-written beam and the LLM prior carry most of the gain, not the query loop: (a) The anchor is decisive. Without it, Easy-Redundant falls to ≈0.67 (Table 7 note). That is below BM25 (0.83) and Direct LLM (0.81), even though the beam still holds other OR candidates. No-anchor results on Hard and Natural are not reported. (b) The stress test stays inside the beam. Table 4 is described as using shapes the anchor does not match. But all three shapes are INITBEAM templates in Appendix G: , , and . No experiment tests a gold structure outside the template list. (c) The Long-100 margin is baseline weakness. LEAF scores exactly the flat ceiling there (0.83; the above-ceiling margin in Table 2 is 0.00). So the whole +0.49 is headroom left by flat baselines that cannot ablate each of 100 spans even once at , while LEAF’s cartographer reads the full context. Disagreement acquisition also has no measurable effect on Easy (Table 7). As it stands, the evidence supports a narrower claim. An LLM prompted with this role vocabulary, plus a matched template beam, recovers this template. That falls short of a general method for recovering Boolean evidence structure.
-
The empirical claims cannot be checked, and several are internally inconsistent. The released code is a re-implementation written from the paper, and the authors state it did not produce the tables. Also withheld: (a) the curated Natural set; (b) the response cache and the judge canonicalisation rules; (c) per-example results, the held-out mask manifest, and the per-budget sweep. So none of the headline numbers can be verified. That would matter less if the reported results were consistent, but several are not: (a) Table 13. Direct LLM never queries the target, so on the Llama-refiltered set it must score the same in the GPT-4o-mini-evaluated and Llama-evaluated rows. Yet best-baseline reads 0.73 vs. 0.62. That implies Direct LLM ≤ 0.62 there and a target-querying flat baseline at 0.73, far above the ≤0.58 flat baselines reach on the original set. Also, with filter and evaluation target matched (Llama/Llama), LEAF scores 0.73. The paper’s explanation of Table 12 predicts a recovery toward 0.84, so this result undercuts it. (b) Appendix E. At over three seeds, LEAF has exactly zero SD at every budget, while Direct LLM (same planner model) varies by ±0.02. With content-hash caching, this suggests cached responses may have been replayed across seeds. (c) Table 6. It is not a strict held-out test, since the mask manifest was not kept. In addition, §5.4 compares LEAF with Flat MAB (0.72) on Hard , but BM25 (0.75) is the best baseline in that column. (d) Point estimates only. Table 5, the shape test, and the continuous-feedback rerun have no intervals.
Questions for authors
-
Zero-query control. What Formula F1 does the MAP-from-prior formula (no target queries, the actual GPT-4o-mini cartographer) reach on Easy, Hard, Long-100 and Natural? If it is close to LEAF, the contribution is the cartographer plus templates rather than the validation loop.
-
Out-of-beam structures. Please report the no-anchor ablation on Hard and Natural. Please also test gold shapes that are not among the INITBEAM templates, such as three disjoint OR clauses or overlapping clauses.
-
Table 13. Please give per-method scores for both Llama-refiltered rows. How can best-baseline differ between them if Direct LLM is target-independent? Why does Llama/Llama not recover toward 0.84?
-
Cache sharing. Were the seven filtering-stage target responses and LEAF’s evaluation queries served from the same content-hash cache? If so, overlapping masks (full prompt, empty prompt, leave-one-out) mean LEAF observes the very responses that defined gold. Does the cache key include temperature and seed?
-
Selection rule. With binary and , one mismatch costs ≈22 nats, far more than . Is LEAF therefore effectively version-space elimination over 16 candidates plus an Occam rule? How are ties between equal-clause formulas broken? If by beam order, the OR-anchor wins whenever observations do not discriminate.
-
Will you release the code that produced the tables, the Natural benchmark with gold, and the response cache? Why was a separate reference implementation written?