- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
- Problem: an LLM answer can be "correct" by the task label but not actually supported by the evidence it was shown (the paper calls this a "false-green" answer). Accuracy alone hides this.
- Proposal: TAFER, an auditing framework. It separates three things: is the answer text correct (A), does the model answer or abstain (B), and is the answer supported by the evidence under a stated rule (S). It defines a false-green risk, splits it into coverage x selective risk (Eq. 6), and gives a bookkeeping identity (Eq. 10) that splits failure into "no good candidate in the pool", "good candidate rejected by the certificate", and "selector picked the wrong one".
- The method part: a deterministic selector that, from a fixed pool of K candidate answers, picks the lowest "recorded cost" candidate that passes a structural check (valid schema, cited IDs exist in the visible evidence), and abstains otherwise. The check is explicitly NOT an entailment check.
- Validation: (a) a controlled synthetic panel with Qwen2.5-7B (600 base examples x 7 perturbation families = 4,200 rows); (b) frozen public pools on HotpotQA, FEVER, SciFact, 2Wiki and a Mistral HotpotQA run, compared against raw, self-correction, abstention, best-of-K, random-feasible, cost-only and RARR-like selection; (c) "cited vs uncited evidence removal" experiments to see whether models abstain or keep asserting after their cited evidence is deleted; (d) a small rule-based programmatic task.
- Main results: on the synthetic panel TAFER gets task-policy success 0.8686 vs 0.6240 raw (Table 15), but the authors themselves say this comparison is confounded (see weaknesses). On public data, TAFER vs random-feasible selection is indistinguishable in all four domains (Appendix H and L; all four FGR intervals include zero). On FEVER the FGR drop (0.0977 to 0.0042, Table 1) comes with coverage falling from 0.5655 to 0.4886, and random-feasible selection reaches almost the same point (0.0062 / 0.4886); the retrospective FEVER audit (Section 6) shows that, versus an unconstrained selector, coverage falls by 0.254 and correct-answer yield by 0.162. On Mistral HotpotQA TAFER is slightly worse than raw (+0.0135 FGR). Removing cited evidence mostly triggers abstention on FEVER (+0.561) and both abstention and continued assertion on HotpotQA (+0.172 / +0.148).
- The historical agreement between the author’s own support labels and one independent annotator is only 84/150 (kappa 0.33), failing the authors’ pre-specified inter-rater gates (Appendix M); the LLM support judges agree with that annotator at 0.726 (Qwen) and 0.836 (DeepSeek) on 73 items (Appendix N), and the paper itself calls natural-language support an uncalibrated proxy (Section 8).
Critical strengths
- Unusual honesty and auditability. The paper refuses to overclaim, reports null and adverse results (Mistral). This is very nice since nothing in the paper appears misleading; the problem is what is left once the caveats are applied...
- The accounting identity in Eq. 10 (candidate unavailability / certificate exclusion / selection loss) is a clean, reusable way to say where a fixed-pool repair pipeline fails, and Figure 2 is a nice use of it.
Critical weaknesses
- No positive, fairly measured result survives. Appendix F shows that across 52 same-pool comparisons against raw, best-of-K, random-feasible and cost-only selection, no simultaneous 95% interval excludes zero (only 8 pointwise ones do), and TAFER is tied with random-feasible selection in every domain (Appendix H/L).. The FEVER "gain" is pure filtering plus lower coverage, and against analytic uniform-feasible selection the composite difference is -0.002. The headline synthetic result (0.8686 vs 0.6240) is confounded: the TAFER objective reads the explicit perturbation family and the family-coded row ID seeds candidate construction, which the baselines do not get (Section 4, Appendix A). Worse, Table 2 shows that simply removing the family term raises task success to 1.0000, i.e. the synthetic panel is essentially solved by any feasible answer, so it carries no information about the selector. This is decisive: a method paper with no demonstrated benefit over a trivial baseline, and an audit paper whose central audit is itself confounded.
- The quantity being audited is not validated. All support scores are either a structural citation-ID subset rule or an LLM judge; the single human comparison gives kappa 0.33 and fails both pre-specified gates (Table 16, Appendix M), and the two ID rules disagree with each other on half of HotpotQA rows (Table 4, agreement 0.495). The paper repeats on nearly every page that none of this is "human support truth", which is correct, but then the claim "TAFER audits selective repair" rests on a measuring instrument that the paper itself shows is unreliable. Fixing this requires a proper blind human study, not a rewrite.
- No demonstrated benefit over trivial baselines, the central controlled comparison is confounded, and the audited quantity is not validated.
Questions for authors
- Is there any single pre-specified comparison in which TAFER beats random-feasible or analytic uniform-feasible selection with an interval that excludes zero? If not, what exactly is the contribution beyond the accounting identity?
- Can you run a fully family-blind Stage3 replay (family term removed AND row ID seed decoupled from the perturbation family) with all baselines given identical information, and report it as the primary controlled result? Given Table 2 reaches 1.0000 task success without the family term, is the synthetic panel too easy to discriminate selectors at all?
- Your two human raters agree at only kappa 0.33, and the LLM judges were checked against one annotator on just 73 items (Appendix N). What is the evidence that the judge-based support numbers (the support-loss differences in Figure 1 and Section 6) track real support loss rather than judge noise?
- Why does the paper need 44 pages when many analyses are explicitly withdrawn, quarantined, unlinked (2Wiki), or lack a verifiable source hash? Which subset has complete provenance, and can the paper be restricted to that subset?