acceptodds
Under review as a conference paper at ICLR 2027

TAFER: Auditing Selective Repair Under Evidence Shift

Abstract

Language-model outputs can match task targets without being supported by the available evidence. We introduce TAFER, a framework for auditing selective repair under evidence shift. TAFER separates answer-text correctness, answer-or-abstain decisions, and support assessments tied to explicit reference rules. In public fixed-pool comparisons, its deterministic selector chooses the lowest recorded-cost candidate that passes structural citation checks, or abstains when none passes. Shared-pool baselines distinguish feasibility filtering from ranking, while matched evidence interventions examine how removing cited or uncited evidence changes answering behavior. Finite-pool accounting separates candidate unavailability, certificate exclusion, and selection error; risk-coverage analysis makes abstention costs explicit. Results are heterogeneous: changes in a support proxy do not alone establish better task utility, and filtering gains need not imply ranking gains. Behavioral diagnostics distinguish increased abstention from continued answering after evidence removal. Natural-language intervention support uses operational verifier judgments, while candidate-pool support uses a gold citation-ID relation. Neither is treated as human support truth. These analyses characterize selective repair within tested protocols without conflating correctness, support, and abstention.

Primary area:
datasets and benchmarks
Keywords:
evidence grounding, selective prediction, abstention, evidence shift, citation verification, risk-coverage, repairability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Utj6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • Problem: an LLM answer can be "correct" by the task label but not actually supported by the evidence it was shown (the paper calls this a "false-green" answer). Accuracy alone hides this.
  • Proposal: TAFER, an auditing framework. It separates three things: is the answer text correct (A), does the model answer or abstain (B), and is the answer supported by the evidence under a stated rule (S). It defines a false-green risk, splits it into coverage x selective risk (Eq. 6), and gives a bookkeeping identity (Eq. 10) that splits failure into "no good candidate in the pool", "good candidate rejected by the certificate", and "selector picked the wrong one".
  • The method part: a deterministic selector that, from a fixed pool of K candidate answers, picks the lowest "recorded cost" candidate that passes a structural check (valid schema, cited IDs exist in the visible evidence), and abstains otherwise. The check is explicitly NOT an entailment check.
  • Validation: (a) a controlled synthetic panel with Qwen2.5-7B (600 base examples x 7 perturbation families = 4,200 rows); (b) frozen public pools on HotpotQA, FEVER, SciFact, 2Wiki and a Mistral HotpotQA run, compared against raw, self-correction, abstention, best-of-K, random-feasible, cost-only and RARR-like selection; (c) "cited vs uncited evidence removal" experiments to see whether models abstain or keep asserting after their cited evidence is deleted; (d) a small rule-based programmatic task.
  • Main results: on the synthetic panel TAFER gets task-policy success 0.8686 vs 0.6240 raw (Table 15), but the authors themselves say this comparison is confounded (see weaknesses). On public data, TAFER vs random-feasible selection is indistinguishable in all four domains (Appendix H and L; all four FGR intervals include zero). On FEVER the FGR drop (0.0977 to 0.0042, Table 1) comes with coverage falling from 0.5655 to 0.4886, and random-feasible selection reaches almost the same point (0.0062 / 0.4886); the retrospective FEVER audit (Section 6) shows that, versus an unconstrained selector, coverage falls by 0.254 and correct-answer yield by 0.162. On Mistral HotpotQA TAFER is slightly worse than raw (+0.0135 FGR). Removing cited evidence mostly triggers abstention on FEVER (+0.561) and both abstention and continued assertion on HotpotQA (+0.172 / +0.148).
  • The historical agreement between the author’s own support labels and one independent annotator is only 84/150 (kappa 0.33), failing the authors’ pre-specified inter-rater gates (Appendix M); the LLM support judges agree with that annotator at 0.726 (Qwen) and 0.836 (DeepSeek) on 73 items (Appendix N), and the paper itself calls natural-language support an uncalibrated proxy (Section 8).
Critical strengths
  • Unusual honesty and auditability. The paper refuses to overclaim, reports null and adverse results (Mistral). This is very nice since nothing in the paper appears misleading; the problem is what is left once the caveats are applied...
  • The accounting identity in Eq. 10 (candidate unavailability / certificate exclusion / selection loss) is a clean, reusable way to say where a fixed-pool repair pipeline fails, and Figure 2 is a nice use of it.
Critical weaknesses
  • No positive, fairly measured result survives. Appendix F shows that across 52 same-pool comparisons against raw, best-of-K, random-feasible and cost-only selection, no simultaneous 95% interval excludes zero (only 8 pointwise ones do), and TAFER is tied with random-feasible selection in every domain (Appendix H/L).. The FEVER "gain" is pure filtering plus lower coverage, and against analytic uniform-feasible selection the composite difference is -0.002. The headline synthetic result (0.8686 vs 0.6240) is confounded: the TAFER objective reads the explicit perturbation family and the family-coded row ID seeds candidate construction, which the baselines do not get (Section 4, Appendix A). Worse, Table 2 shows that simply removing the family term raises task success to 1.0000, i.e. the synthetic panel is essentially solved by any feasible answer, so it carries no information about the selector. This is decisive: a method paper with no demonstrated benefit over a trivial baseline, and an audit paper whose central audit is itself confounded.
  • The quantity being audited is not validated. All support scores are either a structural citation-ID subset rule or an LLM judge; the single human comparison gives kappa 0.33 and fails both pre-specified gates (Table 16, Appendix M), and the two ID rules disagree with each other on half of HotpotQA rows (Table 4, agreement 0.495). The paper repeats on nearly every page that none of this is "human support truth", which is correct, but then the claim "TAFER audits selective repair" rests on a measuring instrument that the paper itself shows is unreliable. Fixing this requires a proper blind human study, not a rewrite.
  • No demonstrated benefit over trivial baselines, the central controlled comparison is confounded, and the audited quantity is not validated.
Questions for authors
  1. Is there any single pre-specified comparison in which TAFER beats random-feasible or analytic uniform-feasible selection with an interval that excludes zero? If not, what exactly is the contribution beyond the accounting identity?
  2. Can you run a fully family-blind Stage3 replay (family term removed AND row ID seed decoupled from the perturbation family) with all baselines given identical information, and report it as the primary controlled result? Given Table 2 reaches 1.0000 task success without the family term, is the synthetic panel too easy to discriminate selectors at all?
  3. Your two human raters agree at only kappa 0.33, and the LLM judges were checked against one annotator on just 73 items (Appendix N). What is the evidence that the judge-based support numbers (the support-loss differences in Figure 1 and Section 6) track real support loss rather than judge noise?
  4. Why does the paper need 44 pages when many analyses are explicitly withdrawn, quarantined, unlinked (2Wiki), or lack a verifiable source hash? Which subset has complete provenance, and can the paper be restricted to that subset?
Reviewer_CC9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at a problem I think is worth studying: a model can give the right answer even after the evidence that supports it has been removed. TAFER tests what happens when cited or uncited passages disappear, and then studies how to pick a repair from a fixed pool of candidates. The selector chooses the cheapest candidate that passes its citation checks, or abstains if none passes. The experiments cover HotpotQA, FEVER, SciFact, 2Wiki and controlled perturbations. The analysis of response changes is useful, but I am not convinced the proposed selector contributes much beyond filtering and abstention.

Critical strengths

The paper makes a useful distinction between a model refusing to answer after losing evidence and a model continuing to give an answer that may no longer be justified. FEVER and HotpotQA show different patterns here. Looking only at whether the answer changed would miss this. I also liked that the authors did not hide the unfavorable results. The selectors use shared candidate pools, and the paper reports coverage alongside its false-green measures. In particular, the FEVER analysis shows why a lower reported risk is not necessarily a better repair system if many more answers are withheld.

Critical weaknesses

My main issue is that TAFER does not really check whether a selected answer is supported. Its public-pool certificate checks the format and whether citation IDs are visible. It does not ask whether those passages actually support the claim. The later gold citation-ID relation does not establish entailment either. The authors acknowledge this, but it is still a serious limit for a paper motivated by unsupported answers. The historical human audit has only 84 agreements out of 150 (κ=0.333\kappa=0.333), and those annotations are not judgments of the new selected repairs. I would want to see a blinded check of the actual TAFER outputs, especially where TAFER and another selector disagree. Otherwise, I think much of the result is about passing citation rules rather than improving evidence grounding.

I also do not see a convincing gain from the cost-based selection rule itself. In Table~1, random feasible selection has lower proxy false-green risk on HotpotQA at the same coverage, while SciFact gives the two methods the same risk and coverage. FEVER looks more favorable until the trade-off is examined: against the same-key unconstrained selector, filtering loses about 25.4 percentage points of coverage and 16.2 points of correct-answer yield. Uniform feasible selection also matches or slightly exceeds TAFER on the reported composite measure. The recorded costs are assigned by candidate type, not measured editing effort, so I am not sure why this is the right way to rank repairs. Stage3 does not answer that question. TAFER's objective uses the perturbation family, and the candidate identifiers retain family information. Removing only the explicit family field changes 552 of 4,200 selections and raises reported task-policy success from 0.8686 to 1.0000. The authors call this a diagnostic rather than an equal-information deployment comparison, which seems right to me. It is hard to use it as evidence that the selector will work under a realistic information boundary.

Questions for authors
  1. Can the authors manually judge the evidence support of selected answers on cases where TAFER and random feasible selection choose different repairs? I would especially like to know how often the citation-valid answer is actually justified by its cited text.
  2. What remains of the Stage3 result when both the explicit perturbation-family field and the family-coded candidate identifier are removed, with all methods given the same information? I think this would be a more meaningful test of the selector.
Reviewer_jiH9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents TAFER, a framework for auditing selective repair of language model outputs under changes in available evidence. The framework generates a finite pool of candidate answers, filters them using structural citation checks, and selects the candidate with the lowest predefined cost or abstains when no feasible candidate exists. The authors evaluate TAFER across several question answering and fact verification datasets, comparing it against baseline selection strategies and analyzing how removing cited versus uncited evidence affects model behavior. The study further examines the relationship between answer correctness, evidence support, and abstention, and decomposes repair failures into candidate unavailability, certificate exclusion, and selection errors.

Critical strengths

Accuracy based evaluation can reward answers that are correct but unsupported by the provided evidence. Correctness, support and abstention are cleanly separated in this paper.

Critical weaknesses

The proposed selector is essentially a structural filter that checks output formats and citation IDs, followed by minimization over manually assigned candidate costs. It introduces no learning, semantic verification, or novel repair mechanism. The authors acknowledge that evidence deletion, abstention, and candidate ranking are established techniques, while the only learned selector failed its own evaluation criterion. The claimed contribution is largely an assembly of existing techniques whose added value remains unclear.

TAFER performs worse than random feasible selection on HotpotQA and Mistral HotpotQA, ties on SciFact, and achieves only a marginal improvement on FEVER. Further analytic comparisons fail to establish any consistent advantage from its ranking strategy.

On FEVER, the reduction in the reported False Green Risk proxy comes at the cost of substantially lower answer coverage and fewer correct answers. Uniform random selection among feasible candidates achieves comparable performance, while the results on other datasets are mixed or unfavorable. Overall, the experiments provide little evidence that TAFER improves actual repair quality rather than simply answering less frequently.

The paper is filled with unexplained internal identifiers and experimental labels, such as V87, V82, Stage3, Track B, and AR3, making the narrative difficult to follow. Important methodological details and results are scattered across extensive appendices that often read like internal audit logs, including quarantined runs, revised claims, and correspondence records.

Questions for authors

In the Controlled Stage3 experiments, TAFER uses perturbation family information that is not equivalently available to the baselines. How much of the reported improvement remains when this information is completely removed?

What would a practitioner concretely do differently after reading this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.