acceptodds
Under review as a conference paper at ICLR 2027

EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy

14 views

Abstract

Long-form factuality verification is commonly implemented as a static decompose-search-verify pipeline, with separately prompted modules processing claims and invoking external search. Treating claims independently makes LLM and search calls scale with claim count and causes repeated searches for overlapping evidence about related claims. We introduce EAVer, an End-to-end Agentic Verifier that learns to control the complete response-level verification workflow as a unified policy. EAVer groups semantically related claims, routes each group to direct verification or targeted search based on confidence, and keeps evidence returned by search in compact in-context memos for cross-claim reuse. To train this policy, we develop a privileged-teacher synthesis pipeline that converts gold claim annotations into executable multi-turn tool-interaction trajectories with live search rather than post-hoc rationales. Structural, label-alignment, tool-use, search-budget, and leakage checks yield 1,447 quality-controlled trajectories. We further construct 794 bidirectional same-trajectory preference pairs that keep claim grouping, search, and evidence fixed, enabling decision-focused Direct Preference Optimization (DPO) over factuality-decision tokens. The results with Qwen3-8B show that EAVer outperforms the strongest search-based baseline on each benchmark by 2.88 Macro-F1 points on VeriFastScore and 4.73 points on the out-of-distribution FaStFact-Bench, while using about 80% fewer searches than the most search-efficient baseline. Moreover, EAVer consistently improves performance across models ranging from 4B to 32B parameters, demonstrating its strong generalizability.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Agent system, Data synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_q6hCOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EAVer, a method for the factual verification of long-form content using an end-to-end agentic strategy, employs an end-to-end agentic verifier that learns to control the entire response-level verification workflow as a unified policy. EAVer groups semantically related claims, routes tasks—based on confidence levels—to either direct verification or targeted search, and stores retrieved evidence in a compact context memo for reuse across claims. This work considers a two-stage training process involving SFT and DPO.

Critical strengths
  1. The writing is easy to follow.
  2. An agentic perspective is highly beneficial for modern work; such perspectives often facilitate scaling up.
  3. This paper covers three LLMs, which is enough.
Critical weaknesses

This paper suffers from two core weaknesses that lead me to conclude it does not meet the scientific standards required for ICLR.

  1. The paper's baselines overlook several significant recent works, such as Veri-R1 [1] and Reasoning-CV [2]. Since the paper focuses on comparing GPT-based algorithms, the authors should include these similar baselines in their comparison. An additional minor point regarding the comparison:
  • Algorithms like FactScore are evaluated only with the GPT-4.1 backend; considering a wider range of backends—particularly the one used during training—is necessary to clearly demonstrate performance differences.
  1. The algorithm is with limited novelty and some contributions are already been covered; works such as [1] and [2] have already investigated the use of SFT combined with policy learning. The paper fails to discuss these similar existing works. Furthermore, regarding the paper's primary contribution—reducing the number of search steps—there is no direct objective that directly optimizes this goal.

[3] also mentions "confidence-based pre-verification," where a model uses its internal knowledge and prediction confidence to determine which claims can bypass an external search. Specifically, FASTFACT derives this confidence from the probability of a label token, applying a calibrated threshold. It also incorporates document-level evidence retrieval. This approach overlaps significantly with the "confidence gating" mechanism used in your EAVer model.

I intend to recommend a relatively clear rejection. However, I would respond very positively and proactively if these issues were effectively addressed.

[1] He, Qi, et al. "Veri-r1: Toward precise and faithful claim verification via online reinforcement learning." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[2] Zheng, Zhi, and Wee Sun Lee. "Reasoning-cv: Fine-tuning powerful reasoning LLMs for knowledge-assisted claim verification." arXiv preprint arXiv:2505.12348 (2025).

[3] Wan, Yingjia, et al. "FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMs." EMNLP (Findings). 2025.

Questions for authors

I have the following questions:

  1. Why are these two datasets referred to as "long-form"? In my view, FEVEROUS is not necessarily shorter than them.

  2. Are there any technical differences between this paper and Veri-R1?

  3. Would integrating advanced agents (such as the "Opus" model mentioned in the paper) with the Codex/Claude harness yield better results?

  4. Why was reinforcement learning not considered? Could RL techniques offer performance advantages?

8 comments
AuthorsOpenReview ↗

W1 Missing Recent Baselines and Backend Comparability.

W1 Missing Recent Baselines and Backend Comparability. The paper's baselines overlook several significant recent works, such as Veri-R1 [1] and Reasoning-CV [2]. Since the paper focuses on comparing GPT-based algorithms, the authors should include these similar baselines in their comparison. An additional minor point regarding the comparison:

  • Algorithms like FactScore are evaluated only with the GPT-4.1 backend; considering a wider range of backends—particularly the one used during training—is necessary to clearly demonstrate performance differences.

Thank you for highlighting these methods.

① Why the reported Veri-R1 and Reasoning-CV scores are not directly comparable.

Veri-R1 and Reasoning-CV verify a supplied claim, which may contain multiple factual propositions. EAVer instead extracts claims from a complete generated response and returns a verdict for each. Direct comparison therefore requires adapting these methods with a claim-extraction front end and aligning evidence access, label mapping, and the response-level resource budget. Their original claim-level scores cannot be directly compared with our end-to-end results.

ComparisonVeri-R1Reasoning-CVEAVer
Input unitA supplied claim, which may be complexA supplied complex claimA question and a complete generated response
Required outputAn overall verdict and selected evidence for the claimAn overall verdict for the claimExtracted claims and an aligned verdict for each claim
Main learning focusEvidence retrieval, reasoning, and judgmentEvidence-based verification reasoningResponse-level claim organization, retrieval, evidence reuse, and judgment

We explain more technical differences between EAVer and Veri-R1 in Q2.

② Backend comparability.

We evaluated the baseline pipelines with gemini-3.1-pro-preview and Claude Opus 4.8, our trajectory-synthesis teacher. We kept each pipeline's logic, prompts, retrieval service, and search budget fixed, apart from necessary API-format adaptations. All methods used the same evaluation samples and scoring protocol. The table reports verification quality, extraction quality, and search usage.

MethodVFS Macro-F1 ↑VFS FSR ↓VFS Ext. F1 ↑FFB Macro-F1 ↑FFB FSR ↓FFB Ext. F1 ↑Searches/response ↓
FActScore (GPT-4.1)64.8356.5759.4364.5847.3930.9735.88
FActScore (gemini-3.1-pro-preview)69.1048.6067.2070.8043.5062.3034.00
FActScore (Opus)70.8044.2071.4072.9037.4067.1032.80
SAFE (GPT-4.1)63.4064.0158.9262.1065.6056.82106.90
SAFE (gemini-3.1-pro-preview)68.2052.7065.1069.7049.2064.3089.40
SAFE (Opus)71.2046.6070.9072.6039.1068.4082.70
VeriScore (GPT-4.1)63.3855.4777.5360.8950.2546.5819.18
VeriScore (gemini-3.1-pro-preview)68.7049.3078.4069.8043.2061.3018.30
VeriScore (Opus)71.6043.8080.2072.8037.2068.8017.60
FIRE (GPT-4.1)63.4064.0058.9269.3248.1332.0437.69
FIRE (gemini-3.1-pro-preview)70.1048.1068.3072.1041.4062.9031.50
FIRE (Opus)72.4042.3074.6073.4036.5069.7028.40
VeriFastScore70.5538.2881.2564.9936.3246.1015.62
EAVer (Qwen2.5-7B-Instruct)71.7738.5180.5266.0141.2071.824.21
EAVer (Llama-3.1-8B)72.4040.9080.7766.5458.5373.393.78
EAVer (Qwen3-8B)73.4341.5781.4874.0535.8774.243.14

EAVer retains the highest Macro-F1 and Extraction F1 on both benchmarks while using the fewest searches.

AuthorsOpenReview ↗

(1/2) W2 Novelty, Optimization objective, and Overlap with existing work.

W2 Novelty, Optimization objective, and Overlap with existing work. The algorithm is with limited novelty and some contributions are already been covered; works such as [1] and [2] have already investigated the use of SFT combined with policy learning. The paper fails to discuss these similar existing works. Furthermore, regarding the paper's primary contribution—reducing the number of search steps—there is no direct objective that directly optimizes this goal.

[3] also mentions "confidence-based pre-verification," where a model uses its internal knowledge and prediction confidence to determine which claims can bypass an external search. Specifically, FASTFACT derives this confidence from the probability of a label token, applying a calibrated threshold. It also incorporates document-level evidence retrieval. This approach overlaps significantly with the "confidence gating" mechanism used in your EAVer model.

We think W2 includes four questions: novelty, differences from Veri-R1 and Reasoning-CV, the optimization objective, and the overlap with FASTFACT. We address each of these questions below.

①Novelty

We train EAVer to handle the full verification process for a long response: extracting and grouping claims, deciding when to search, reusing evidence, and judging each claim. We design and construct interactive training trajectories with three key components: claim grouping, confidence-based search gating, and memory-based evidence reuse. These designs are key parts of our contribution and make end-to-end verification of multiple claims more efficient and reliable. Table 3 shows their effectiveness: removing any of the three components lowers Macro-F1 and increases the number of searches.

Existing work such as Veri-R1 and Reasoning-CV focuses on verifying a single claim. Starting from a given claim narrows the decisions the verifier needs to make. EAVer must also decide which claims to extract and how to organize retrieval and share evidence across multiple claims. We explain these differences below.

②Differences with Veri-R1 and Reasoning-CV

Both make valuable contributions to claim verification. EAVer differs from these works in its task formulation, training-data construction, and learning objectives.

  • Veri-R1

    • Task Setting Retrieval, reasoning, evidence selection, and judgment for a single claim;

    • Data GPT-4o-generated verification trajectories for its SFT baseline, used to fine-tune instruction-tuned models through LoRA;

    • Training Policy Online GRPO as the main method, with SFT as a comparison baseline.

  • Reasoning-CV

    • Task Setting Evidence-based verification of a claim, which can be logically complex but is still treated as a single claim;
    • Data GPT-4o-generated reasoning paths conditioned on gold labels, followed by label-consistency filtering;
    • Training Policy SFT followed by iterative DPO, using reasoning paths generated under alternative label conditions for the same claim and evidence.
AuthorsOpenReview ↗

(2/2) W2 Novelty, Optimization objective, and Overlap with existing work.

EAVer differs along three dimensions:

  • Task formulation. EAVer takes a question and a complete generated response, ①extracts its factual claims, ②retrieves evidence, and ③returns an aligned verdict for each claim. Here, long-form refers to a response containing multiple claims that can share entities, context, and evidence requirements. Claim decomposition is itself part of the task, and retrieval decisions must account for opportunities to reuse evidence across claims. Reasoning-CV remains focused on verifying a single complex claim. EAVer instead learns a response-level verification process with multiple aligned decisions.

  • Training-data construction. EAVer uses a privileged teacher to generate executable, multi-turn verification demonstrations under explicit constraints on entity grouping, confidence-conditioned search, and evidence-memo construction. The teacher interacts with a retrieval environment, producing trajectories that demonstrate how to organize multiple claims, acquire evidence selectively, and reuse it across related claims. Retained trajectories undergo structural, label-alignment, tool-use, budget, and leakage checks, supplemented by random human audits; we report a 98% pass rate among inspected trajectories in section 4.1. The contribution lies in the construction and validation of these response-level interaction demonstrations and the verification behaviors they supervise.

  • Training method and objective. EAVer’s SFT supervises assistant actions throughout the response-level interaction, including claim extraction, grouping, confidence assignment, query formulation, memo construction, and individual verdicts. Its subsequent DPO stage constructs controlled preference pairs by correcting selected verdicts within the same trajectory, while preserving claim text, searches, retrieved evidence, and memos. Preference loss is applied specifically to verdict tokens. Unlike preferences between separately generated reasoning paths, this construction isolates local judgment corrections within a fixed interaction context. Actually, we tried GRPO while developing EAVer. However, some structural issues prevented us from successfully applying RL to EAVer. We will discuss these experiments and their observed failure modes in Q4.

③Direct Optimization of Search Count

EAVer promotes search efficiency through confidence-conditioned retrieval and evidence reuse across related claims; its preference objective targets verdict quality. A per-search reward or penalty is a possible additional objective, but search frequency alone does not capture whether retrieval is useful. In some of our preliminary RL configurations, positive tool-use rewards encouraged searches up to the budget limit, whereas search-cost penalties led to severe search suppression and degraded verification. These observations motivated our emphasis on learning when retrieval is needed and how its results can serve multiple claims. Our goal is accurate verification with demand-driven evidence acquisition, rather than minimizing search count in isolation. We will detail the tested reward configurations and their outcomes in Q4.

④ Differences from FASTFACT

Both methods use confidence to reduce unnecessary retrieval. FASTFACT [3] (Section 4.1) skips external verification only when a definite pre-verification label passes a calibrated label-token probability threshold. EAVer explicitly generates a separate high/medium/low confidence field, supervised as part of its SFT trajectories. We use these categories to route entity groups: only an all-high group skips retrieval; any medium- or low-confidence claim triggers a shared search and memo.

FASTFACT also collects full webpages and retrieves relevant evidence chunks (Sections 4.2–4.3). EAVer writes concise memos within the ongoing trajectory for reuse across related claims. Our contribution combines explicit confidence prediction, group-level retrieval, and memo reuse in a trained response-level verification policy.

Reviewer_pt71OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EAVer learns a unified policy for fact-checking long-form responses, encompassing claim decomposition, grouping of related claims, selective retrieval, evidence reuse, and final verdict generation. This work generate 1,447 interaction demonstrations from a "teacher" that has access to ground-truth claims and labels, and further propose a decision-token-centric Direct Preference Optimization (DPO) method using 794 preference pairs. EAVer outperforms existing retrieval-based approaches on two benchmarks, achieving higher Macro-F1 scores while requiring fewer retrieval steps. The paper validate their design through ablation studies and by training the model across various base architectures.

Critical strengths
  1. The paper's approach improves both verification performance and retrieval efficiency. On two benchmarks, Qwen3-8B outperforms the best existing retrieval-based method by 2.88 and 4.73 points in Macro-F1, respectively. Furthermore, the average number of retrieval steps across both datasets is reduced from 15.62 the lowest among the comparison methods to 3.14 (Table 2). This offers practical value by lowering the operational burden of fact-checking.

  2. These improvements are not limited to a single model; they are observed across six different backbones trained on the same data. Removing components such as grouping, the confidence gate, or the memo mechanism leads to a decline in performance and an increase in the number of retrieval steps. This confirms the effectiveness of the integrated design.

Critical weaknesses
  1. The comparability between the evaluation inputs and the metrics is not sufficiently clear. While the student input includes num_claims, an approximate count of claims, it is not explained how this value is derived or how it is incorporated into the comparison methodology. Furthermore, since the Macro-F1 metric is calculated only for claims matched via string similarity, methods that yield different extraction results are effectively evaluated against different sets of claims. Although reporting "Extraction F1" separately is appropriate, it remains to be seen whether the performance advantage persists when accounting for both omitted claims and classification errors.

  2. It is difficult to isolate the specific gains contributed by policy learning itself. The branching logic for retrieval, based on confidence levels is explicitly defined as a rule within the student prompt. In the absence of a comparison against an agent (using the same model, retrieval tools, and grouping/memory rules) prior to policy learning, it is unclear what portion of the improvement stems from the policy learning process versus the pre-designed procedural framework. While Table 3 demonstrates the utility of individual components, it does not directly address this issue.

Questions for authors
  1. Clarification of input conditions: How is "num_claims" calculated during evaluation, and is the same method applied to the comparison baselines? Do the main conclusions still hold if this hint is removed?

  2. Clarification of evaluation methodology: How are unmatched gold claims, specifically "UNSUPPORTED" claims handled in the denominator for the False Support Rate (FSR) and in error calculations? Do the primary rankings remain consistent when the evaluation accounts for both omissions and labeling errors?

Reviewer_2NUqOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Eaver, a two-stage framework for training LLMs to verify the factuality of long statements containing multiple claims. The first step is SFT based on trajectories generated by a strong commercial LLM. The second is DPO post-training, which slightly boosts the verdict prediction performance. Furthermore, the paper proposes to revisit the standard decompose-search-verify process: semantically related claims are grouped into clusters. For each cluster, the LLM first decides whether to answer the claim directly when confidence is self-reported as high, reuse available memory when confidence is moderate, or retrieve external evidence when confidence is reported as low. Retrieving external evidence once for all claims in the same cluster reduces the number of retrieval steps compared to prior work. The experiments show that Eaver can be successfully applied to several base LLMs of various sizes and outperforms the baselines on one in-domain and one out-of-domain dataset.

Critical strengths
  • The methodology is based on two relevant criticisms about prior long-form factuality verification baselines: the different steps of the pipeline need to be optimized in a single trajectory, and retrieving external evidence for every claim separately introduces a lot of unneeded redundancy and overhead. I disagree slightly with the claim that "search should be conditioned on confidence". If the scope is an internal factuality checker, then it is fine. However, if the goal is to write a factuality report and show it to a human reader, then I would argue that supporting the verdict with external evidence is a non-optional step, even under high confidence.
Critical weaknesses
  • The evaluation setup for claim extraction appears inappropriate. Matches between extracted and ground-truth claims are determined by a string similarity threshold > 0.5. It is not specified in the main text how this similarity is computed (is it using sentence embedding models?). I find this approach problematic. A claim and its negation, for example, can have high string similarity while having actually opposite verdicts. Similarly, changing the year in a temporal claim will typically not bring the similarity below the 0.5 threshold. Could you discuss this in the rebuttal? A manual analysis, or a combination with an LLM judge, would strengthen the evaluation methodology.

  • The experimental setup is underspecified and needs some important clarifications. I have the following comments and questions that would absolutely need to be answered to trust the results: (1) How is the search with the Google API actually implemented? Does it rely on the Google Custom Search API (which is closing on 01/01/2027 iirc)? (2) Where is the training data coming from? This is not made explicit but I suppose it comes from the Verifastscore train set, (3) Why filter the Verifastscore test set, and how is it done in practice? Is it something that baselines from prior work did too? Examples without valid binary claims are removed, does it mean that the scope of Verifastscore is broader and it is narrowed down here to fit Eaver? The same questions apply to the filtering of FastFactBench, (4) did you verify that Fastfactbench is actually out-of-domain (no overlapping claims or topics, ...)?

Questions for authors
  • I did not understand from reading the paper what determines the order in which the claims are processed after the extraction stage. I am wondering, in particular, how the ordering impacts the groups of claims with confidence Low (L) and Medium (M). L adds new external evidence to the memory, while M uses whatever info is available in the memory so far. Hence, I would assume that the performance in M cases depends heavily on which L cases have been processed previously. Could you comment on this?

  • The SFT data contains confidence scores self-reported by Claude. These are based on Claude's own parametric knowledge, capabilities, and biases. For example, Claude will report confidence High on facts that are within its parametric knowledge. However, not all LLMs have the same extent of parametric knowledge as Claude. I am wondering how much training a weaker LLM to report confidence High on claims that are actually outside its parametric knowledge, thereby missing a much-needed external evidence retrieval step, negatively affects performance. Could you comment on this?

  • Suggestion: including equations for F1, Precision, Recall, and FPR metrics seems completely unneeded. These are very well-known metrics. The corresponding space can be reused to address some of my comments.

  • On line 226, the following checks of the SFT trajectories are listed: "label-alignment, tool-use,search-budget,andprivate-keyleak age checks". These checks are not explained. Could you add explanations or point directly in the text to a relevant appendix?

  • "Label shares cover 19,334 SFT decisions and 15,686 decisions in chosen DPO completions". I could not understand this sentence on page 5 or how it relates to the rest of the paragraph. Could you clarify it? I would also advise against discussing the DPO completions on page 5; wait until lines 276-279 on page 6 to elaborate on them.

  • Could you report Eaver without DPO in Table 2? Figure 4 shows limited improvement in Macro-F1 with DPO, suggesting that most of the improvement comes from SFT. The y-axis in Figure 4 is also too narrow, which visually exaggerates the extent to which the DPO stage improves Macro-F1 performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.