- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EAVer, a method for the factual verification of long-form content using an end-to-end agentic strategy, employs an end-to-end agentic verifier that learns to control the entire response-level verification workflow as a unified policy. EAVer groups semantically related claims, routes tasks—based on confidence levels—to either direct verification or targeted search, and stores retrieved evidence in a compact context memo for reuse across claims. This work considers a two-stage training process involving SFT and DPO.
Critical strengths
- The writing is easy to follow.
- An agentic perspective is highly beneficial for modern work; such perspectives often facilitate scaling up.
- This paper covers three LLMs, which is enough.
Critical weaknesses
This paper suffers from two core weaknesses that lead me to conclude it does not meet the scientific standards required for ICLR.
- The paper's baselines overlook several significant recent works, such as Veri-R1 [1] and Reasoning-CV [2]. Since the paper focuses on comparing GPT-based algorithms, the authors should include these similar baselines in their comparison. An additional minor point regarding the comparison:
- Algorithms like FactScore are evaluated only with the GPT-4.1 backend; considering a wider range of backends—particularly the one used during training—is necessary to clearly demonstrate performance differences.
- The algorithm is with limited novelty and some contributions are already been covered; works such as [1] and [2] have already investigated the use of SFT combined with policy learning. The paper fails to discuss these similar existing works. Furthermore, regarding the paper's primary contribution—reducing the number of search steps—there is no direct objective that directly optimizes this goal.
[3] also mentions "confidence-based pre-verification," where a model uses its internal knowledge and prediction confidence to determine which claims can bypass an external search. Specifically, FASTFACT derives this confidence from the probability of a label token, applying a calibrated threshold. It also incorporates document-level evidence retrieval. This approach overlaps significantly with the "confidence gating" mechanism used in your EAVer model.
I intend to recommend a relatively clear rejection. However, I would respond very positively and proactively if these issues were effectively addressed.
[1] He, Qi, et al. "Veri-r1: Toward precise and faithful claim verification via online reinforcement learning." Findings of the Association for Computational Linguistics: ACL 2026. 2026.
[2] Zheng, Zhi, and Wee Sun Lee. "Reasoning-cv: Fine-tuning powerful reasoning LLMs for knowledge-assisted claim verification." arXiv preprint arXiv:2505.12348 (2025).
[3] Wan, Yingjia, et al. "FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMs." EMNLP (Findings). 2025.
Questions for authors
I have the following questions:
-
Why are these two datasets referred to as "long-form"? In my view, FEVEROUS is not necessarily shorter than them.
-
Are there any technical differences between this paper and Veri-R1?
-
Would integrating advanced agents (such as the "Opus" model mentioned in the paper) with the Codex/Claude harness yield better results?
-
Why was reinforcement learning not considered? Could RL techniques offer performance advantages?
8 comments
(1/2) W2 Novelty, Optimization objective, and Overlap with existing work.
W2 Novelty, Optimization objective, and Overlap with existing work. The algorithm is with limited novelty and some contributions are already been covered; works such as [1] and [2] have already investigated the use of SFT combined with policy learning. The paper fails to discuss these similar existing works. Furthermore, regarding the paper's primary contribution—reducing the number of search steps—there is no direct objective that directly optimizes this goal.
[3] also mentions "confidence-based pre-verification," where a model uses its internal knowledge and prediction confidence to determine which claims can bypass an external search. Specifically, FASTFACT derives this confidence from the probability of a label token, applying a calibrated threshold. It also incorporates document-level evidence retrieval. This approach overlaps significantly with the "confidence gating" mechanism used in your EAVer model.
We think W2 includes four questions: novelty, differences from Veri-R1 and Reasoning-CV, the optimization objective, and the overlap with FASTFACT. We address each of these questions below.
①Novelty
We train EAVer to handle the full verification process for a long response: extracting and grouping claims, deciding when to search, reusing evidence, and judging each claim. We design and construct interactive training trajectories with three key components: claim grouping, confidence-based search gating, and memory-based evidence reuse. These designs are key parts of our contribution and make end-to-end verification of multiple claims more efficient and reliable. Table 3 shows their effectiveness: removing any of the three components lowers Macro-F1 and increases the number of searches.
Existing work such as Veri-R1 and Reasoning-CV focuses on verifying a single claim. Starting from a given claim narrows the decisions the verifier needs to make. EAVer must also decide which claims to extract and how to organize retrieval and share evidence across multiple claims. We explain these differences below.
②Differences with Veri-R1 and Reasoning-CV
Both make valuable contributions to claim verification. EAVer differs from these works in its task formulation, training-data construction, and learning objectives.
-
Veri-R1
-
Task Setting Retrieval, reasoning, evidence selection, and judgment for a single claim;
-
Data GPT-4o-generated verification trajectories for its SFT baseline, used to fine-tune instruction-tuned models through LoRA;
-
Training Policy Online GRPO as the main method, with SFT as a comparison baseline.
-
-
Reasoning-CV
- Task Setting Evidence-based verification of a claim, which can be logically complex but is still treated as a single claim;
- Data GPT-4o-generated reasoning paths conditioned on gold labels, followed by label-consistency filtering;
- Training Policy SFT followed by iterative DPO, using reasoning paths generated under alternative label conditions for the same claim and evidence.
(2/2) W2 Novelty, Optimization objective, and Overlap with existing work.
EAVer differs along three dimensions:
-
Task formulation. EAVer takes a question and a complete generated response, ①extracts its factual claims, ②retrieves evidence, and ③returns an aligned verdict for each claim. Here, long-form refers to a response containing multiple claims that can share entities, context, and evidence requirements. Claim decomposition is itself part of the task, and retrieval decisions must account for opportunities to reuse evidence across claims. Reasoning-CV remains focused on verifying a single complex claim. EAVer instead learns a response-level verification process with multiple aligned decisions.
-
Training-data construction. EAVer uses a privileged teacher to generate executable, multi-turn verification demonstrations under explicit constraints on entity grouping, confidence-conditioned search, and evidence-memo construction. The teacher interacts with a retrieval environment, producing trajectories that demonstrate how to organize multiple claims, acquire evidence selectively, and reuse it across related claims. Retained trajectories undergo structural, label-alignment, tool-use, budget, and leakage checks, supplemented by random human audits; we report a 98% pass rate among inspected trajectories in section 4.1. The contribution lies in the construction and validation of these response-level interaction demonstrations and the verification behaviors they supervise.
-
Training method and objective. EAVer’s SFT supervises assistant actions throughout the response-level interaction, including claim extraction, grouping, confidence assignment, query formulation, memo construction, and individual verdicts. Its subsequent DPO stage constructs controlled preference pairs by correcting selected verdicts within the same trajectory, while preserving claim text, searches, retrieved evidence, and memos. Preference loss is applied specifically to verdict tokens. Unlike preferences between separately generated reasoning paths, this construction isolates local judgment corrections within a fixed interaction context. Actually, we tried GRPO while developing EAVer. However, some structural issues prevented us from successfully applying RL to EAVer. We will discuss these experiments and their observed failure modes in Q4.
③Direct Optimization of Search Count
EAVer promotes search efficiency through confidence-conditioned retrieval and evidence reuse across related claims; its preference objective targets verdict quality. A per-search reward or penalty is a possible additional objective, but search frequency alone does not capture whether retrieval is useful. In some of our preliminary RL configurations, positive tool-use rewards encouraged searches up to the budget limit, whereas search-cost penalties led to severe search suppression and degraded verification. These observations motivated our emphasis on learning when retrieval is needed and how its results can serve multiple claims. Our goal is accurate verification with demand-driven evidence acquisition, rather than minimizing search count in isolation. We will detail the tested reward configurations and their outcomes in Q4.
④ Differences from FASTFACT
Both methods use confidence to reduce unnecessary retrieval. FASTFACT [3] (Section 4.1) skips external verification only when a definite pre-verification label passes a calibrated label-token probability threshold. EAVer explicitly generates a separate high/medium/low confidence field, supervised as part of its SFT trajectories. We use these categories to route entity groups: only an all-high group skips retrieval; any medium- or low-confidence claim triggers a shared search and memo.
FASTFACT also collects full webpages and retrieves relevant evidence chunks (Sections 4.2–4.3). EAVer writes concise memos within the ongoing trajectory for reuse across related claims. Our contribution combines explicit confidence prediction, group-level retrieval, and memo reuse in a trained response-level verification policy.
W1 Missing Recent Baselines and Backend Comparability.
Thank you for highlighting these methods.
① Why the reported Veri-R1 and Reasoning-CV scores are not directly comparable.
Veri-R1 and Reasoning-CV verify a supplied claim, which may contain multiple factual propositions. EAVer instead extracts claims from a complete generated response and returns a verdict for each. Direct comparison therefore requires adapting these methods with a claim-extraction front end and aligning evidence access, label mapping, and the response-level resource budget. Their original claim-level scores cannot be directly compared with our end-to-end results.
We explain more technical differences between EAVer and Veri-R1 in Q2.
② Backend comparability.
We evaluated the baseline pipelines with gemini-3.1-pro-preview and Claude Opus 4.8, our trajectory-synthesis teacher. We kept each pipeline's logic, prompts, retrieval service, and search budget fixed, apart from necessary API-format adaptations. All methods used the same evaluation samples and scoring protocol. The table reports verification quality, extraction quality, and search usage.
EAVer retains the highest Macro-F1 and Extraction F1 on both benchmarks while using the fewest searches.