acceptodds
Under review as a conference paper at ICLR 2027

EVIDENCE LEDGERS: AUDITABLE CLAIM-LEVEL GROUNDING FOR VISION–LANGUAGE MODELS

Abstract

Vision–language model answers often contain several factual claims, but standard generation provides no reliable record of the input evidence that supports each claim. We introduce evidence ledgers, which align evidence atoms, typed claims, and answer units through two routes, a direct evidence-to-answer alignment and one composed through the claims, with a null source that absorbs claim mass not assigned to the available evidence. Training combines a claim-level unlikelihood term weighted by the null allocation, a counterfactual swap loss, and consistency between the two routes on the model’s own drafts; inference emits a claim-level certificate whose SUPPORTED verdict is calibrated by learn-then-test to control the false-support rate among SUPPORTED claims, provided that calibration and deployment claims follow the same distribution. On Qwen3-VL-235B the strict, human-audited unsupported-claim rate falls from 11.0% to 4.6%. The corresponding rates are 7.2% for on-policy grounder-DPO and 7.0% for GRPO trained with the same grounding signal at matched compute, and 6.8% for grounder-DPO with tool outputs in context; HallusionBench accuracy rises by 10.4 points, and the gains persist on long-form hallucination benchmarks and relabelled POPE and replicate on InternVL3.5-38B. Controlled comparisons assess the contributions of the loss terms: the unlikelihood term contributes most, the swap and alignment terms add smaller increments, and coupling claims through shared evidence capacity adds under one point overall. In a study using identical answers, annotators reading ledger certificates needed 19.7% less verification time than those reading grounder certificates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Hallucination Mitigation, Claim-Level Grounding, Evidence Attribution, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_hvK4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an approach to create connections between multimodal tool outputs (evidence atoms), generated model sentences and its contained claims. They use so called evidence ledgers to represent the interaction between evidence atoms, claims and sentences. Training on their algorithm uses the ledgers to improve the model's ability to support claims it makes based on grounded evidence from the tool output. They test their approach on 12 VQA benchmarks and 2 multimodal hallucination datasets.

Their main goal is to have fewer unsupported claims in the original model's output, and in addition more transparency by having trained submodules form a so-called "claim certificate", which allows interpreting the model's answers for evidence support. Multiple submodules are used during training and inference, such as a claim tagger, an evidence projector to move evidence into the same embedding space, a verifier head generating logits per claim and separate post-hoc (evaluation) scorer that generates claim support scores.

During training, the model answer is tagged by the claim tagger, the evidence is projected and both are inputted into the solver, which runs the algorithm to solve the ledger pairs into a claim budget plan. Those plans are then converted into losses for training, which forms part of a multi-objective loss function. They then update the adapters, projectors and tagger during training.

During inference, the pipeline also tags claims and projects evidence, and the solver generates the plans. The verifier head then outputs a score for each claim, and a separate post-hoc scorer generates the final support score, which together with some additional criteria are converted into verdicts of SUPPORTED, WEAK or NULL. Together, this forms the claim certificate.

Contributions are the training multi-objective loss function, the claim certificates, a new evaluation metric verified by a human study and replicating the approach on a second backbone.

Critical strengths
  • The problem of hallucination of VLMs is well known and this approach connects generated claims, sentences and tool outputs in a transparent way, since per-claim evidence support enables verifying VLM output.
  • The authors run a lot of ablations regarding verifying parts of the training objective, such as loss terms, the algorithm solving the transport problem, the null source for claims without a counter part, and verify the consistency of the ledgers.
  • The human audit study is well designed and seems sound.
Critical weaknesses
  • I often struggled to follow the paper. Tables, definitions of metrics and methods, and figures are not clearly labelled and seem in the wrong order also. In a lot of sections, the main results are in the appendix, and the main paper just references them. This made it very difficult to read in a flow (see questions 1, 2, 4, 6).
  • It would be helpful to have a better defined task, and more examples, to explain how evidence ledgers and claim certificates look like in practice for different dataset task types (e.g. VQA tasks vs captioning vs hallucination benchmarks, see question 3 below)
  • The figures are not readable with bad formatting, are way too small, which makes it hard to understand the task and the method.
Questions for authors

Questions:

  1. Table 1: Explanations for 6.2k, Cov., Lat. They are not defined in the main paper. I understand that only columns Hall.Acc and POPE F1 refer to those two HallusionBench and POPE. What are the datasets they are evaluating on for the other columns? The caption does not state this.
  2. "a text-only decomposer trained before this work" (L 337) and the "frozen evaluator J-ind" (L 339) are not defined in the main paper. But those are both critical components for the central metric for evaluation. Please define them and explain how they are trained.
  3. I can see how captioning tasks work well with this approach. However, how do we ensure that the model just doesn't generate any reasoning, and therefore doesn't hallucinate? Wouldn't this undermine the metric? This is mentioned in the appendix, but since the datasets and example ledgers are not explained well in the main paper, I find it hard to understand this.
  4. Tools are used to generate evidence atoms, but the specific tools are only defined in the appendix. What is input & output of the tool calls? This needs to be defined better and earlier in the paper. In addition, the method relies on tool confidence to define the atom capacity limit. However, I didn't find information on how they get the tool confidence.
  5. What is currently the input into the evidence projector? Are tools just used as text-only in the input? Would a multimodal representation of tool outputs (e.g. manipulated images with lines, bounding boxes, ...) work here and improve the evidence projection, and therefore the overall algorithm?
  6. For section 5.5, what are the 5 corruptions? Are they training models here on the corrupted data?
  7. How often do the tools miss correct evidence? If no evidence is found, the claim is penalised in training and marked NULL. How does the tool performance affect overall performance? This is touched on in the appendix, but the main paper would benefit from this.

Suggestions:

  • Some re-ordering of sections, and clean up of the figures, captions and tables would make the paper more readable. In the current state, it seems like too many important details are hidden in the appendix.
  • Section 5.2 describes the main results, and mentions mainly numbers from the appendix. I think it would be beneficial to have a bigger results table in the main paper. I think some of the ablations for the loss objective could be moved into the appendix to make more space (e.g. Table 3). Table 1 is unclear on what dataset it reports on.
  • I would move effects of tool quality (Q7) and vague and empty answers (Q3) into the main paper.
  • The figures need fixing, the table font needs to be larger. They are both difficult to read.
8 comments

Dear Reviewer hvK4,

Thanks for your suggestions. Each item gives the question, where the main text answers it, and the answer.

Q3, does the model gain by saying less? Section 5.2 (p. 8) and Appendix B. On the 520 audit prompts, annotators blind to system identity and shown no certificate judged whether each frozen answer correctly and adequately addresses the question, refusals counting as failures. The ledger is judged adequate on 84.6% of prompts against 83.1% for the backbone and 83.7% for on-policy grounder-DPO, a paired difference of +1.5 pp, interval [−0.6, 3.6], so no decline was observed; this is not a non-inferiority test, and the paper no longer says "at least as often". It emits as many claims per answer as the backbone (2.91 against 2.88), more supported claims (2.78 against 2.56), refuses marginally more prompts (3.1 against 2.5%), and gains on every benchmark of Table 2. Appendix B states one cost: 0.9 to 2.4 pp fewer correct claims about small objects, out-of-prompt classes, and non-Latin text, where the tools see least.

Q7 and Q4, the tools and their misses. Section 3.1 names the tools (Grounding DINO prompted with the 80 COCO classes plus the question's noun phrases, PaddleOCR, DePlot, a 1-fps frame sampler), says what one atom is, and defines conf_i per tool; thresholds, versions, and the prompt are in Appendix C; every atom in Figures 1 and 3 carries its tool and raw output. Misses were measured under three conditions (Table I.28, p. 56): on the 124 audited claims whose evidence the tools missed, certified coverage is 31.5% against 71.5% elsewhere, and 3 of 39 SUPPORTED rows are false, a subgroup rate of 7.7% with interval [1.6, 20.9] that the marginal guarantee does not control; random removal of 10 / 20 / 30% of atoms lowers coverage to 63.1 / 57.4 / 50.9% at a false-SUPPORTED rate of 2.4 to 2.6%; oracle atoms give 74.6% and Cert.F₁ 91.0; the conclusion lists tool recall among three limits.

Q1, the Table 1 columns. The caption states the population of every column: UCR_ver and UCR_J-ind pool the parsed claims of the fourteen benchmark slices and are seed means; hUCR, aUCR, Cert.F₁, and claim coverage are human-graded on the 520-prompt audit of seed-17 outputs; Hall.Acc and POPE F₁ are the benchmarks' own metrics; controls that reuse the ledger's 6,200 annotated claims are marked, with the partition in Table C.4; latency moved to Appendix G; the backbone row's coverage of 84.2%, the grounder's own rule coverage, is noted as unrelated to the ledger's Cert.F₁ of 84.2. A Protocol paragraph in Section 5.1 gives numerator and denominator for every quantity, and a provenance paragraph in Appendix B (p. 25) names population, label source, seed, and aggregation for every headline cell, including why the human-graded Cert.F₁ (84.2) and the automatic seed-17 value in Table G.18 (83.96) differ.

Q2, the evaluation atomizer and J-ind. Section 5.1: a text-only Qwen2.5-7B decomposer trained before this work on 1,500 human-decomposed answers, disjoint from the ledger's atomizer in data, architecture, and initialisation, 93.6% recall and 94.2% precision on 1,200 held-out claims; J-ind is a claim-support classifier fine-tuned from Qwen2.5-VL-72B on 12,000 claims annotated by a separate team, never updated.

Q5, the projector input. Section 3.1 and Table G.13 (p. 48): the tool's text embedding with frozen vision features of the atom's region; text alone gives UCR 5.6% and Cert.F₁ 80.9, the default 4.9 and 84.2, a rendered overlay 4.8 and 84.4 at 1.26x latency. Q6, the corruptions. Section 5.5 lists the five, applied to every system's inputs at seven severities with no training on corrupted data; recipes in Appendix H.

Task, examples, figures, density. The introduction's second paragraph states the task and inputs; Figures 1 and 3 show certificates for a description, a chart, a scene-text, and a diagram question, and Appendix J adds a long description and a video question (p. 58). Figures 1 to 3 are redrawn at full column width in a formal typeface; a programmatic check on the vector output finds no label overlapping or touching another and every label inside its box or on an opaque chip; the main-text tables are set at 7 pt without scaling. Density was reduced by structure, not by removing results: the Findings open with the three main results and define certified coverage and the two loss terms where first used; Section 5.4 is three paragraphs (components; the strongest pipeline without transport; the certificate), each pointing to the appendix table with the detail; and, as you suggested, the main text keeps complete results tables (baselines in Table 1, long-form benchmarks and RePOPE in Table 3, the audit in Table 4), with the ablation detail in Appendix G.

We hope the adequacy and tool analyses answer the soundness questions and that the main text now reads on its own.

Thanks again for your opinion, and in particular for whether the lower unsupported rate could have been bought by saying less; that question shaped the adequacy study. Our reply above gave the headline answers at the character limit, so we add the supporting measurements here, together with a short account of how the main text now reads on its own.

The purple text marks both revised text passages and original submission text for addressing reviewer questions, highlighted and reordered for readability. Pages are those of the revised PDF.

Q3, in full. Beyond the adequacy judgement (84.6 against 83.1% of prompts answered correctly and adequately, +1.5+1.5 pp with interval [−0.6,3.6][-0.6,3.6]; 83.7% for on-policy grounder-DPO), three specificity measures on the 520-prompt audit show no drift towards vaguer claims: filled typed slots per claim 2.71 against 2.70, claims whose head noun is a hypernym of the noun the backbone used 4.1 against 3.9%, claims carrying a number, date, or count 28.1 against 27.5%; blinded quality ratings (1 to 5, two raters, 200 prompts) are 4.21 against 4.19 (Appendix B, p. 26).

Robustness of the headline number (Section 5.3). Counting every excluded item (3.1% of claims) against the ledger shrinks the ledger–backbone gap from −6.4-6.4 to −3.2-3.2 pp, interval [−4.5,−2.0][-4.5,-2.0]; fully manual atomization on 250 prompts gives 11.3 against 4.9%; the gap holds under every leave-one-benchmark-out split; the two automatic graders and the human panel rank the systems with Kendall τ=0.98\tau=0.98 and ≥0.79\ge0.79 (Figure A.1), and the ledger's margin over on-policy grounder-DPO is 2.5 to 2.7 pp under all three.

HallusionBench supplies half of the Avg$_{14}$ gain through a reduced "yes" bias: the rate of answering "yes" on "no" items falls from 42.4% (backbone) to 19.0%, pair consistency rises from 51.0 to 63.7%, and a yes/no logit-bias correction of the backbone tuned on held-out data recovers only part of this (34.6%, 53.9%, Hall.Acc 67.0 against 74.6; Table K.30, p. 59). Saturated document and OCR tasks barely move, while reasoning, chart, and grounding tasks gain one to two points.

Q4 and Q7, the tools' footprint. The tools produce an atom for 92.4% of the evidence items annotators cited (91.3% of objects, 95.8% of text lines, 88.7% of chart elements, 97.0% of video frames). On the 124 audited claims whose evidence they missed, 33 of the 115 supported claims are NULL and 46 WEAK, so a miss mostly withholds a verdict rather than inventing one; the 3 false SUPPORTED rows among the 39 there are reported with their interval (Table I.28, p. 56).

Reading the main text alone. Tables 1, 3, and 4 now hold the full baseline set, the long-form and RePOPE results, and the blinded audit; Section 5.1 defines every grader and population and Table 1's caption names the population of each column; Appendix B's provenance paragraph (p. 25) explains why the human-graded Cert.F1F_1 (84.2) and the automatic seed-17 value (83.96, Table G.18) differ; Figure J.13 (p. 58) adds a long-form description and a video question to the examples of Figures 1 and 3; the main-text tables are set at 7 pt without scaling, and the figure labels (5.8 to 7.6 pt) pass a programmatic check for overlaps.

The certificate in human hands. Fourteen trained annotators (200 prompts, three per prompt, blinded) preferred the ledger certificate in 64.8, 70.2, and 68.5% of judgements for helpfulness, trust, and auditability, needed 19.7% less verification time, and found 81% of unsupported claims against 69%; seeded errors showed the limit, 22% of false-SUPPORTED rows caught, which is why the deployment policy adds a random 10% spot-check of SUPPORTED rows (recall 34% at 6% more time; Appendices M and N, pp. 64 to 65).

We hope these details show that the soundness questions were met with measurements rather than assurances, and we are happy to supply any further detail.

Dear authors, thank you for your detailed clarifications!

After checking out the paper again, I think the authors have definitely improved the paper. I appreciate the added task definition and the added figure with the examples (Figure 3). The main results section reads a lot better now, with fewer references to relevant numbers in the appendix. I also appreciate the addition of the answer quality and blind spot section in the results. I however still think that many sentences are way too long (semicolons all the way), and the paper would overall benefit from shorter sentences which would make it a lot more easily readable.

Some suggestions:

  • Figure 2 definitely still has overlap of components and looks rather messy
  • For the task definition in the introduction: “we ask for” reads a bit awkward, and the double mention of “answer” makes this a bit confusing. How can an answer be input and output? This is my understanding of the task/ approach: The task is Visual Question Answering, where given an image/ video/… input and corresponding question, a model produces an answer. The goal is to have fewer unsupported claims in the model's answer. The authors therefore introduce a framework that links the claims in the model’s own answer to evidence from external tools. During training, claims that no tool output supports are penalised, which lowers the rate of unsupported claims. During inference, the same framework produces a per-claim certificate, giving the supporting evidence and a verdict with a controlled false-SUPPORTED rate.
Reviewer_BBrLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Evidence Ledgers align input evidence, typed claims, and answer units through direct and composed transport plans with a null source. Training combines unlikelihood, counterfactual swaps, and alignment; inference adds calibrated claim certificates. A blinded audit reports unsupported claims falling from 11.0% to 4.6%, alongside matched baselines, a second backbone, and a fixed-answer human study.

Critical strengths
  • The blinded, prompt-paired human audit, manual atomization checks, exclusion sensitivity analysis, and matched-compute on-policy baselines make the reduction in unsupported content unusually well substantiated (Sections 5.1-5.4).
  • Separating generated-answer quality from certificate quality, and evaluating certificates on identical answers, supports their practical value. The paper also explicitly analyzes the limited contribution of capacity coupling rather than attributing every gain to transport.
Critical weaknesses
  • The distinctive coupled solver adds limited demonstrated value: capacities bind on only 6.1% of inputs, and the simpler softmax control is within 0.5 points with an interval spanning zero (Section 5.4). This weakens the case for the full machinery; the simpler construction deserves a more prominent role.
  • Certification remains conditional on the adjudicated claim distribution. Appendix D's finite prompt-pool sampling does not itself establish control on a fresh prompt stream, and tool blind spots reduce some correct content (Appendix B). These are acknowledged limitations, but they materially restrict the deployment interpretation of the certificate.
Questions for authors
  1. Do existing results show whether the coupled solver improves certified coverage or fixed-answer certificate utility over the B1 softmax control, beyond its small generation effect?
2 comments

Dear Reviewer BBrL,

Thanks for your helpful feedback. Both weaknesses are now measurements, in purple; pages refer to the revised PDF.

W1, Q1: does the coupled solver improve the certificate? Where the generator result predicted. We held the answers fixed and compared the ledger scorer with Control B1's own verifier head, trained on B1's softmax plans with the same architecture, data, and tuning budget and calibrated separately at ϑ=3%\vartheta=3\% (Table G.23, p. 52). On identical ledger answers the ledger scorer certifies 68.2% of claims at a test false-SUPPORTED rate of 2.3% (AUROC .86, Cert.F1F_1 84.2) against 67.6% at 2.4% (.85, 83.4), +0.6+0.6 pp with interval [0.1,1.1][0.1,1.1]; on the slack subset +0.3+0.3 pp, interval [−0.3,0.9][-0.3,0.9], on backbone answers +0.5+0.5, interval [−0.1,1.1][-0.1,1.1]; on the constructed binding slice, where several claims compete for one atom, outside every headline number, 58.9 against 54.1%, +4.8+4.8 pp, interval [3.1,6.5][3.1,6.5]. As B1 also lacks the row-degree penalty, the penalty-matched coupled control's scorer was run on the same answers: 68.0 and 58.4%, so the penalty accounts for 0.2 and 0.5 pp and the coupled solve for the rest. A same-answer human arm on 60 binding-slice prompts found 83% of seeded unsupported claims against 76%, interval on the difference [2.0,11.9][2.0,11.9], in 6.2% less time; on 60 slack prompts 80 against 79% (Appendix N). The certificate thus tells the generator's story, and we drew the conclusion you suggested: Section 3.3 describes the solver as a per-claim softmax over candidate atoms and the null source, corrected where an atom is over-subscribed, and recommends the lazy form in deployment, softmax first and the coupled iteration only when a capacity binds, with every plan unchanged. What the plans add in deployment, lazily solved or coupled, is the two-route residual: under shift at severity 0.75 the ledger's false-SUPPORTED rate is 2.8 to 4.4% against 4.6 to 7.9% for a matched non-transport scorer on the same answers, and at equal coverage, with thresholds selected on shifted calibration claims, the non-transport scorer remains 1.1 to 1.9 pp higher while the B1 scorer, which has two routes of its own, is within 0.3 pp (Table D.10, p. 38); deleting the atoms a row cites flips 91.4% of SUPPORTED rows for the ledger and 90.2% for B1 against 73.6% for the non-transport scorer, 84.6 against 61.2% on the 806 rows that all five scorers stamp SUPPORTED, one matched atom deleted (K.33, p. 62). Corollary 1 reads the NULL rule's null-dominance branch, under slack capacities, as a threshold on the soft minimum of candidate costs.

W2: certification is conditional on the adjudicated claim distribution. Agreed, and the text says so at the theorem. We add an empirical test of the deployment protocol on two benchmarks absent from every training, calibration, and audit set, MathVista-mini (1,000 prompts) and RealWorldQA (765), each split at the image level before any scoring into an evaluation pool of 210 prompts, all parsed claims adjudicated (600 per stream), and a recalibration pool with 1,500 adjudicated claims; every rate is on the evaluation claims and the alert level of 0.11 was fixed beforehand. Table D.11 (p. 39) gives each rate as a count: at the in-distribution threshold the false-SUPPORTED rate is 10/401=2.5%10/401=2.5\% on RealWorldQA, interval [1.2,4.5][1.2,4.5], and 12/367=3.3%12/367=3.3\% on MathVista-mini, interval [1.7,5.6][1.7,5.6]; both include the 3% target, so neither stream is shown to violate or meet it. The label-free monitor stays below its alert level on RealWorldQA (0.097) and alerts on MathVista-mini (0.118), where recalibration on the 1,500 claims moves the threshold one grid step and gives 8/350=2.3%8/350=2.3\%, interval [1.0,4.5][1.0,4.5], for 2.9 pp of coverage; on the 35 stress cells of Appendix D the monitor alerts in all 21 cells whose false-SUPPORTED rate exceeds 3% and in 2 of the 14 whose rate does not. The generator, never tuned on either stream, lowers J-ind UCR from 12.9 to 6.1% and 10.8 to 4.7%; this is evidence about the protocol, not an extension of the guarantee.

Tool blind spots. Three conditions (Table I.28, p. 56). Natural omissions: on the 124 audited claims whose evidence includes an item the tools missed, certified coverage is 31.5% against 71.5% elsewhere, 33 of 115 supported claims are NULL, 46 WEAK, and 3 of 39 SUPPORTED rows are false, a subgroup rate of 7.7% with interval [1.6,20.9][1.6,20.9] that the marginal guarantee does not control and whose interval straddles the target; missing evidence mostly moves a claim to NULL or WEAK. Random removal of 10 / 20 / 30% of atoms lowers certified coverage to 63.1 / 57.4 / 50.9% at a false-SUPPORTED rate of 2.4 to 2.6%; oracle atoms raise it to 74.6% and Cert.F1F_1 to 91.0.

Presentation. Every grader and quantity is defined in Section 5.1; Figures 1 to 3 are redrawn at full width with no label overlapping another or crossing a box edge. We hope these measurements resolve both weaknesses as framed.

Thank you again for the review; the two weaknesses you raised led to the same-answer scorer comparison and the fresh-stream protocol that now anchor the revision. A few further measurements did not fit in our reply above, and since they speak to the same two questions we add them here.

The purple text marks both revised text passages and original submission text for addressing reviewer questions, highlighted and reordered for readability. Pages are those of the revised PDF.

W1. Two checks on the coupled solver beyond the same-answer comparison. First, with demand-scaled capacities, supply about 1.2×1.2\times demand so that rows bind on 38% of examples, the ledger's margin over Control B1 on the full evaluation set is 1.1 pp while its own UCR is 5.1% (Tables G.13 and G.22, p. 52): coupling does more where it is asked to do more, and the default capacities, under which 6.1% of examples bind, are the setting we report. Second, Gemma-3-27B repeats the ordering with the same small gap on the natural distribution, Control B1 at 6.6% against the ledger's 6.3% (Table G.15, p. 49). Both are why Section 3.3 presents the solver as a per-claim softmax corrected where capacities bind and recommends the lazy form in deployment.

W2. Three operational facts about recalibration and monitoring. Under shift at severity 0.75, recalibrating on 1,500 adjudicated claims from the shifted regime restores the target at a coverage cost of 4 to 9 pp; in bootstrap replicates of the modality-drop regime the procedure finds an admissible threshold in 91% of replicates with 1,500 claims but in only 12% with 400, so 1,500 claims is the recommended recalibration size (Appendix D). The label-free monitor's error rates are measured on the 35 stress cells rather than on the two streams, whose intervals include the target: at the fixed alert level of 0.11 it alerts in all 21 cells whose false-SUPPORTED rate exceeds 3% and in 2 of the 14 whose rate does not, and the diagnostic Δcyc+Δ∅\Delta_{\text{cyc}}+\Delta_{\varnothing} tracks J-ind UCR at Spearman 0.97 across the cells and 0.71 within regimes after partialling out severity (Figure A.4). On the two fresh streams the counts are 10/401 on RealWorldQA and 12/367 on MathVista-mini at the in-distribution threshold and 8/350 after recalibration (Table D.11, p. 39).

Blind spots in the generator as well as the certificate. Relative to the backbone the ledger emits 0.9 pp fewer correct claims about objects smaller than 32232^2 pixels, 1.2 pp fewer about classes outside the detector prompt, and 2.4 pp fewer about non-Latin text, with a NULL over-flag rate of 12.4% on these categories against 5.9% overall (Appendix B, p. 26); the random-removal rows of Table I.28 (p. 56) show the same pattern in the certificate, coverage falling with tool recall while the false-SUPPORTED rate stays within 2.4 to 2.6%.

Reviewer_4WJMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a framework for auditing vision-language model outputs on a claim-by-claim basis. A model output is decomposed into claims, which are connected to evidence atoms through a sparse transport structure: a direct evidence-to-answer route and a composed evidence-to-claim-to-answer route. A claim that is not supported by any of the evidence atoms can be assigned to a null source. These allocations are used during training and to produce a per-claim certificate containing supporting evidence, a calibrated support score, and a verdict. The training combines answer likelihood with unlikelihood on unsupported claims (weighted by the null-source attribution), counterfactual evidence swaps, consistency between the direct and composed routes, and (supervised) support calibration. The paper also uses "learn-then-test" risk control (from prior work) to select the threshold for the SUPPORTED verdict. The evaluation is broad, includes state-of-the-art large vision-language models and shows that the unsupported claim rate can be reduced compared to compute-matched post-training alignment baselines.

Critical strengths
  1. The paper tackles an important task. Evidence accounting on a claim-by-claim level is more useful and interpretable for auditing multimodal outputs than answer-level confidence or free-form generated confidence reports.

  2. The methodology and evaluation appear exhaustive. The main claim results are validated through a blinded human user study rather than solely through an automated verifier, and the paper also studies manual atomization, alternative graders, benchmark-family robustness, and cross-backbone transfer.

  3. The experimental controls are strong, the theoretical claim is appropriately scoped, and the implementation and experimental accounting are described in detail.

  4. The certificate appears practically useful and the paper is very transparent about the contribution of individual components. It explicitly shows that much of the performance improvement comes from null-weighted unlikelihood and that the coupled transport solver itself contributes relatively modestly to the overall distribution.

Critical weaknesses
  1. My main concern is methodological novelty attributable specifically to the ledger construction. The ablations show that null-weighted unlikelihood accounts for a substantial fraction of the improvement, while weighting the same unlikelihood objective with a conventional frozen grounder performs similarly when considered in isolation. Related to that, the full coupled optimal-transport solver appears to contribute only modestly to the overall distribution.

  2. The theoretical result is useful but generic. The "learn-then-test" guarantee, in my understanding, applies to any thresholded support scorer and therefore does not theoretically validate the transport mechanism itself. Thus, the ledger-specific contribution is mainly the certified coverage achieved under this general risk-control procedure.

  3. Despite the detailed appendix, I still have serious concerns about correctness and reproducibility. Specifically, some proofs seem insufficiently justified or inconsistent. For example, the convergence argument in Lemma 1 assumes the existence of a strictly positive feasible plan on the sparse support, which is not guaranteed by the presented construction. The proof of Proposition 2 changes the degree caps from the method's stated Kc=8,Kr=2K_c=8, K_r=2 to Kc=Kr=1K_c=K_r=1. Proposition 4 appears to omit the cost of constructing the Top-KK candidate supports when claiming near-linear complexity. These issues make it difficult to assess if the described system can be reconstructed. Because of the unusually large experimental scope and computational scale, releasing code, configurations, and ideally trained checkpoints would be very valuable for reproducibility.

  4. I am not fully able to reconstruct how the trainable claim atomizer is learned from the description. It is described as a trainable span tagger but I do not see an explicit span-tagging or slot-classification loss. Gradients through claim embeddings can train their continuous representations, but it is unclear how discrete span and slot decisions themselves are supervised or differentiated through.

  5. Some evidence-tool details remain underspecified. In particular, Grounding DINO requires a textual detection query or vocabulary, but the manuscript specifies the checkpoint and threshold without clearly stating the detector prompt or class vocabulary. The paper only refers to objects outside the detector's COCO vocabulary, which suggests that a fixed vocabulary may be used but this choice can determine which evidence atoms exist and should be specified explicitly.

  6. There remains a conceptual gap between factual correctness and evidence-grounded support. The evaluation treats standard world knowledge that is not contradicted by the input as supported. However, the ledger is designed to account for support from available evidence atoms and therfore tends to over-flag such claims. This does not invalidate any results, but it affects the intended notion of support. This is a minor weakness which does not need to be addressed in the rebuttal. It could just be delineated more sharply in a future revision.

  7. There are a number of presentation and writing issues. In some of the visualizations, there is text overflow and text overlapping with other geometric shape boundaries, which makes them difficult to read (e.g., "above threshold" in Figure 1). There are numerous sentences which are difficult to understand, e.g.: Section 1 Findings "First, the solver is analytically transparent: whenever no evidence capacity binds, which holds on 94% of benchmark inputs, each column of the plan reduces to a softmax over candidate atoms and the null source (Proposition 1); a control built from that construction performs within 0.5 pp of the full model, and a penalty-matched comparison credits the coupled solve with 1.3–1.5 pp only where capacities bind." --> The Findings and Contributions sections are barely comprehensible without reading the rest of the paper.

Questions for authors
  1. How exactly is the claim atomizer supervised? What loss trains the span boundaries and the entity, attribute, and value assignments? If these decisions are discrete, how are gradients propagated through to the tagging component?

  2. What textual query is supplied to DINO when constructing evidence atoms? Is this a fixed COCO vocabulary, terms extracted from the user question, or something else?

  3. The per-claim softmax control performs quite close to the full coupled solver. Do the authors view the principal contribution as the complete training and certification framework rather than specifically the coupled optimal-transport formulation?

  4. Could the authors discuss a control combining grounder-weighted unlikelihood, counterfactual swaps, and a simpler non-transport representation-alignment objective to quantify how much of the advantage requires the tri-partite transport representation?

4 comments

Dear Reviewer 4WJM,

Thanks. We've included detailed explanations for your suggestions.

W1, Q3, Q4. The contribution is full training and certification framework, as Q3 suggests, and the abstract, Findings, and conclusion say so; transport is representation its parts read (null shares, funding sets, swap target, two-route residual). Q4's recipe, grounder-weighted unlikelihood plus counterfactual swaps plus a simpler alignment, is Control D (T5, p. 9; G.13, p. 48), which with an independently trained non-transport scorer is the strongest transport-free pipeline we could build. Against it the full framework lowers hUCR from 5.5 to 4.6% ([−1.6,−0.2][-1.6,-0.2]), raises Cert.F1F_1 from 80.4 to 84.2 ([2.4,5.2][2.4,5.2]), and raises certified coverage at ϑ=3%\vartheta=3\% from 60.9 to 68.2% ([5.6,9.0][5.6,9.0]); 3.0 of the 7.3 pp is the ledger scorer's margin on Control D's own answers (G.24, p. 53; Table 5) and the rest accompanies the change of generator; the paper quotes the same-answer figure. Control D$',ControlDwiththefrozenreferenceledger′snull−shareweights,trailsby0.7pp(, Control D with the frozen reference ledger's null-share weights, trails by 0.7 pp ([0.1,1.3]$ audit, [0.3,1.1][0.3,1.1] seeds), a sensitivity analysis, those weights being frozen (Appendix G, p. 47). Controls C and D$'$ coincide to the reported precision (0.0 pp, [−0.5,0.5][-0.5,0.5]); as this does not establish equivalence, the two-route term is estimated as ledger against C, 0.7 pp under Holm. Two same-answer tables show what the allocation adds: deleting cited atoms flips 91.4% of the ledger's SUPPORTED rows against 73.6% for the non-transport scorer, 84.6 against 61.2% on the 806 rows all five scorers stamp SUPPORTED with one matched atom deleted (K.33, p. 62); under shift the ledger's false-SUPPORTED rate is 2.8 to 4.4 against 4.6 to 7.9, 1.1 to 1.9 pp lower at matched coverage, the two-route B1 scorer within 0.3 pp (D.10, p. 38).

W2. Learn-then-test is adopted as a reliability tool; ledger-specific content is the coverage attained at the controlled level, 68.2% against 64.6, 61.4, and 58.7 for the comparison scorers on the same answers (G.24); Corollaries 1 and 2 (pp. 44--45) cover null dominance under slack capacities and the joint level δ1+δ2\delta_1+\delta_2.

W3, proofs. Lemma 1 (p. 40) states b,q,u>0b,q,u>0, exhibits a strictly positive feasible plan, and derives convergence from the dual; Proposition 2 (p. 42) is re-proved with the method's caps Kc=8K_c=8, Kr=2K_r=2 on a twelve-atom construction. Pp 4 (p. 43) counts the dense cost matrix and the candidates' costs, ranks candidates by the cosine term as the exact path does, and bounds composed-column support by min⁡(n+1,Keffℓ+1)\min(n{+}1,K_{\text{eff}}\ell{+}1); Algorithm 1 and Eq. 7 use each plan's own budget in the degree penalty and sum over concrete rows only.

W3, reproducibility. Table C.5 (p. 31) records every description, each marked completed or corrected; and two corrections of record: the compute total, which omitted twelve sensitivity runs reported in Table G.20, and the comparison scorers' AUROC, computed under the verifier's labels in the submission; on the adjudicated labels of every other cell it is .74/.72, and the per-claim score files of both computations are supplied. The supplementary material holds configuration files, per-step loss logs per term of Eq. 3, emitted certificates, anonymised adjudications, human-study records, and risk-control draws; the certificate on p. 30 prints its three plans with the question, detector prompt, residuals, gap components, and null masses, so rj≤max⁡kdk≤Δcyc≤r_j\le\max_k d_k\le\Delta_{\text{cyc}}\le gap and Ωs\Omega_s can be recomputed. The training codebase is coupled to institution-internal data-access layers and schedulers that carry identifying information and could not be anonymised within the discussion period; it is released on acceptance with the trained modules, annotations, and graders.

W4, Q1. Ltag\mathcal L_{\text{tag}} is a BIO span plus slot cross-entropy on head-training claims and swap pairs' silver spans, λtag=1.0\lambda_{\text{tag}}=1.0; span and slot decisions are taken by argmax and treated as constants, so no ledger loss differentiates through a discrete decision and the ledger losses reach the atomizer only through the pooled claim embedding zCz^C (Appendix C, p. 27).

W5, Q2. Grounding DINO is prompted with the 80 COCO classes plus the question's noun phrases; every atom in Figures 1 and 3 and the printed certificate carries its tool and raw output; Appendix I (p. 55) measures the false-premise risk of those phrases.

W6. Appendix B separates the two support notions.

W7. Figures 1 to 3 are redrawn at full width; no label overlaps or touches another or crosses a box edge.

We believe our work is solid. We hope these answer W1–W7, and we will gladly discuss any remaining questions during the discussion.

AuthorsOpenReview ↗

Where the framework's gain comes from, and what can be checked.

Thanks again. The purple text marks both revised text passages and original submission text for addressing reviewer questions, highlighted and reordered for readability. Pages are those of the revised PDF. Our reply above reached the character limit before several measurements bearing on W1 and W3 could be included, so we add them here; nothing changes the position stated above, and every figure is in the revised PDF at the page given.

1. The gain is not the unlikelihood term alone. Control A, answer-only LoRA with grounder-weighted unlikelihood and nothing else, reaches hUCR 6.9% (UCR 7.2%); the full framework reaches 4.6% (4.9%), so 2.3 of the 6.4 pp gained over the backbone come from the parts beyond unlikelihood (Table 5). Those parts interact rather than add: removing the direct route from the full objective returns UCR to 10.1%, removing the null source to 12.3%, and removing the claim layer to 9.5%, each at or above answer-only LoRA's 10.8%, while fixing Π^CA to span overlap gives back most of the gain, 8.7% (Table G.13, p. 48). This is why Section 5.4 attributes by the additive order on matched controls, and it keeps the representation in view: the null shares weight the unlikelihood term, the funding sets define the swap target, and the two routes supply the residual.

2. Capacity coupling, measured where it should act. Against the B1 scorer the certificate gains 0.6 pp of coverage overall and 4.8 pp on the constructed binding slice; the penalty-matched coupled scorer certifies 68.0 and 58.4% on the same answers, so the row-degree penalty accounts for 0.2 and 0.5 pp of those margins and the coupled solve for the rest (Table G.23, p. 52). With demand-scaled capacities, under which rows bind on 38% of examples, the ledger's margin over B1 on the full evaluation set is 1.1 pp at UCR 5.1% (Tables G.13 and G.22).

3. What a reader of the certificate gains. On identical answers, annotators found 292 of 360 seeded unsupported claims with the ledger certificate against 266 with the non-transport certificate (81.1 against 73.9%, interval [2.4, 12.1] on the difference), in 8.9% less time, and preferred it in 60.3% of judgements (Appendix N, p. 64). An off-the-shelf claim verifier, AlignScore-large applied to each claim and the text of its candidate atoms, certifies 54.8% of claims at a 2.7% false-SUPPORTED rate (AUROC .69), below every scorer trained on the annotations, and cites no atom (Appendix K, p. 61).

W6, the two notions of support. The audit's factual criterion and the ledger's evidence-grounded support coincide on perception, OCR/numeric, and reasoning claims, 88% of all claims, and separate on world-knowledge claims, where the ledger has no source to fund a true fact: it flags 9.2% of them NULL while 4.1% are unsupported, emits 0.36 such claims per answer against the backbone's 0.37, and their audited support rate rises from 93.6 to 95.9% (Appendix B, Table B.1, p. 24).

W3, the proofs in one line each. Lemma 1 (p. 40): mass ε on every supported concrete entry, the remainder of each column on the null row, is feasible and strictly positive for ε below min_j b_j/K_eff and min_i u_i/ℓ, and b·qᵀ plays that role for the dense Π^CA. Proposition 2's pure-overflow case (p. 42) uses the method's caps: twelve atoms with u ≡ 1, one claim, one unit, Π^EC_i1 = 1/12, so every row has soft degree h(1/12) ≈ 0.94 < K_r = 2 while the column has 12·h(1/12) ≈ 11.3 > K_c = 8. The printed certificate (p. 30) lists its three plans; from them d_k = .019/.011/.009, Δ_cyc = .039, Γ = 0.

Label noise and compute. Redrawing every disputed label (21% of claims) in 1,000 replicates, the procedure certifies in 960 of them (918 at 0.50, 42 at 0.525) with the test rate within [1.9, 2.9]%; the 40 failures are non-rejections at the first grid point, τ = 0.975, where n_S = 352 (Appendix D). The project total is 147,127 active H100-hours over 309 runs (Table C.6).

I thank the authors for their response. Some of my concerns have been addressed adequately (W4, W6 and W7). However, I continue to have significant concerns about the reproducibility of this work (e.g. W3), and the revised manuscript still lacks clarity. Therefore, I will maintain my score.

4WJM, thanks for reviewing the revision and confirming W4, W6, and W7 are settled. We broadly understand your two remaining concerns and appreciate your review. We address both fully below, citing today’s PDF. The main text provides substantial soundness evidence, and other reviewers have no concerns on this aspect.

The implementation and the trained weights are in use in an industrial deployment of the method for hallucination control, and they are bound by the confidentiality obligations of that deployment for the duration of the review, in addition to the anonymisation problem we described. The release on acceptance set out in Appendix C (p. 29), codebase, trained modules, claim-level records, risk-control sets, graders, and table scripts, has been agreed and stands. What we can do now is the following.

A demonstration implementation for inspection. We have written a single-file implementation of Algorithms 1 and 3 from the specification in the paper (numpy only, under 300 lines): candidate sets by the cosine term with protected priors, the semi-relaxed Sinkhorn solve with exact column budgets, clipped concrete rows, and a free null row, the balanced claim-to-unit solve, the composed plan, Γ\Gamma, Ωs\Omega_s over concrete rows, the gap, the null masses, and the emitter (shares, funding sets, residuals, contradiction flag, verdict rule). Run on the certificate printed on p. 30, it recomputes every derived field from the printed plans, the funding sets, shares, residuals, Δcyc\Delta_{\text{cyc}}, Γ\Gamma, Ωs\Omega_s, gap, null masses, and verdicts, and reproduces the printed values exactly; it also checks feasibility on constructed instances. It is a demonstration of the specification, not the production code, and is provided for viewing during the discussion; we are adding it to the supplementary material.

What else can be verified now. Algorithms 1 to 3 (pp. 45 to 46) give the solver, the training step, and the emitter operation by operation, with every constant in Appendix C and the checkpoint identifiers in Table C.3 (p. 29). Appendix F reports the implementation checked against an interior-point reference and an independent Sinkhorn on 2,000 instances, with the instances and reference solutions supplied, so any re-implementation can be tested against them (p. 45). The supplementary material holds the configuration of every trained system, the detector and grader prompts, the per-step loss logs with each term of Eq. 3 separately, the per-run evaluation files behind every table, the anonymised adjudications of the audit and the fresh streams, the risk-control draws with the grid and the counts of every grid point, and a manifest from every number to its files (p. 29); every table can be recomputed from these files without training anything. The 8B reference configuration (p. 34) runs the same recipe at 253 active H100-hours per seed and reports three-seed values (strict UCR 11.0 to 6.9%, Cert.F1F_1 81.3, seed SDs 0.19 pp and 0.31), a reproduction target within a single-node budget. If a particular file among those listed would help during the discussion, please name it and we will provide it.

The proofs. Lemma 1 (p. 40) states b,q,u>0b,q,u>0, exhibits the strictly positive feasible plan, and derives convergence from the dual; Proposition 2 (p. 42) is proved with the method's caps Kc=8K_c=8, Kr=2K_r=2; Proposition 4 (p. 43) counts the construction of the candidate sets and bounds the composed support; Algorithm 1 and Eq. 7 agree on the degree penalty. If any step still reads as insufficiently justified, we will expand it here.

Clarity. The PDF uploaded today carries a further revision of the main text made by hand for readability: sentences average about 20 words, against nearly 40 before; a Terms paragraph closing Section 1 defines the terms the paper relies on before they are used; Sections 5.2 to 5.5 open with their conclusions; the attribution and certificate detail added during the discussion sits in Appendices G and K with one-sentence pointers. The Findings sentence you quoted in W7 is now one sentence, with its measurements in Section 5.4, and Figures 1 to 3 pass a programmatic check for overlapping labels.

W1 and W2. We rest on our first reply: the contribution is the full training and certification framework, as your Q3 suggested, measured end to end against Control D with a non-transport scorer (Table 5, p. 9; G.24, p. 53); learn-then-test is adopted as a tool, and the ledger-specific content is the coverage attained at controlled risk together with the two same-answer properties of the certificate (K.33, p. 62; D.10, p. 38).

We hope this makes clear what can be inspected now and what follows on acceptance, and we would be grateful if you could weigh the revised manuscript on that basis.

Reviewer_zrosOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Proposes a claim-level grounding method for vision-language models that aligns answer claims with visual evidence and emits calibrated, auditable certificates without changing the generated answer (post-generation, claim-level evidence-grounding audit). The paper evaluates the method across hallucination benchmarks, controlled baselines, human audits, and a second vision backbone, reporting lower unsupported-claim rates and faster human verification.

The evaluation includes a broad set of baselines, but the main controlled comparisons are concentrated on Qwen3-VL-235B; additional replications across independently trained backbones and methods would better establish generality. The paper reports agreement between human and automatic graders (Kendall τ ≥ 0.79), with the same overall ordering of key systems, but does not compare training on human-annotated versus fully automatically generated claim data. Its automatically constructed swap pairs are derived from human annotations, so the robustness of the method to fully automated supervision remains untested.

Critical strengths
  • claim-level evidence certificates with explicit calibrated SUPPORTED/WEAK/NULL verdicts
  • multiple validation layers, including blinded human audits, calibration analysis, robustness tests, and a second-backbone replication
Critical weaknesses

The system’s NULL verdict identifies unsupported claims only modestly well: it catches 50% of unsupported claims, and only 39.3% of NULL-flagged claims are actually unsupported in the audited sample. While the high precision of SUPPORTED verdicts makes the method useful for confirming evidence, these results suggest that it should not be treated as a standalone hallucination detector; its practical value depends on human review or a downstream process for handling flagged claims.

Questions for authors

Could the system be calibrated for a complementary operating point that targets detection of unsupported claims, rather than primarily controlling the false-SUPPORTED rate? For example, could you select the NULL/WEAK decision threshold to achieve a specified precision or recall for unsupported-claim detection, and report the resulting precision–recall/coverage trade-off on a held-out set? This would clarify whether the method can serve as a reliable flagging tool, in addition to confirming claims it supports.

2 comments

Dear zros,

Thanks. The rule was designed to make SUPPORTED reliable, and NULL as emitted is a pointer for review rather than a detector (recall 50%, precision 39%). The revision adds the requested operating point and the generality checks.

Q1. A detection-targeted operating point. We define the flag score fj=1−s^jf_j = 1-\hat s_j and flag claims with fj≥τflagf_j \ge \tau_{\text{flag}}; either the miss rate Pr⁡(fj<τflag∣unsupported)\Pr(f_j<\tau_{\text{flag}}\mid \text{unsupported}) or the false-discovery rate serves as the risk in the fixed-sequence procedure of Theorem 1. Table K.32 (p. 62) reports, on the 1,512 audited test claims (70 unsupported, 4.6%), flagged count nn, true positives, precision, and recall for the ledger scorer and the grounder; thresholds were selected on the 2,000-claim risk-control set at δ=0.05\delta=0.05 and applied without tuning.

Flag setScorernn (%)TP / P / R
Score threshold at the NULL rateledger89 (5.9)34 / 38.2 / 48.6
NULL ruleledger89 (5.9)35 / 39.3 / 50.0
LTT, miss rate ≤30%\le 30\%ledger249 (16.5)51 / 20.5 / 72.9
LTT, miss rate ≤30%\le 30\%grounder652 (43.1)49 / 7.5 / 70.0
LTT, FDR ≤50%\le 50\%ledger33 (2.2)17 / 51.5 / 24.3

The NULL rule and the score threshold at the same flag rate sit at similar operating points (35 against 34 true positives); the rule is not a point of the score's curve, as its contradiction branch is not a threshold on s^j\hat s_j, and the curve supplies the other points. The practical reading is the review set: controlling the miss rate at 30% selects 16.5% of claims that hold 73% of the unsupported ones, with intervals [60.9,82.8][60.9,82.8] on recall and [15.6,26.0][15.6,26.0] on precision; 19 unsupported claims lie outside it, 1.5% of the 1,263 unflagged rows, and the grounder needs 43.1% of rows for the same recall, because 23 of its 70 unsupported claims rank among its 888 highest-scored rows, beyond the reach of any flag set of moderate size (K.32). The table is a single run because audited labels exist for seed 17 only; under the verifier's labels the five seeds' review sets span 15.9 to 17.1% of claims at recall 70.3 to 74.9%. No threshold of either scorer reaches recall 0.7 at precision above 0.22, so the certificate ranks claims for review rather than detecting unsupported claims on its own (Section 5.4, Appendix K, p. 62). The deployment policy gains a "Review" action; Corollary 2 shows that SUPPORTED and review thresholds selected on one calibration set hold jointly at level δ1+δ2\delta_1+\delta_2.

Generality across language models. Section 5.5 and Table G.14 state that InternVL3.5-38B's language model is initialised from Qwen3, so the submitted replication varied vision tower, density, and scale, not the language-model family; we add Gemma-3-27B-IT (SigLIP encoder, Gemma LM), three seeds (Table G.15, p. 49): strict J-ind UCR falls from 13.4% for the matched answer-only LoRA to 6.3%, a 53% reduction as on the other backbones, with a blinded 200-prompt audit of 12.3 against 5.8%. Since the request was for replication across methods as well as backbones, we also retrained the compute-matched on-policy grounder-DPO baseline with the same grounder signal on Gemma (GRPO and tools-in-context were not retrained): 9.3% (audit 8.6%), so the ledger's margin over it is 3.0 pp, interval [2.3,3.7][2.3,3.7] over seeds, against 2.5 pp on Qwen3 under the same grader; Control B1 reaches 6.6%, the unlikelihood term alone 8.5%, and the transferred frozen atomizer 6.9% at 38% of the compute. With verifier-labelled calibration the certificate certifies 65.9% of claims on InternVL and 63.4% on Gemma at test false-SUPPORTED rates of 2.4 and 2.5%, and two prompt streams unseen at every stage show the generator gain as well (Table D.11, p. 39).

Human versus tool-derived supervision. To test the dependence on human claim labels we built a tool-derived set: a claim is labelled supported when its entity slot lexically matches a detector label, OCR string, or chart series in the atom's region and its value slot agrees with the atom. Every human-supervised training component is replaced (atomizer from MiniLM on rule-based silver spans; verifier, calibration, and post-hoc maps on silver labels); human labels enter through the 2,000 risk-control claims, since Theorem 1 concerns adjudicated labels, and through the evaluation. The silver labels agree with adjudication on 88.4% of a 1,000-claim check; the ledger reaches UCR 5.4% and Cert.F1F_1 81.6 against 4.9% and 84.2, certifying 64.0% of claims at a test false-SUPPORTED rate of 2.5% (Appendix C, supervision regimes; Table G.13, p. 48).

We hope the operating point, with its counts and intervals, answers the main concern with a documented capability and a stated review cost, and that the third backbone with its retrained baseline and the tool-derived regime close the generality gaps.

Thanks for the review and for the clear framing of the detection question. Our reply above was written to the character limit, and a few supporting details had to be left out; we add them here in case they are useful when you weigh the operating point and the generality checks, and we are happy to stop at whatever level of detail you find sufficient during the rebuttal period.

The purple text marks both revised text passages and original submission text for addressing reviewer questions, highlighted and reordered for readability. Pages are those of the revised PDF.

Joint control of the two operating points. The SUPPORTED threshold and the review threshold are selected on the same calibration set, and Corollary 2 (p. 45) gives their joint level δ1+δ2\delta_1+\delta_2: at 0.05 each the two hold jointly with probability at least 0.90; a deployment that needs 0.95 runs both at 0.025, which leaves τ^sup\hat\tau_{\text{sup}} at 0.500 (the next grid point already has p=0.067p=0.067) and moves the review threshold by one grid step, so the review set grows from 16.5% to 17.3% of claims. Under the calibrated verifier's labels the five training seeds' review sets span 15.9 to 17.1% of claims at recall 70.3 to 74.9%, so the operating point is not a property of seed 17. A precision-oriented point also exists: controlling the false-discovery rate at 50% selects 33 claims (2.2%) with precision 51.5%, interval [33.5,69.2][33.5,69.2] (Table K.32, p. 62).

The certificate on the other backbones. With a threshold selected on 2,000 verifier-labelled claims of each backbone's own answers, the certificate certifies 65.9% of claims on InternVL3.5-38B at a 2.4% false-SUPPORTED rate under J-ind and 63.4% at 2.5% on Gemma-3-27B (Tables G.14 and G.15, p. 49). On Gemma the ordering of the controls repeats: Control B1 6.6%, the unlikelihood term alone 8.5%, the transferred frozen atomizer 6.9% at 38% of the compute, the full ledger 6.3%, against 9.3% for compute-matched on-policy grounder-DPO (audit 8.6%).

Tool-derived supervision. The silver labels agree with adjudication on 88.4% of a 1,000-claim check (precision 91.2% on supported and 74.5% on unsupported labels); 200k such records give UCR 5.2% and Cert.F1F_1 82.3 against 5.4 and 81.6 with 46k (Appendix C, p. 33), and adjudicated labels remain only in the 2,000 risk-control claims and the evaluation. Redrawing every disputed adjudication (21% of claims) in 1,000 replicates leaves the certified test rate within [1.9,2.9]%[1.9,2.9]\%, with certification in 960 replicates (Appendix D), so the guarantee's dependence on human labels is a dependence on their adjudicated distribution rather than on any single annotator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.