acceptodds
Under review as a conference paper at ICLR 2027

CertiHop: Certified Necessary-Evidence Supervision for Long-Context Reasoning

Abstract

Long-context question answering benchmarks often annotate supporting evidence, yet support is weaker than necessity: a model may answer correctly without combining every designated item. This gap grows when a compact proof is expanded into long documents, where redundancy, alternative paths, answer leakage, and parametric knowledge can reduce effective dependency order. We introduce CertiHop, a construction and certification framework for long-context necessary-evidence reasoning. Each example begins with an executable dependency tree, expands into a multi-document dossier, and is certified only on the final context. Full must uniquely determine the answer, whereas Zero and every required-leaf leave-one-out intervention must preserve the gold answer as possible but not uniquely determine it. The resulting family supplies paired sufficient/insufficient supervision and parent-level Joint Irreducibility (JI); non-critical interventions distinguish selective sensitivity from generic abstention. Across four Qwen-series backbones, CertiHop yields LongBench-3 point estimates 11.92–36.25 points above equal-size continued-normal training. The controlled A3 comparison yields positive LongBench-3 differences at all four scales. A 9B RULER decline exposes a checklist-completeness boundary. CertiHop turns intended graph dependency into an auditable property of the final text.

Primary area:
datasets and benchmarks
Keywords:
long-context reasoning, multi-hop question answering, evidence necessity, synthetic data generation, supervised fine-tuning, counterfactual evaluation, dataset certification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_fCxBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors seek to build a long-context reasoning benchmark that is free of ambiguity: that is, there is only one evidence tree that leads to a unique answer for a question, and also every element in the evidence is required to ascertain the answer. They propose the CertiHop pipeline in which you begin with a question, evidence tree, and an answer. Then you introduce the surrounding content around it. Their main contribution is a certification method which checks for ambiguity. It consists of three parts: (i) a full check which verifies that only one answer is possible from the full document, (ii) a 'zero' check that verifies that the document without the implanted evidence results in more than one possible answer, and finally (iii) leave-one-out checks which ensure that every claim from the tree is required to establish a unique final response. It is unclear how these checks are performed in practice.

The experimental evaluation section is very difficult to understand due to lack of description of metrics/datasets, table and figure captions, and experimental setup. It is hard to summarize it, even at a high level.

Critical strengths

Focusing on unambiguous reasoning in LLM output answers is a new and possibly interesting idea.

Critical weaknesses
  1. Presentation issues make it very difficult to understand the paper

From the beginning, in both abstract and introduction, a lot of paper-specific terminology is used without being introduced. They introduce the problem reasonably well but do not provide any understanding of their method itself.

Much of the prose only consists of phrases and not complete sentences which further makes the reading difficult. It was hard to understand even after multiple careful readings.

The experimental evaluation section is incoherent and it is hard to understand the quantitative results since it lacks a description of the metrics and datasets. Table and figure captions are also unhelpful. It is unclear how the certification method is implemented in practice. There are several mentions of "training" and "evaluation" but no description of what is the SFT training target.

The paper requires significant rewriting.

  1. Lack of analysis of how models perform on the new benchmark after removing ambiguity.

While it seems that there is some evaluation performed on how the method produces a better dataset without reasoning ambiguities, there is a complete lack of evaluation and analysis of how models perform on their new benchmark. Do models find this dataset harder since there is no ambiguity? This is a necessary component for a paper introducing a benchmark creation methodology.

Questions for authors
  1. What are the main metrics and base datasets being considered? Could you describe them within each table?
  2. How is the certification procedure implemented? Is it using a strong model? What justification exists for the correctness of each of the checks?
  3. Please describe what is the training target and process. What is being trained and how do we interpret the results?
  4. What insights did you get about whether models find your new dataset harder or easier to answer?
  5. How would you improve the presentation of the paper based on previous comments?
Reviewer_hSh2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose CertiHop, a framework (but to my understanding more of a dataset) for better distinguishing between supporting and truly necessary evidence in long-context question answering tasks. Frankly, I feel like the writing of this paper is not up to ICLR standards and actively prevented me from fully understanding the authors' intentions with this research. As it stands, I believe they proposed this framework/dataset (but do not indicate whether they will release it publicly) and evaluated 3 Qwen models and a Qwen MoE model on it, finding that their proposed method of providing Stage 2 SFT data from CertiHop improved performance on one benchmark LongBench-3 and had no significant effect on another benchmark RULER-VT-3.

Critical strengths
  • The improved performance on LongBench-3 is impressive, but there is no adequate justification/explanation for why such big increases are seen on LongBench when there is no improvement, or even decreased performance on RULER. So, this strength becomes more of a weakness once readers dig in and ask questions like will these results hold up under more random seeds, under different model families, etc.
Critical weaknesses
  • The writing is actively making this research indecipherable:
    • The abstract is jargon-heavy and full of terms unique to this sub-area of research. I would recommend a more approachable abstract.
    • High frequency of "clause :/; clause" sentences which I associate with LLM-generated writing. I'd recommend the authors take a pass over this writing to make sure their messaging is being conveyed as intended.
    • I dislike the paragraph from line 69. It starts with "Our contribution is not another taxonomy of increasingly complicated trees. ..." This is super LLM-generated-looking to me. I know the authors disclose the use of AI tools in the writing, but this is not up to ICLR conference standards in my opinion.
    • I find the sentence starting on li 107 to be indecipherable.
    • The conclusion is frankly annoying. I'd direct the authors to resources like this: https://writingcenter.fas.harvard.edu/conclusions to understand how a conclusion should be structured. I'd also recommend that they treat the conclusion like the last impression readers and reviewers are left with from their work, and 3 cursory sentences is not a good impression.
    • This is not a "kind" paper to readers. Even down to little choices like not properly defining "LOO" as "leave-one-out" and leaving it to readers to figure out. I'd recommend that the authors start the writing process from the ground up with the intention that human readers should be able to understand it.
  • Issues with prioritization of space and sensibly structuring the paper:
    • I have issues with the structure of this paper. Putting the Related work as a dense paragraph in Section 7 without properly distinguishing how the authors' work differs or differentiating among the mass of cited works. Their introduction does some of this, but I also have an issue with using the Introduction as a Related Work rather than a standard Introduction without really justifying this somehow (e.g., they don't seem to have needed space savings as they're 1/3 of a page under the page limit).
    • I have an issue with the way section 3 is structured. This is just a listing of all the aspects of the framework, which I consider to be more appropriate for a technical document rather than a conference article. I would suggest that the authors look at how pre-LLM-assistant papers have structured their proposed framework/benchmark contribution sections for guidance. A key distinction would be to properly motivate and build intuition rather than dumping the qualities of the final product on readers.
    • I find the discussion of results in section 6.1 to be severely lacking. The authors just dump results without interpreting them. This is especially difficult because of the very dense, scattershot approach to structuring the paper which means providing the interpretation is essential for readers to understand what's going on.
    • The discussion of Table 3 is even worse, I don't see any justification for why the table is constructed this way, e.g., two metrics per backbone except for Qwen 3.5-27B and Qwen3.6 MoE model which both only get LooGLE, and deltas against LongFaith which was not in Table 2 and hits metrics which are not discussed at all in section 6.1. Why this seed? Why no LongFaith results in Table 1? What is this adding over Table 2 to justify taking up this space which could have been used to properly explain Table 1?
    • Section 6.2 is put in the main text but the results it discusses are entirely in the appendix. This is not up to conference standards. The authors' prioritization for what to include in the main text (a coherent stand-alone narrative for their research and findings) should be revisited.
  • Issues with visualizations, properly designing and contextualizing them:
    • Re: Figure 1. The figure has overlapping and ill-formatted text, e.g., "executable dependency" overruns its block. "e_i" does not subscript i. Figure 1 is also not referenced in the main text in any way, it's just plopped into the paper without textual context.
  • Miscellaneous:
    • The conditions of Equation 1 are ill-formatted.
Questions for authors

None

Reviewer_7BwTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Certihop builds long-context examples to study whether several pieces of evidence are all needed for an answer; it starts from an executable dependency tree and turns the required facts into a set of documents. The checks are applied to the final text after generation using Full/Zero/LOO evidence. With the full evidence, the answer should be unique; removing a required fact should leave it possible but no longer certain. Noncritical changes also check whether the model abstains just because the text has changed. The examples provide paired training targets with enough evidence and with missing evidence. The main behavioral metric checks the full evidence answer and the required evidence removals together.

Critical strengths

(1) The neutral replacement removes support without making the original answer false or forcing another answer. Non critical replacements keep the required evidence available; this checks whether the model refuses simply because some text changed.

(2) A3 keeps the dossier text exactly the same and changes whether the question needs one leaf or several. All 12 seed/ backbone Longbench differences are positive; this give more direct evidence for the role of multi evidence questions than changing the context length.

(3) Another plus is the fixed hash for choosing the training missing leaf (Sec 3.7); this does not use student/verifier scores or observed difficulty. All required leaf removals are still checked before accepting the example.

Critical weaknesses

(1) It is not clear how often the automated checks accept an invalid example. App.H says the fixed 120 parent annotation file is still empty. The extra verifier audits in App.K are also pending. It would be good to give any available audit findings and the observed error rate.

(2) The main JI result uses 1 frozen 9B checkpoint on validation data. Matched controls/held out tests/repeat seeds are still pending. This somehow limits the evidence for the intended necessity behavior.

(3) In A1, Full+ Missing gives lower Longbench scores than Full only on 3 of the 4 backbones. the row budget and output protocol are matched. A3 changes the evidence requirement instead. It would be good to separate gains from multi evidence questions and gains from Missing supervision.

Questions for authors

please see weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.