- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors seek to build a long-context reasoning benchmark that is free of ambiguity: that is, there is only one evidence tree that leads to a unique answer for a question, and also every element in the evidence is required to ascertain the answer. They propose the CertiHop pipeline in which you begin with a question, evidence tree, and an answer. Then you introduce the surrounding content around it. Their main contribution is a certification method which checks for ambiguity. It consists of three parts: (i) a full check which verifies that only one answer is possible from the full document, (ii) a 'zero' check that verifies that the document without the implanted evidence results in more than one possible answer, and finally (iii) leave-one-out checks which ensure that every claim from the tree is required to establish a unique final response. It is unclear how these checks are performed in practice.
The experimental evaluation section is very difficult to understand due to lack of description of metrics/datasets, table and figure captions, and experimental setup. It is hard to summarize it, even at a high level.
Critical strengths
Focusing on unambiguous reasoning in LLM output answers is a new and possibly interesting idea.
Critical weaknesses
- Presentation issues make it very difficult to understand the paper
From the beginning, in both abstract and introduction, a lot of paper-specific terminology is used without being introduced. They introduce the problem reasonably well but do not provide any understanding of their method itself.
Much of the prose only consists of phrases and not complete sentences which further makes the reading difficult. It was hard to understand even after multiple careful readings.
The experimental evaluation section is incoherent and it is hard to understand the quantitative results since it lacks a description of the metrics and datasets. Table and figure captions are also unhelpful. It is unclear how the certification method is implemented in practice. There are several mentions of "training" and "evaluation" but no description of what is the SFT training target.
The paper requires significant rewriting.
- Lack of analysis of how models perform on the new benchmark after removing ambiguity.
While it seems that there is some evaluation performed on how the method produces a better dataset without reasoning ambiguities, there is a complete lack of evaluation and analysis of how models perform on their new benchmark. Do models find this dataset harder since there is no ambiguity? This is a necessary component for a paper introducing a benchmark creation methodology.
Questions for authors
- What are the main metrics and base datasets being considered? Could you describe them within each table?
- How is the certification procedure implemented? Is it using a strong model? What justification exists for the correctness of each of the checks?
- Please describe what is the training target and process. What is being trained and how do we interpret the results?
- What insights did you get about whether models find your new dataset harder or easier to answer?
- How would you improve the presentation of the paper based on previous comments?