- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a neuro-symbolic reasoning framework for long-document understanding, leveraging components such as the Evidence Loop to improve structured evidence retrieval, verification, and reasoning. Experiments on MMLongBench, DocBench, and LongDocURL show improvements in accuracy, recall, and token efficiency.
Critical strengths
-
The motivation is clear. The paper identifies fine-grained evidence retrieval and evidence-grounded reasoning as two bottlenecks in long-document understanding, and designs corresponding mechanisms around evidence sufficiency and reasoning support.
-
The components of the framework are well defined. The paper provides clear terminology and representations for the different modules in the proposed neuro-symbolic reasoning framework.
Critical weaknesses
The paper's formulation of long-document understanding may be somewhat biased or overly simplified. The paper abstracts the task into a clearly separated "evidence retrieval–reasoning" pipeline, which may be too coarse for genuinely complex long-document understanding. Equation (4) appears to rely on the same assumption, namely that the question itself provides enough information to derive most of the evidence that needs to be located in the document. However, in more challenging long-document QA tasks, such as multi-hop reasoning, evidence requirements may emerge dynamically rather than being fully enumerated before reasoning begins. In other words, evidence retrieval and reasoning can be intrinsically coupled. An analysis or discussion of how the proposed framework would handle such tasks, beyond the current setting considered in the paper, would substantially strengthen the contribution.
I have reservations about the novelty of the proposed framework. The paper appears to use a rather broad definition of symbolic, while symbolic reasoning differs considerably from classical notions such as formal logical deduction, program execution, or theorem proving. A substantial portion of the actual reasoning is still carried out by the LLM, and key components such as Evidence Requirements are not replaced by a formal reasoning system. Even the Evidence Assembler mainly serves to organize evidence rather than functioning as a new reasoning module. Overall, the method appears closer to an agentic tool-use protocol built around an LLM (or in the paper's terminology, a "neural model"). The relationship between the proposed framework and symbolic reasoning therefore needs to be clarified further.
The experiments do not provide sufficiently strong evidence for the paper's core claims. For example:
-
Figure 4 appears to present a relatively simple case. The target pages are already explicitly specified in the question, and the model only needs to identify the signatures and count them to obtain the answer. The symbolic component appears to play a rather limited role. From both the task difficulty and the execution trajectory, this example is not very convincing evidence for the advantage of neuro-symbolic reasoning. The repair case in the appendix also involves relatively simple evidence completion, and the paper does not show how the baseline methods fail on the same case.
-
Table 1 lacks results for many existing methods using Qwen3.6-Plus, which leaves relatively few comparisons under the same base model. In addition, have the authors considered evaluating more conventional retrieval models, such as ColPali, together with Qwen3.6-Plus on MMLongBench?
-
Given the concern above that only a limited part of the reasoning process is actually symbolic, it is unclear whether the benefits of the proposed framework would persist when using frontier LLMs with substantially stronger native reasoning capabilities.
Questions for authors
Regarding the details of the proposed neuro-symbolic reasoning framework, I have several questions:
-
The linear combination in Equation (5) appears to merely balance two relevance estimators. The evidence captured by the two estimators is not necessarily complementary, and both could still miss the same relevant evidence.
-
Building on this issue, once in Equation (6) fails to fully cover the ground-truth evidence set, the coverage objective will continue operating over an incomplete evidence set. The repair mechanism triggered by the Evidence Loop also does not seem particularly suited to making the model realize that some evidence has been overlooked. Instead, it mainly guides the model to retrieve evidence that has already been identified as necessary but has not yet been found.
Did I misunderstand the mechanism here?
Other questions for the authors are noted above in the Weaknesses section.
General Response and Summary of Rebuttal Updates
Dear Area Chair and Reviewers,
Thank you for your constructive feedback during the rebuttal period. We summarize our main responses and manuscript updates below.
Detailed responses are available in the corresponding reviewer threads. Thank you for considering our rebuttal and revised manuscript.
Best regards,
Authors of Submission 24461