- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the relationship between the soundness of model’s evidential reasoning trace at an individual level (local), and the evidence set’s support for a final answer (global). The authors analyze a gap between locally sound evidence trace and support for final correct answer (Local-Global Gap), finding existing faithfulness verifiers are insufficient to capture such gaps. The analysis motivate a training method E-CLOSURE, which generates evidence- and question- edited augmentations to enhance model reasoning which reduces the LGG and improves model performance on multi-hop QA tasks.
Critical strengths
- The authors’ effort to formalize the problem through defining local soundness, global sufficiency and LGG is appreciated. It leads to some interesting observations such as that in regular SFT, local gains can accompany global degradation.
- The proposed learning method’s three objectives are well-motivated by the analysis (evidence-to-step support, question-to-trace alignment, and trace-to-answer closure)
- Human annotation study to measure how often locally sound traces are globally insufficient, across three benchmarks and three models, is valuable.
Critical weaknesses
- Models are tested with thinking disabled: The primary problem the paper’s analysis reveals, and addresses, are path deviation and incomplete reasoning, but I have doubts these are significant sources of failures in models with reasoning post-training, whereas the model evaluates models without thinking enabled.
- Generalizability beyond multil-hop QA: The generality of the contribution is limited to multi-hop QA, and it is unclear whether the LGG is similarly prevalent and measurable in more complex agentic QA or problem solving scenarios involving tools or longer task horizons.
- GPT-5.6 Sol checks local soundness, alignment, and closure when constructing the training traces, then grades the trained models on the same criteria, so there may be some model bias in the results.
Questions for authors
- Could the authors test with thinking enabled for models?
- Answers to generalizability concerns above?
- Why is backbone’s LGG much lower on Musique?
- Answer decoupling means the trace reached the right answer, but the model submits a different one. Would a simpler fix without training, such as a second pass that reads the trace and extracts the answer it reaches, resolve these cases as well as E-CLOSURE does?
2 comments
Response to Reviewer K7sX (2/2)
Weakness 3
GPT-5.6 Sol checks local soundness, alignment, and closure when constructing the training traces, then grades the trained models on the same criteria, so there may be some model bias in the results.
Our Response
To assess this potential evaluator bias, we additionally evaluate Opus 5.5 (High), from a different model family than the GPT-based data constructor, while keeping the annotation guidelines fixed.
We evaluate the same 360 responses from 90 questions used in our human audit. The sample contains 10 questions per dataset–backbone setting, shared across four methods, with all checkpoints using training seed 42.
| Method | Original judge TRR (%) | Human TRR (%) | Opus TRR (%) | Opus LGG (%) |
|---|---|---|---|---|
| SFT | 83.33 | 82.22 | 83.33 | 12.22 |
| CF-SFT | 82.22 | 77.78 | 80.00 | 11.11 |
| RM | 82.22 | 81.11 | 80.00 | 12.22 |
| E-Closure | 86.67 | 84.44 | 85.56 | 4.44 |
E-Closure achieves the highest TRR under all three label sources. Under Opus, its TRR advantage over both CF-SFT and RM is 5.56 percentage points, and it has the lowest LGG rate.
These results provide evidence that the observed reliability gains persist beyond the original evaluator family on this audit sample, consistent with the human judgments. We will include this additional evaluation in the revised manuscript.
Question 3
Why is backbone’s LGG much lower on Musique?
Our Response
The backbone’s lower LGG coexists with substantially lower local soundness. Locally unsound responses are excluded from LGG, so a lower LGG rate alone does not indicate more reliable reasoning. On MuSiQue, the changes from the backbone to E-Closure are:
| Backbone | LSR (%) | TRR (%) | LGG (%) |
|---|---|---|---|
| Qwen3-8B | 43.1 → 84.2 | 32.2 → 65.9 | 11.0 → 18.3 |
| Qwen3-32B | 58.4 → 91.4 | 52.5 → 79.2 | 5.9 → 12.2 |
| Ministral-3-8B | 30.2 → 90.2 | 24.7 → 71.6 | 5.5 → 18.6 |
Here, the improvement in LSR exceeds the improvement in TRR for all three backbones. By definition, LGG = LSR − TRR, so LGG increases even though the proportion of fully reliable responses rises substantially. The lower backbone LGG should therefore be interpreted alongside its much larger proportion of locally unsound responses.
Question 4
Answer decoupling means the trace reached the right answer, but the model submits a different one. Would a simpler fix without training, such as a second pass that reads the trace and extracts the answer it reaches, resolve these cases as well as E-CLOSURE does?
Our Response
We thank the reviewer for suggesting this simple alternative. We tested one-pass answer re-extraction with Qwen3-8B on the same 866 evaluation questions. The untuned model receives the original question and reasoning trace, with the submitted final-answer field removed, and extracts the answer supported by the trace.
We evaluate the overall effect using the same deterministic answer-matching rule for all methods:
| Method | 2WikiMultiHopQA | MuSiQue | HotpotQA | Mean ACC |
|---|---|---|---|---|
| Untuned backbone | 92.45% | 50.20% | 84.98% | 75.88% |
| Backbone + answer re-extraction | 81.65% | 43.92% | 80.48% | 68.69% |
| E-Closure | 99.52% | 76.60% | 87.29% | 87.80% |
The mean gives equal weight to each dataset, and E-Closure uses its existing three-seed results.
Re-extraction lowers mean accuracy by 7.2 percentage points relative to the backbone and remains 19.1 percentage points below E-Closure. Thus, the tested second-pass procedure does not provide a comparable alternative in overall answer accuracy.
We thank the reviewer again for taking the time to provide valuable feedback on our work. We would be glad to further engage in discussion on any additional concerns.
Best regards,
The Authors
Response to Reviewer K7sX (1/2)
Dear Reviewer K7sX,
We thank you for taking the time to review our work. We especially appreciate your positive assessment of our formalization of LGG, the motivation for our three training objectives, and the value of our diagnostic study. Below, we address your concerns about evaluation with thinking enabled, generalizability beyond multi-hop QA, and potential evaluator bias, along with your questions about MuSiQue and simpler inference-time alternatives.
Weakness 1 and Question 1
Our Response
We thank the reviewer for suggesting this additional evaluation. We test the untuned Qwen3-8B with thinking enabled on all 866 evaluation questions. We retain
E-Closurein its original non-thinking configuration and reuse its existing three-seed results.All configurations are evaluated using the same deterministic answer-matching accuracy, with equal weighting across the three datasets:
disabledenableddisabledEnabling thinking improves the backbone’s mean accuracy from 75.88% to 81.61%. Nevertheless,
E-Closureretains a 6.2-percentage-point advantage in mean accuracy, including an 18.6-point advantage on MuSiQue.These results show that the overall accuracy advantage persists when comparing against a thinking-enabled backbone. We will include the results and clarify their scope in the revised manuscript.
Weakness 2 and Question 2
Our Response
We agree that extending these findings to agentic QA, tool use, and longer task horizons requires further validation. We chose multi-hop QA because explicit questions, supplied evidence, and inspectable reasoning traces allow us to separately measure evidence support, question alignment, and answer closure.
Within this setting, our contribution combines a validated diagnosis with an effective training approach. The diagnostic study of 2,598 responses across three benchmarks and three backbones identifies LGG in 46.3% of globally insufficient responses. Guided by this diagnosis,
E-Closureimproves trace reliability rate over RM by 7.8 percentage points across nine settings. These results establish the practical importance of the distinction and the value of supervising the affected dependencies.The distinction also provides a concrete starting point for evaluating agentic QA. A step supported by tool observations does not by itself establish that the agent is pursuing the right goal or has justified its final answer. This motivates checking evidence support, task alignment, and answer closure jointly as agents gather and use information. We will clarify the scope of our evidence and identify validation in these broader settings as future work.
Continued in the next response...