- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The work studies investigative closure, where an LLM investigator can request evidence from a case over multiple turns, then decide to close the case or leave it open based on the evidence. It introduces Nautil, a benchmark composed of 731 cases from four incident investigation sources, including transport and chemical-safety boards, aviation, vehicle defects, and production server incidents. Each case is labeled with whether the official report determined the cause, with training trajectories generated by a teacher model (GPT-6 Sol).
The authors train Nautil-SFT and Nautil-RLVR from Qwen3.5-9B, and report that Nautil-RLVR improves closure prediction accuracy over untrained base models and frontier LLMs, while decreasing overstatement. In particular, reinforcement learning raises accuracy to 83.3, matching the teacher.
Critical strengths
- I find the problem important and well-motivated. An agent can claim a cause as plausible without having enough evidence to establish it, and this distinction matters in real investigative settings. The Nautil benchmark spans multiple scenarios and is well constructed, and it could be a useful resource for future research.
- The evaluation suite appears well-designed. Using within-source balanced accuracy and counterfactual tests that increasingly remove evidence helps distinguish models learning investigative closure from predicting labels solely based on source information rather than evidence. Additionally, judging conclusions for overstatement captures when the model names the right cause, but claims it as established instead of merely suggested by evidence.
Critical weaknesses
- The scientific novelty and methodological advance of this manuscript given close precedents like CaRT (Liu et al.) and Agentic Abstention (Luo et al.) seems unclear. The manuscript cites these papers, stating that their settings are “single-turn or give the information up front and grade whether to answer.” I find the statement to be inaccurate after reading the cited papers. Both precedents are multi-turn. Agentic Abstention does not provide decisive information upfront. In addition, CaRT also generates synthetic reasoning traces for SFT and RL training.
- The Nautil-RLVR checkpoint was selected using 30 test set and 20 OOD cases, which risks overfitting to those cases and thus inflating test set and OOD performance. The motivation for using a screening set containing test cases does not seem clearly explained in the manuscript.
- Nautil-RLVR’s is essentially tied with a rule that uses only the source (83.3 versus 83.0, Table 2) on balanced accuracy, so this result alone does not demonstrate the intended capability. The within-source evaluation is more informative, but its estimates depend on very small minority-label groups since the test set has only 2 should-close NHTSA cases, 3 should-close NTSB cases, and 5 should-stay-open board cases. A larger evaluation with adequate representation of both labels within each source would strengthen the central claim.
Questions for authors
- What exactly are your contributions with respect to precedents in the field? Please provide a more precise comparison with existing literature.
- Why do you choose to use a screening set containing test cases and OOD cases, instead of a separate validation set when selecting the RLVR checkpoint? Can you report Nautil-RLVR on a fresh test set not used in any screening?
- Can you provide a larger within-source evaluation with more minority-label cases?
- Have you evaluated how human raters judge overstatement? Do the results hold if the teacher’s answer is removed from the judge’s reference?
3 comments
Thank you for your responses, corrections, and new experiments. I read through all the other reviews and responses, as well as the revised manuscript. With the updated comparison with close precedents, I don't see how this manuscript contributes beyond existing literature in terms of teaching agents when to abstain. From a benchmarking perspective,I also have concerns about the relatively small evaluation subsets and the rather unbalanced sources and labels, so I find it difficult to assess it's broader research value.
With all things considered, I intend to keep my initial rating.
Thank you for reading all reviews, responses and the revised manuscript.
On the contribution: we do not claim a new training method. Our aim is to bring this decision to real-world, high-stakes incident investigations, where “the cause cannot be determined” is the finding of an official investigation, rather than absent or constructed by removing or withholding information.
On the evaluation: we agree that the original test set was small and unbalanced. As a best-effort step, we constructed a 100-case test set balanced within every source, on which a source-only rule scores 50. We will state the remaining limits of size and source composition in the limitations.
Thank you again for your engagement throughout the discussion.
We appreciate the careful reading, including of the cited papers, and the precise questions.
W1/Q1 (precedents). Thank you for pointing this out. Our description of these works was inaccurate, and we have corrected it. Table 1 compares CaRT, Agentic Abstention, AgentAbstain and other work on data, evidence gathering, undetermined outcome and training. In CaRT a correct answer exists once enough is gathered; in agentic abstention the task is made infeasible by construction; in ours the correct outcome can be that the complete record does not establish the cause, as decided by an official investigation.
W2/Q2 (screening on test cases). The validation split has only 42 non-host cases (14 should close), and screening on it alone was noisy; adding test cases was a mistake. We now select on validation only and report every checkpoint on every set (Sec. 5, App. D). The new test set below was never used for selection.
W3/Q3 (tie with the source-only rule; small minority groups). New 100-case test set balanced within every source (boards 11/11, NHTSA 10/10, NTSB 29/29); the source-only rule scores 50 overall and within every source. Balanced accuracy: base 53.3, Nautil-SFT 56.3, Nautil-RLVR 60.7, frontier models 66–73 (Fig. 3; per source in App. E).
Q4 (human raters; teacher's answer). The results hold without the teacher's answer in the judge reference: overstatement is 35% for Nautil-SFT vs. 95% for the base model, and right-and-not-overstated answers are 40% vs. 12% for Gemini (App. H); one secondary claim (Nautil-RLVR addressing alternatives less) weakened and is now qualified in Sec. 6.4. Human rating of overstatement is difficult here: deciding whether a claim goes beyond the evidence requires domain experts such as accident investigators. We therefore use judges from three model families (GPT-6 Sol, Claude Sonnet 5, Grok 4.7). They agree on the cause item in 88% of answers (Cohen's κ ≈ 0.75), and their conclusion scores correlate at Spearman ρ 0.74–0.82; absolute overstatement rates differ, but all three find that trained models overstate far less (App. C).
Thank you again for your thoughtful review. If our response has addressed your concerns, we would be grateful if you would consider updating your rating accordingly. We are happy to clarify any remaining questions.