acceptodds
Under review as a conference paper at ICLR 2027

Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case

Abstract

Accident, defect, and outage investigations end with a decision that ordinary question answering never faces: whether the evidence collected so far is enough to close the case. We study this decision for LLM investigators. Given a case brief and an index of evidence items, the model requests evidence, revises its hypotheses, and either closes the case with a conclusion grounded in what it read, or leaves it open and names what is missing. This judgment does not come with capability: an untrained 9B model overstates its evidence in 97% of its answers, and a frontier model that identifies the right cause in 84% of cases still overstates in 91% and closes 17 of the 41 cases whose official finding is “cause undetermined”. Measuring the judgment is itself non-trivial, because the source of a case largely predicts its label: a rule that looks only at the source reaches 83.0 balanced closure accuracy. We therefore evaluate closure with three tests: closure accuracy, reported against the source-only rule and within each source; evidence dependence, which removes the grounds of a conclusion and checks whether the model stops closing; and conclusion and gap quality, a judged checklist of what the model asserts and what it says is missing. We build Nautil, 731 audited investigation cases from aviation, rail, maritime, chemical-safety and vehicle-defect reports and from production server incidents, with teacher trajectories, an out-of-distribution test set and counterfactual evidence versions. Fine-tuning a 9B model on these trajectories makes its closures follow the evidence: removing the grounds lowers its closure rate by 26 points (base model: 6; frontier model: 7), overstatement falls from 97% to 35%, and conclusions that are both correct and not overstated rise from 3% to 43% (frontier model: 9%). Reinforcement learning that rewards only the correctness of the closure decision then raises balanced closure accuracy from 69.2 to 83.3, on par with the teacher, at a measurable cost in evidence dependence. We release the models, the dataset and the evaluation suite.

Primary area:
foundation or frontier models, including LLMs
Keywords:
abstention, abductive reasoning, evidence-grounded reasoning, reinforcement learning with verifiable rewards, multi-turn tool use, investigative reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xJLwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The work studies investigative closure, where an LLM investigator can request evidence from a case over multiple turns, then decide to close the case or leave it open based on the evidence. It introduces Nautil, a benchmark composed of 731 cases from four incident investigation sources, including transport and chemical-safety boards, aviation, vehicle defects, and production server incidents. Each case is labeled with whether the official report determined the cause, with training trajectories generated by a teacher model (GPT-6 Sol).

The authors train Nautil-SFT and Nautil-RLVR from Qwen3.5-9B, and report that Nautil-RLVR improves closure prediction accuracy over untrained base models and frontier LLMs, while decreasing overstatement. In particular, reinforcement learning raises accuracy to 83.3, matching the teacher.

Critical strengths
  1. I find the problem important and well-motivated. An agent can claim a cause as plausible without having enough evidence to establish it, and this distinction matters in real investigative settings. The Nautil benchmark spans multiple scenarios and is well constructed, and it could be a useful resource for future research.
  2. The evaluation suite appears well-designed. Using within-source balanced accuracy and counterfactual tests that increasingly remove evidence helps distinguish models learning investigative closure from predicting labels solely based on source information rather than evidence. Additionally, judging conclusions for overstatement captures when the model names the right cause, but claims it as established instead of merely suggested by evidence.
Critical weaknesses
  1. The scientific novelty and methodological advance of this manuscript given close precedents like CaRT (Liu et al.) and Agentic Abstention (Luo et al.) seems unclear. The manuscript cites these papers, stating that their settings are “single-turn or give the information up front and grade whether to answer.” I find the statement to be inaccurate after reading the cited papers. Both precedents are multi-turn. Agentic Abstention does not provide decisive information upfront. In addition, CaRT also generates synthetic reasoning traces for SFT and RL training.
  2. The Nautil-RLVR checkpoint was selected using 30 test set and 20 OOD cases, which risks overfitting to those cases and thus inflating test set and OOD performance. The motivation for using a screening set containing test cases does not seem clearly explained in the manuscript.
  3. Nautil-RLVR’s is essentially tied with a rule that uses only the source (83.3 versus 83.0, Table 2) on balanced accuracy, so this result alone does not demonstrate the intended capability. The within-source evaluation is more informative, but its estimates depend on very small minority-label groups since the test set has only 2 should-close NHTSA cases, 3 should-close NTSB cases, and 5 should-stay-open board cases. A larger evaluation with adequate representation of both labels within each source would strengthen the central claim.
Questions for authors
  1. What exactly are your contributions with respect to precedents in the field? Please provide a more precise comparison with existing literature.
  2. Why do you choose to use a screening set containing test cases and OOD cases, instead of a separate validation set when selecting the RLVR checkpoint? Can you report Nautil-RLVR on a fresh test set not used in any screening?
  3. Can you provide a larger within-source evaluation with more minority-label cases?
  4. Have you evaluated how human raters judge overstatement? Do the results hold if the teacher’s answer is removed from the judge’s reference?
3 comments

We appreciate the careful reading, including of the cited papers, and the precise questions.

W1/Q1 (precedents). Thank you for pointing this out. Our description of these works was inaccurate, and we have corrected it. Table 1 compares CaRT, Agentic Abstention, AgentAbstain and other work on data, evidence gathering, undetermined outcome and training. In CaRT a correct answer exists once enough is gathered; in agentic abstention the task is made infeasible by construction; in ours the correct outcome can be that the complete record does not establish the cause, as decided by an official investigation.

W2/Q2 (screening on test cases). The validation split has only 42 non-host cases (14 should close), and screening on it alone was noisy; adding test cases was a mistake. We now select on validation only and report every checkpoint on every set (Sec. 5, App. D). The new test set below was never used for selection.

W3/Q3 (tie with the source-only rule; small minority groups). New 100-case test set balanced within every source (boards 11/11, NHTSA 10/10, NTSB 29/29); the source-only rule scores 50 overall and within every source. Balanced accuracy: base 53.3, Nautil-SFT 56.3, Nautil-RLVR 60.7, frontier models 66–73 (Fig. 3; per source in App. E).

Q4 (human raters; teacher's answer). The results hold without the teacher's answer in the judge reference: overstatement is 35% for Nautil-SFT vs. 95% for the base model, and right-and-not-overstated answers are 40% vs. 12% for Gemini (App. H); one secondary claim (Nautil-RLVR addressing alternatives less) weakened and is now qualified in Sec. 6.4. Human rating of overstatement is difficult here: deciding whether a claim goes beyond the evidence requires domain experts such as accident investigators. We therefore use judges from three model families (GPT-6 Sol, Claude Sonnet 5, Grok 4.7). They agree on the cause item in 88% of answers (Cohen's κ ≈ 0.75), and their conclusion scores correlate at Spearman ρ 0.74–0.82; absolute overstatement rates differ, but all three find that trained models overstate far less (App. C).

Thank you again for your thoughtful review. If our response has addressed your concerns, we would be grateful if you would consider updating your rating accordingly. We are happy to clarify any remaining questions.

Thank you for your responses, corrections, and new experiments. I read through all the other reviews and responses, as well as the revised manuscript. With the updated comparison with close precedents, I don't see how this manuscript contributes beyond existing literature in terms of teaching agents when to abstain. From a benchmarking perspective,I also have concerns about the relatively small evaluation subsets and the rather unbalanced sources and labels, so I find it difficult to assess it's broader research value.

With all things considered, I intend to keep my initial rating.

Thank you for reading all reviews, responses and the revised manuscript.

On the contribution: we do not claim a new training method. Our aim is to bring this decision to real-world, high-stakes incident investigations, where “the cause cannot be determined” is the finding of an official investigation, rather than absent or constructed by removing or withholding information.

On the evaluation: we agree that the original test set was small and unbalanced. As a best-effort step, we constructed a 100-case test set balanced within every source, on which a source-only rule scores 50. We will state the remaining limits of size and source composition in the limitations.

Thank you again for your engagement throughout the discussion.

Reviewer_rKptOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript investigates whether LLMs can be appleid to examine complex cases (different than simple Q&As) where multiple evidence is gathered in multi-turn process. The goal is to decide whether a case is closed on it remains opened. The study argues that LLMs are not good investigators by default. To solve this, the study tries to define the task and corresponding evaluations (3 metrics are itroduced), to introduce data (731 audited cases from four sources) , to evaluate existing models, and to propose improved models using supervised fine tuning or reinforcement learning.

Critical strengths

The problem is relevant and under-studied. The dataset is interesting with plans to release it, including real reports and production incidents, multi-turn, with large case files.

Critical weaknesses
  1. I find the technical novelty limited.The methods are standard, LoRA fine-tuning on teacher trajectories + GRPO. The evaluation tests adapt known ideas, namely partial-input baselines and contrast sets / counterfactual edits, to this setting.
  2. All baselines are zero-shot. On the other hand, the proposed models are trained in-distribution on the same sources, answer format and judge-favoured style. No alternative training/fine-tuning approach is evaluated on the same data.
  3. Ground truth and metrics rest on unvalidated LLM steps. An LLM (GPT-5.6 Sol) turns each report into the case file and records the binary label, decides which sentences are withheld as conclusion-revealing, and writes the item titles. Host labels are LLM judgments. The counterfactual grounds are defined by the teacher’s own citations, and conclusion quality is scored by an LLM judge (GPT-6 Sol) that is also the teacher. The same model family thus builds the cases, writes and audits the training trajectories, and grades the answers, so errors may be correlated rather than independent.
Questions for authors
  1. Which components do you consider technically new, beyond applying existing SFT/GRPO recipes and existing counterfactual and partial-input tests?
  2. Current baselines are zero-shot. Can you compare against alternative training approaches on the same data? Without this, it is unclear whether the trajectory-based recipe is what produces the gains.
  3. How is data quality guaranteed? Since the same model family builds the cases, records the labels, writes the training trajectories and grades the answers (W3), is there are need for human verification?
1 comment

Thank you for the careful review and for recognising the relevance of the problem and the data.

W1/Q1 (novelty). We do not claim the training recipe as new. Our contributions are: (i) the task: deciding whether a real investigation record establishes the cause, where “it does not” is a correct outcome set by official investigations; because the source of a case predicts its label in such data, we evaluate it with tests that hold the source fixed; (ii) an open training set built from real investigation reports; (iii) open 9B models trained with SFT and RLVR on it, with an analysis of what training changes: it teaches restraint and grounding rather than cause-finding, and a reward on the verifiable closure decision improves it, whereas adding a judge’s quality score to the reward does not help and can collapse closure.

W2/Q2 (alternative training on the same data). On the same 545 trajectories: SFT with loss on the final answer only (new) stops investigating and almost always leaves cases open (53.2 vs. 68.8 for Nautil-SFT in the same evaluation; 51.7 on the balanced test set); SFT without the hypothesis ledger, without not-closed demonstrations, and with 10% / 50% of the data; three RL rewards from the same SFT model (Sec. 6.4, Table 4). Closure accuracy is scored by a program, not a judge, so it does not reward judge-favoured style.

W3/Q3 (correlated LLM steps; data quality). Each LLM step is cross-checked with other models (General response 3): labels (App. F), judging without the teacher's answer and with Claude Sonnet 5 and Grok 4.7 (App. C, H), incident identity of all non-host cases (App. I). Residual errors are reported: about 4% of references have extraction errors, and 4 of 412 cases mix two incidents (Sec. 7). Human verification needs domain experts; left to future work.

Thank you again for your thoughtful review. If our response has addressed your concerns, we would be grateful if you would consider updating your rating accordingly. We are happy to clarify any remaining questions.

Reviewer_GaBXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies LLM handling of fault investigations, in which a probable cause has to be determined, and it has to be decided whether the current evidence sufficiently supports the cause in order to close the case (closure). The authors contribute a dataset called Nautil consisting of data from seven transportation, server, and chemical domains; an analysis of frontier models on the closure decision; and a study of training 9B parameter models with SFT and RL to improve the closure decision. The authors find that smaller models (Gemini 3.8 Flash and Qwen3.5-9B) do not leave cases open at nearly the same accuracy that they close cases that should be closed, and that removing grounding evidence does not affect those decisions. Training with SFT, they improve closure prediction while improving evidence sensitivity. RL with a simple outcome plus format reward further increases closure balanced accuracy, exceeding that of GPT-6-Sol, but has less evidence sensitivity than the SFT model.

Critical strengths
  1. The discovery that small models (Gemini 3.8 Flash and Qwen3.5-9B), without training, do not retract their closure decisions even as more of the grounding evidence is removed (Figure 3), establishes an important problem.

  2. The discovery that multi-turn interaction to request the evidence results in better accuracy than reading everything (closing 93% of should-close cases versus 61%, line 457) is interesting and surprising. Has this phenomenon been observed in related work or is this a new contribution?

Critical weaknesses
  1. The data resource is not carefully curated.

(a) Various portions are severely unbalanced (NHTSA and NTSB have only 2 and 3 should-close cases (line 322); in the out of domain supplements, "only the medical domain has enough cases on both sides" (line 1101)).

(b) The data is very heterogenous. Why are chemical and transportation cases considered as part of the same "boards" section, whereas NTSB cases (which could also be considered transportation) are separate?

(c) The "host" data is repeatedly excluded, making the reader wonder why it is part of Nautil in the first place.

  1. Other reinforcement learning rewards should have been considered. An important conclusion appears to be a tradeoff between closure decision accuracy and evidence dependence (line 441) with their outcome plus format reward, but rewarding evidence citations instead of the judgment checklist might have changed this result.

  2. The discovery that "not-closed demonstrations are the source of restraint" for closure predictions for a small model (line 427), even after supervised fine-tuning, highlights the insufficiency of the small, unbalanced, and heterogenous training set, because with a more plentiful dataset, systems in the Fact Extraction and VERification (FEVER) challenges have successfully learned to predict missing evidence of a claim [1]. A similar ablation to this one for Nautil-SFT (Table 5) should be done for Nautil-RL.

[1] Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER), pages 1–13, Dominican Republic. Association for Computational Linguistics.

Questions for authors
  1. Could you provide the exact text of the judgment checklist and clarify how it is used to define qconcq_{conc} and qgapq_{gap} in the reward for the checklist variants (line 859)?
3 comments

Thank you for the detailed review and the constructive suggestions.

Strength 2 (own fetching vs. reading everything). To our knowledge it has not been reported; the closest work, with at most 10 evidence items, finds the opposite (Mohiuddin et al., 2026). The benefit shows on large files: on the 20 largest, own fetching reaches 63.0 vs. 47.2 balanced accuracy, since reading everything needs 32k–87k tokens, beyond the 32k training context, while own fetching stays at a median of 16.6k (App. L).

W1a (imbalance). In-distribution: the new source-balanced test set (General response 2). OOD tests whether the investigative behaviour transfers to unseen domains, with the model bringing its own domain knowledge; label balance differs by domain, so results are reported per domain.

W1b (grouping). The four boards' reports mostly establish a cause (110 closed / 30 open); NTSB (19 / 123) and NHTSA (7 / 123) cases mostly stay open. We group by source because base rates differ by source, which is the shortcut we control for. Per-agency counts: App. A, Table 5.

W1c (host data). Report cases alone were too few for training, so we added real production incidents. Their labels are teacher judgments, so host results are reported separately.

W2 (other rewards). A citation-based term was trained: both checklist rewards score whether every causal claim cites read evidence that supports it, and all variants penalise citing unread items. These variants stay on the same trade-off: counted only when the decision is right, accuracy 73.0 with counterfactual effect −20.3, between Nautil-SFT (69.2, −26.1) and Nautil-RLVR (83.3, −15.9); counted always, the policy collapses into never closing (Sec. 6.4).

W3 (not-closed demonstrations; FEVER). RL with the same reward from the SFT model trained without not-closed demonstrations leaves only 22% of undetermined cases open after 30 steps (54.5 vs. 84.6 for Nautil-RLVR in the same evaluation): groups in which every rollout closes carry no learning signal (Sec. 6.4, Table 4). The training set is small; accuracy grows with data (45.9, 64.9, 69.2 at 10%, 50%, 100%). FEVER's "not enough info" is constructed; our open outcome is set by official investigations.

Q1 (checklist). Full text in App. G; its use in the two reward variants in App. B.

Thank you again for your thoughtful review. If our response has addressed your concerns, we would be grateful if you would consider updating your rating accordingly. We are happy to clarify any remaining questions.

Reviewer_GaBXOpenReview ↗

Comments After Author Discussion

I appreciate the experiments with the new label-balanced test sets, as well as the experiment to check the citation reward for RL. Thank you for the explanations of why the datasets were grouped as they were. On this basis, I am raising my overall score from 2 to 3.

I am still skeptical of the value of such a small, source-unbalanced, label-unbalanced heterogenous resource, especially when training is required. I think the conclusions of the paper might be significantly different if a bigger or more homogenous training set could be obtained.

Thank you for reconsidering our work and raising your score.

We agree that the size and heterogeneity of the training set limit how far our conclusions generalise, and we will state this explicitly in the limitations. We are building a larger and more balanced dataset to address this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and Common Concerns

We thank the reviewers for their careful reading and valuable suggestions. A revised PDF is uploaded; page 1 lists the revisions, changed passages are in red.

1. Prior work (xJLw W1/Q1, rKpt W1/Q1). Our sentence on CaRT and Agentic Abstention was wrong: both are multi-turn, and CaRT trains with SFT and RL. Corrected; Table 1 now compares the closest work. In prior work an "undetermined" outcome is either absent (a correct answer exists once enough is gathered) or constructed by removing or withholding information. In ours it is the finding of an official investigation on a real record. We do not claim new training algorithms; the contributions are this task, the data, and an evaluation that separates evidence use from source recognition.

2. Checkpoint selection and a balanced test set (xJLw W2/W3, GaBX W1a). Screening the RLVR checkpoint on a set that included test cases (30 test, 22 non-host; 20 OOD) was a mistake. We now select on validation only and report every checkpoint on every set (Sec. 5, App. D). We added a 100-case test set balanced within every source (boards 11/11, NHTSA 10/10, NTSB 29/29), on which the source-only rule scores 50 and nothing was selected (Sec. 4). Balanced accuracy: base 53.3, Nautil-SFT 56.3, Nautil-RLVR 60.7, lower-cost frontier models 66–70, flagship models 70–73 (Fig. 3).

3. Labels, judges, data (rKpt W3/Q3, xJLw Q4).

  • Labels: other LLMs reading only the official report (four non-GPT families on a 20% sample; six models on the balanced test set) agree with 99–100% of closed labels; disagreements on open labels are mostly about granularity (App. F).
  • Judges: removing the teacher's answer from the judge reference leaves the findings unchanged (App. H); three judges from different families agree on the cause item in 88% of answers (App. C).
  • Identity: of 412 non-host cases, 4 mix two incidents, none in the test split (App. I).
  • Human verification needs domain experts (accident investigators); left to future work.

4. Other changes. Label definition (Sec. 3); two ablations: SFT with loss on the final answer only, and RL from SFT without not-closed demonstrations (Sec. 6.4); per-agency counts (App. A); judge checklist (App. G); example trajectories (App. N). For space, the counterfactual and conclusion-quality tables moved to the appendix (shown as figures), and the former Sec. 6.5 moved to App. K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.