- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SLEUTH, a prompting-based framework for maintaining within-episode epistemic working memory in language agents. After tool use, the agent records sourced confirmed facts, ranks candidate hypotheses, and maintains open questions that guide subsequent actions. A budget-dependent commitment nudge encourages synthesis after approximately 70% of the turn budget, subject to a minimum fact count. Experiments cover three QA datasets, with MuSiQue separated into three hop-depth settings, using Claude Sonnet 4.6 and GLM-5. The reported results show consistent improvements over the evaluated agent baselines, and component and stopping-policy ablations investigate the contributions of state organization and commitment.
Critical strengths
- Useful empirical gains from a lightweight intervention. On four-hop MuSiQue, SLEUTH improves Sonnet EM from 42.0% to 53.1% over ReAct. The per-question comparison reports 59 wins and 14 losses, consistent with the approximately 11.1-point gain. The paired bootstrap analysis and corresponding F1 improvements provide evidence that the advantage is meaningful and not solely an exact-match formatting effect. A method that achieves these improvements without training has practical value.
- Substantial ablation coverage of the proposed design. The paper evaluates unstructured notes, component subsets, commitment timing, alternative stopping strategies, and retrieval and turn-budget sensitivity. In particular, Facts-Only reaches 49.9% on four-hop MuSiQue, full SLEUTH reaches 53.1%, and removing the nudge reduces SLEUTH to 49.1%. ReAct receives essentially no benefit from the tested nudge. These controls provide an informative starting point for understanding the interaction between evidence organization and commitment.
Critical weaknesses
- The evidence supports the performance improvement more strongly than the central mechanistic claims. The 3.2-point four-hop gain over Facts-Only is important to establishing the value of the full tripartite schema, but the reported significance analysis concerns SLEUTH versus ReAct rather than this increment or the structure-by-nudge interaction. A negligible nudge effect for one unstructured baseline does not establish organized epistemic state as a general necessary condition for effective commitment. Conditional accuracy also depends on which questions a policy elects to answer, and gold-answer occurrence in a history is not sufficient evidence of a completed reasoning chain. Finally, Sonnet's within-MuSiQue gains over ReAct are approximately 11.9, 11.7, and 11.1 points at two, three, and four hops: they remain substantial but do not increase with depth. Comparing HotpotQA to MuSiQue additionally changes the dataset and context distribution. These distinctions matter because the claimed scientific contribution goes beyond a successful prompting recipe.
- Reporting inconsistencies and confounded cross-model interventions weaken the supporting analysis. Under the stated definitions, EM should equal the answered fraction times conditional accuracy. In Tables 2 and 8, EM of 53.1% and timeout of 11.1% imply conditional accuracy of about 59.7%, rather than 58.9%; the no-nudge row similarly implies about 65.2%, rather than 64.6%. The four-hop SLEUTH categories in Table 15 sum to 100.7%, which is too large to be explained by rounding mutually exclusive categories. These discrepancies may be reporting errors, but they affect the central stopping analysis. Separately, the GLM-5 adherence intervention combines prompt adaptation and runtime retries, and Appendix M states that rejected turns do not consume the 20-turn budget. The 19-point improvement, therefore, does not isolate adherence from prompt changes and additional model calls. This concern does not apply to the primary Sonnet results, which use neither adaptation nor runtime enforcement.
Questions for authors
- Please reconcile Tables 2, 8, and 15 using counts from the same evaluated trajectories: correct answers, all answered questions, timeouts, and mutually exclusive error categories. Were any rows computed from different runs or with additional exclusions?
- What are the paired confidence intervals for full SLEUTH versus Facts-Only and for the structure-by-nudge interaction? Please provide the exact nudge text and explain how its fact-count condition is handled for ReAct and whether further searches remain possible. Would an unstructured-note control with matched commitment and source-extraction instructions retain a nudge benefit? This would help distinguish the proposed schema from more general evidence organization.
- What is GLM-5's accuracy with the adapted prompt alone, before runtime enforcement? Does the improvement persist when rejected generations count toward the total model-call budget? Please provide the adapted prompt, validation rules, retry counts, and token accounting that includes rejected responses.
- Were the nudge threshold and the best timings reported in Table 11 selected independently of the final evaluation questions? If not, can you provide a held-out or fixed-threshold result and appropriately qualify the difficulty-scaling claim?
5 comments
I thank the authors for a detailed response and for the substantial new work in Appendix M. Several of my points are now settled, and I want to be explicit about which, before explaining why my assessment has not moved.
Settled. Q1: the reconciliation is complete and convincing — the duplicate rows from the resumed run (407–408 for 405 questions) and the inconsistent timeout definition explain the discrepancies I reported, and with a single definition EM = (1 − timeout) × conditional accuracy holds while the Table 15 categories sum to 100%. Q3: the accounting is now transparent, and I accept that the GLM-5 gain is an effect of the retry policy rather than of adherence. Q2's factual requests — the exact commitment texts, the trigger conditions, the paired intervals — are all answered.
Not settled. My concern was never that these numbers were unreported; it was what they would show once reported. The new controls show it.
First, the matched-instruction Notes control recovers most of the effect the paper attributes to organization: 64.3 / 64.1 / 62.8 / 52.1 / 75.9 against SLEUTH's 66.2 / 66.5 / 63.7 / 53.1 / 76.0. The residual is +1.4 [+0.5, +2.3] pooled, individually significant on two of five splits. Second, and more damaging to the thesis, that residual does not grow with depth: +2.4 at 2-hop, +1.0 at 4-hop, +0.1 on 2Wiki. The central claim is that typed epistemic organization is what makes long chains tractable. If the advantage of the full schema over a flat record carrying the same instructions is largest at two hops and shrinks at four, the data point the other way. Third, the interaction the paper needs — structure × commitment — is +3.7 [−1.0, +8.4], and the authors have correctly withdrawn the necessity claim, which was the paper's distinctive scientific claim.
Two further points on the new experiments. The GLM-5 result, once rejections are charged, is −0.2 on 3-hop; please report the paired interval for that difference, and state in the caption of Table 17 that this configuration is a retry-policy intervention rather than a result about model capability. And the held-out threshold check runs on MuSiQue's training split, where turn 14 reaches 60.8 EM against 53.1 on the evaluation split of the same hop count — a different difficulty regime; on that holdout the selection procedure preferred turn 6 (60.2) to turn 14 (60.8), so the threshold surface is flat. A flat landscape is consistent with the instruction helping and the specific operating point not being identifiable.
On reporting: I would ask the authors to state in the paper that the submitted Tables 2, 8 and 15 were computed on trajectories containing duplicate rows, and whether any qualitative conclusion beyond the ≤0.2 EM shift depended on the affected rows.
Assessment. What remains is a well-executed empirical study of two instructions — source extraction and commitment — plus a small, non-scaling increment from typed structure. That is worth reporting, and the revision is honest about it. But the paper has not only become smaller in what it establishes; the claim that survives no longer matches the frame it is written in, and a reader who takes the title and abstract as the contribution will be misled by them. I am therefore keeping my rating at 2, and I would not have moved it upward even if every remaining question had been closed.
I would genuinely encourage a resubmission reframed around the matched-Notes comparison. The +9.6 to +10.4 points that source extraction and commitment add to a plain notebook is the most useful number in this paper, and it deserves to be the headline rather than the control that deflates the headline.
Response to Reviewer s3nb (follow-up)
We sincerely thank the reviewer for the careful reading of the new experiments and for stating precisely which points are now settled. We deeply appreciate the reviewer's active engagement during the discussion phase, and we are committed to improving the paper regardless of the outcome. We have made the three additions the reviewer requested to the revised PDF (Appendix M).
GLM-5 paired interval and the Table 17 caption. With every rejected generation charged to the budget, the paired 95% bootstrap interval for SLEUTH minus ReAct on GLM-5 is [−3.8, +3.3] on 3-hop, so the two are statistically indistinguishable there; the corresponding intervals are [+1.1, +6.1] on 2-hop and [+0.2, +10.6] on 4-hop. We added these intervals to the caption of Table 17, which now also states that the runtime check is a retry-policy intervention on how generations are executed, not a result about model capability.
Deduplicated trajectories. We agree with the reviewer that the paper should state this explicitly. Appendix M now states that the submitted Tables 2, 8 and 15 were computed from trace files in which three SLEUTH configurations (commitment at turn 8, commitment at turn 14, no commitment) contained duplicate records from a resumed run (407–408 records for 405 questions), and that episodes answering on the final turn were counted as timeouts. No qualitative conclusion depended on the affected rows: reported EM changes by at most 0.2 points, and every comparison keeps its direction. With the deduplicated trajectories and one timeout definition, SLEUTH has a lower timeout rate and higher conditional accuracy than ReAct on every MuSiQue split (4-hop: 8.1% vs. 17.5% and 57.8% vs. 50.9%), and commitment at turn 14 still gives the highest EM in the sweep.
The matched-Notes comparison. We agree with the reviewer that the gain from source extraction and commitment over a plain notebook (+9.6 [+7.5, +11.7], +9.7 [+7.2, +12.2] and +10.4 [+6.9, +13.8] on 2-, 3- and 4-hop) is among the most useful results of the study, and we will give it a more prominent place in the final version, alongside the comparison of the complete configuration with the matched control (+1.4 [+0.5, +2.3] pooled). We also agree with the reviewer's reading of the held-out threshold check: it shows that the commitment instruction keeps its benefit on unseen questions, both at an independently selected threshold and at turn 14 (+8.0 [+4.2, +11.8] and +8.5 [+4.8, +12.2] over ReAct), and the revised appendix states that it does not establish turn 14 as optimal.
We thank the reviewer again for the constructive and detailed feedback throughout the discussion, which has made the paper's claims considerably more precise.
Gentle reminder: author–reviewer discussion period ending in a few days
Dear Reviewer s3nb,
We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.
In our response, we reconciled Tables 2, 8 and 15 from deduplicated trajectories with one timeout definition (reported EM changes by at most 0.2 points); gave the exact commitment texts; reported paired confidence intervals for SLEUTH vs. Facts-Only and the structure × commitment interaction; added a matched-instruction Notes control; separated prompt adaptation from enforcement on GLM-5; and ran a held-out check of the commitment threshold on MuSiQue training-split questions, where turn 14 keeps its advantage over ReAct (+8.5 [+4.8, +12.2]).
If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.
Thank you again for the time and effort you put into reviewing our work.
Best regards,
Authors