- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a way (NLI) to proble early errors that could cause cascading effect on later agent executation. Throughout the study, it is shown that NLI is associated with worse downstream task completion performances. NLI is annotated by 5 stages with DeepSeek V4 pro model, presumably stronger than other tested baseline models, which posts a question of whether this is feasible in the realworld deployment when the model might already be the SOTA. Nexus as an external reviewer is proposed to addressed the NLI issue, which is shown to be effective. However, no cost study is provided and no similar baseline compared.
Overall, the findings of NLI is somewhat useful, but lacks nolvety. However the solution provided is not rigorous enough to hit the ICLR paper bar.
Critical strengths
- NLI formuation may be useful for LLM agent verification
- Comprehensive experiments for NLI are provided
Critical weaknesses
- The writing of paper lacks focus. NLI formuation despite potentially useful, is quite well expected. The paper spends a large portion of the narrative about this study (which is fine), but the alleviation approaches are rushed, particularly the study of Nexus lacks baselines and cost study. If the authors would like to focus on the alleviation approach, I would recommend shorten the NLI study as brief motivation, and then expand the proposed methods and the experiments.
- NLI annotations use stronger model to verify weaker models. This does not generalize to realworld setting where the deployed models might be the SOTA already.
- NLI, annotations accuracy is not provided - it purely relied on downstream observation that NLI is worse. This is not rigorous.
Questions for authors
- NLI is a well-known NLP task, do you feel this naming can cause confusion?
- What model does Nexus uses and what's the additional costs?
3 comments
Response to Reviewer iZ2s (2/2)
Annotation for frontier models
Human annotators can apply the rubric directly to interaction records from current and future frontier models. In the new blind audit, two humans independently applied the rubric, and automated labels matched their definite consensus on 64/64 trajectories from the GPT and Claude families. The rubric asks what interpretation the agent formed, whether it acted on that interpretation, and whether its behavior changed after counterevidence. Annotators can judge these behaviors from the full interaction record without solving the task or relying on another model.
Revisions: Appendix B.8 reports the human audit on GPT and Claude trajectories (L970–975). Appendix D explains how to apply the rubric to newer models’ interaction records (L1300–1309).
Annotation accuracy
New human annotation audit. Automated labels closely matched definite human consensus in the new blind audit. We sampled 200 trajectories stratified by benchmark and original label. Two annotators independently labeled all 200 trajectories and agreed on 196/200 (κ = 0.963). The automated labels matched human consensus on 188 of the 190 trajectories with a definitive consensus (98.9% accuracy, 95% CI [96.2%, 99.7%]), with 97.8% precision and 100% recall.
Blinding and the submitted audit. Both annotators applied the same criteria to the full trajectories, with automated labels, model identities, and task scores hidden. We used task scores in the subsequent analysis. Appendix B.8 of the submission also reports an initial blind human audit, with 49 of 50 labels matching and one marked uncertain.
Revisions: Appendix B.8 describes the original audit and expanded blind audit with two independent annotators, including the sampling, hidden information, and agreement results (L964–975).
Q1. The NLI abbreviation
The revised manuscript uses narrative lock-in (N-lock) throughout to avoid ambiguity with the established NLP abbreviation NLI.
Revisions: The revised abstract and introduction introduce the N-lock abbreviation (L11–26, L54–85), which is used throughout the manuscript.
Q2. Which model does Nexus use, and what is the additional cost?
Mentor uses DeepSeek V4 Pro, and Nexus uses slightly more tokens than unsupervised execution and slightly fewer than generic supervision.
Reviewing model and component roles. Only Mentor calls DeepSeek V4 Pro. Nexus uses the same Observer, WorldMap, and Mentor roles described in the submission. Observer records tool calls, returns, and errors in code and assigns evidence IDs. WorldMap applies rules to maintain structured task state. Mentor uses the task, recent tool evidence, and the available WorldMap summary to decide whether to intervene. Generic supervision uses the same reviewing model with the task and recent tool evidence; Nexus also provides the structured state summary.
Tokens per task run. Nexus uses slightly more billed tokens than unsupervised execution and slightly fewer than generic supervision. Across the 225 matched cases, unsupervised execution, generic supervision, and Nexus averaged 711k, 768k, and 764k tokens per task run, respectively, counting all actor and supervisor calls. Nexus used 7.4% more than unsupervised execution and 0.5% fewer than generic supervision; the largest increase over unsupervised execution was 14.9% on AppWorld.
Component text overhead. Appendix C measures component text volume per trigger. It reports 6,295 component text tokens per trigger. Averaged over all agent turns, this text volume is 30.7% of the baseline actor’s tokens per turn. WorldMap accounts for 3,865 text tokens per trigger. It makes no model calls; the summary passed to Mentor counts as Mentor input in the per-run totals.
Using the same metric, the component text overhead was 23.4% across AppWorld and WildClawBench and 23.5% when ClawMark was included. Appendix C reports both results alongside the original figure.
Revisions: The component roles and reviewing model are specified in §5 (L409–431, Figure 6) and Appendix C (L1216–1224). The per-run billed tokens and per-turn component text are reported in §5 (L472–491, Figure 7) and Appendix C (L1197–1214, Table 25, and L1230–1235).
We hope these results address your concerns. We would appreciate your reconsideration of the paper’s informativeness, soundness, and overall assessment in light of its empirical contribution: a protocol that distinguishes verbal acknowledgment, behavioral updating, and strict recovery, the findings it reveals, and the supervision method those findings motivate.
Dear Reviewer iZ2s,
Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.
As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!
Best regards, The Authors
Response to Reviewer iZ2s (1/2)
We thank the reviewer for their time and for recognizing the potential value of this work for agent verification and the breadth of the experiments. We also appreciate the reviewer’s suggestions to sharpen the paper’s focus, expand the description of Nexus, and add baseline and cost comparisons.
Contributions, novelty, and Nexus evaluation
Paper focus and contributions. The paper defines, measures, and tests how to correct narrative lock-in. Two comparisons connect the diagnosis to the design: we compare corrections from the same starting points, then test whether evidence-grounded state supervision improves on generic supervision. The paper makes three contributions. First, we define and operationalize narrative lock-in (N-lock) at the trajectory level, tracing how an incorrect interpretation enters task state, guides subsequent behavior, and persists after counterevidence appears. Second, we annotate and analyze trajectories across four benchmarks and five models to characterize how lock-in forms and propagates, how it relates to task outcomes, and how it responds to correction. Third, we distinguish verbal acknowledgment, behavioral updating, and strict recovery, and use these findings to design and evaluate Nexus, an evidence-grounded approach to state supervision.
Outcome improvement and strict recovery. Concrete counterevidence led to more strict recoveries than an explicit solution path, even though the two improved outcomes equally often. From the same 728 starting points, concrete counterevidence and an explicit solution path each improved 352 outcomes. Counterevidence produced 265 strict recoveries compared with 206 for an explicit solution path (+8.1 percentage points, 95% CI [4.4, 12.0]). Among the improved cases, persistent lock-in and residual damage co-occurred in 40 cases with counterevidence and 97 with an explicit solution path. A separate test on 40 difficult cases also informed the design. After a session reset, a task reference and brief failure summary yielded 8 recoveries, while restoring the original task materials with structured corrective guidance yielded 22.
Relation to prior work and the rationale for Nexus. Our analysis tracks whether a correction changes subsequent actions and whether damage from the original error persists in the artifact. The closest studies in Table 23 of the submission (Table 26 in the revision) examine performance degradation under split instructions, changes in predictive distributions under goal drift, and belief updating in embodied tasks. Counting improved outcomes alone gives the two corrections the same result. This finding motivates Nexus to track counterevidence, artifacts, and outstanding requirements so the agent can update its task state.
Three-arm Nexus evaluation. Nexus outscored same-model generic supervision on every benchmark in the expanded evaluation. Following your suggestion, we extended the evaluation to three benchmarks, reran all three arms on each benchmark, and added results by acting model and total tokens per run.
Native scores are rescaled to 0–100. Overall, generic supervision improves on the unsupervised baseline by 3.8 points, calculated from the unrounded means. Nexus adds 20.5 points over generic supervision with the same reviewing model. Across the three benchmarks, Nexus improved over generic supervision for all five acting models (45 matched cases per model): Claude +19.2, Gemini +19.4, GPT +18.9, Kimi +19.0, and Qwen +26.0 points. The revised §5 describes the components and intervention procedure and reports the baseline comparisons and costs.
Revisions: The contribution statement and recovery comparisons appear in §1 (L70–85), §§4.1–4.3 (L328–406, Figure 5 and Table 2), and Appendix B.9 (L1078–1082). The reset test is in Appendix B.10 (L1124–1138, Table 23). The prior-work comparison is in Appendix D (L1243–1271, Table 26), and the expanded Nexus evaluation is in §5 (L450–491, Figure 7 and Table 4) and Appendix C (L1216–1228). The abstract reports its main result (L28–30).