acceptodds
Under review as a conference paper at ICLR 2027

Let It Go: Narrative Lock-in in Open-ended Real-Life AI Agents

Abstract

Long-horizon, tool-using agents must maintain an understanding of task goals, evidence, and current state across many actions. Yet we observe that when an early, plausible but incorrect interpretation becomes the task state on which later decisions are based, agents may continue to plan, verify, use tools, and produce deliverables around that error even as counterevidence accumulates. We call this trajectory-level failure Narrative Lock-in (NLI). We study how these interpretations shape subsequent execution and whether correcting them produces behavioral recovery. To characterize this phenomenon, we analyze 6,035 trajectories from four benchmarks and five models using a five-dimension rubric that spans narrative formation and behavioral propagation. Within the same task and model, NLI trajectories score lower in 34 of 42 matched groups, with mean completion of 0.207 compared with 0.623 for non-NLI trajectories. We then evaluate six correction strategies on a complete matched cohort of 728 selected cases. Strict recovery ranges from 10.7% to 36.4%, while verbal acknowledgment substantially overestimates behavioral recovery. This gap suggests that asking an agent to reconsider often leaves the task state governing its later actions unchanged. Motivated by this finding, we evaluate Nexus, a unified external state-supervision loop that uses preserved evidence to identify conflicts and guide task-state revision. On 75 ClawMark cases, Nexus records 44 wins, 15 ties, and 16 losses against the original trajectories. Together, these results characterize NLI as a measurable failure of long-horizon agent execution and show that evidence-grounded state supervision offers a practical response.

Primary area:
datasets and benchmarks
Keywords:
long-horizon agent, Agent failure, Agent trajectory analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iZ2sOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a way (NLI) to proble early errors that could cause cascading effect on later agent executation. Throughout the study, it is shown that NLI is associated with worse downstream task completion performances. NLI is annotated by 5 stages with DeepSeek V4 pro model, presumably stronger than other tested baseline models, which posts a question of whether this is feasible in the realworld deployment when the model might already be the SOTA. Nexus as an external reviewer is proposed to addressed the NLI issue, which is shown to be effective. However, no cost study is provided and no similar baseline compared.

Overall, the findings of NLI is somewhat useful, but lacks nolvety. However the solution provided is not rigorous enough to hit the ICLR paper bar.

Critical strengths
  • NLI formuation may be useful for LLM agent verification
  • Comprehensive experiments for NLI are provided
Critical weaknesses
  • The writing of paper lacks focus. NLI formuation despite potentially useful, is quite well expected. The paper spends a large portion of the narrative about this study (which is fine), but the alleviation approaches are rushed, particularly the study of Nexus lacks baselines and cost study. If the authors would like to focus on the alleviation approach, I would recommend shorten the NLI study as brief motivation, and then expand the proposed methods and the experiments.
  • NLI annotations use stronger model to verify weaker models. This does not generalize to realworld setting where the deployed models might be the SOTA already.
  • NLI, annotations accuracy is not provided - it purely relied on downstream observation that NLI is worse. This is not rigorous.
Questions for authors
  • NLI is a well-known NLP task, do you feel this naming can cause confusion?
  • What model does Nexus uses and what's the additional costs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer iZ2s (1/2)

We thank the reviewer for their time and for recognizing the potential value of this work for agent verification and the breadth of the experiments. We also appreciate the reviewer’s suggestions to sharpen the paper’s focus, expand the description of Nexus, and add baseline and cost comparisons.

Contributions, novelty, and Nexus evaluation

Paper focus and contributions. The paper defines, measures, and tests how to correct narrative lock-in. Two comparisons connect the diagnosis to the design: we compare corrections from the same starting points, then test whether evidence-grounded state supervision improves on generic supervision. The paper makes three contributions. First, we define and operationalize narrative lock-in (N-lock) at the trajectory level, tracing how an incorrect interpretation enters task state, guides subsequent behavior, and persists after counterevidence appears. Second, we annotate and analyze trajectories across four benchmarks and five models to characterize how lock-in forms and propagates, how it relates to task outcomes, and how it responds to correction. Third, we distinguish verbal acknowledgment, behavioral updating, and strict recovery, and use these findings to design and evaluate Nexus, an evidence-grounded approach to state supervision.

Outcome improvement and strict recovery. Concrete counterevidence led to more strict recoveries than an explicit solution path, even though the two improved outcomes equally often. From the same 728 starting points, concrete counterevidence and an explicit solution path each improved 352 outcomes. Counterevidence produced 265 strict recoveries compared with 206 for an explicit solution path (+8.1 percentage points, 95% CI [4.4, 12.0]). Among the improved cases, persistent lock-in and residual damage co-occurred in 40 cases with counterevidence and 97 with an explicit solution path. A separate test on 40 difficult cases also informed the design. After a session reset, a task reference and brief failure summary yielded 8 recoveries, while restoring the original task materials with structured corrective guidance yielded 22.

Relation to prior work and the rationale for Nexus. Our analysis tracks whether a correction changes subsequent actions and whether damage from the original error persists in the artifact. The closest studies in Table 23 of the submission (Table 26 in the revision) examine performance degradation under split instructions, changes in predictive distributions under goal drift, and belief updating in embodied tasks. Counting improved outcomes alone gives the two corrections the same result. This finding motivates Nexus to track counterevidence, artifacts, and outstanding requirements so the agent can update its task state.

Three-arm Nexus evaluation. Nexus outscored same-model generic supervision on every benchmark in the expanded evaluation. Following your suggestion, we extended the evaluation to three benchmarks, reran all three arms on each benchmark, and added results by acting model and total tokens per run.

BenchmarkPairsUnsupervisedGeneric supervisionNexusNexus gain over generic [95% CI]W/T/L
ClawMark7523.625.145.8+20.7 [14.7, 26.7]52/20/3
AppWorld7542.952.175.4+23.3 [14.9, 31.8]33/39/3
WildClawBench7525.626.544.0+17.6 [6.9, 28.8]41/31/3
Overall22530.734.655.0+20.5 [15.5, 25.6]126/90/9

Native scores are rescaled to 0–100. Overall, generic supervision improves on the unsupervised baseline by 3.8 points, calculated from the unrounded means. Nexus adds 20.5 points over generic supervision with the same reviewing model. Across the three benchmarks, Nexus improved over generic supervision for all five acting models (45 matched cases per model): Claude +19.2, Gemini +19.4, GPT +18.9, Kimi +19.0, and Qwen +26.0 points. The revised §5 describes the components and intervention procedure and reports the baseline comparisons and costs.

Revisions: The contribution statement and recovery comparisons appear in §1 (L70–85), §§4.1–4.3 (L328–406, Figure 5 and Table 2), and Appendix B.9 (L1078–1082). The reset test is in Appendix B.10 (L1124–1138, Table 23). The prior-work comparison is in Appendix D (L1243–1271, Table 26), and the expanded Nexus evaluation is in §5 (L450–491, Figure 7 and Table 4) and Appendix C (L1216–1228). The abstract reports its main result (L28–30).

AuthorsOpenReview ↗

Response to Reviewer iZ2s (2/2)

Annotation for frontier models

Human annotators can apply the rubric directly to interaction records from current and future frontier models. In the new blind audit, two humans independently applied the rubric, and automated labels matched their definite consensus on 64/64 trajectories from the GPT and Claude families. The rubric asks what interpretation the agent formed, whether it acted on that interpretation, and whether its behavior changed after counterevidence. Annotators can judge these behaviors from the full interaction record without solving the task or relying on another model.

Revisions: Appendix B.8 reports the human audit on GPT and Claude trajectories (L970–975). Appendix D explains how to apply the rubric to newer models’ interaction records (L1300–1309).

Annotation accuracy

New human annotation audit. Automated labels closely matched definite human consensus in the new blind audit. We sampled 200 trajectories stratified by benchmark and original label. Two annotators independently labeled all 200 trajectories and agreed on 196/200 (κ = 0.963). The automated labels matched human consensus on 188 of the 190 trajectories with a definitive consensus (98.9% accuracy, 95% CI [96.2%, 99.7%]), with 97.8% precision and 100% recall.

Blinding and the submitted audit. Both annotators applied the same criteria to the full trajectories, with automated labels, model identities, and task scores hidden. We used task scores in the subsequent analysis. Appendix B.8 of the submission also reports an initial blind human audit, with 49 of 50 labels matching and one marked uncertain.

Revisions: Appendix B.8 describes the original audit and expanded blind audit with two independent annotators, including the sampling, hidden information, and agreement results (L964–975).

Q1. The NLI abbreviation

The revised manuscript uses narrative lock-in (N-lock) throughout to avoid ambiguity with the established NLP abbreviation NLI.

Revisions: The revised abstract and introduction introduce the N-lock abbreviation (L11–26, L54–85), which is used throughout the manuscript.

Q2. Which model does Nexus use, and what is the additional cost?

Mentor uses DeepSeek V4 Pro, and Nexus uses slightly more tokens than unsupervised execution and slightly fewer than generic supervision.

Reviewing model and component roles. Only Mentor calls DeepSeek V4 Pro. Nexus uses the same Observer, WorldMap, and Mentor roles described in the submission. Observer records tool calls, returns, and errors in code and assigns evidence IDs. WorldMap applies rules to maintain structured task state. Mentor uses the task, recent tool evidence, and the available WorldMap summary to decide whether to intervene. Generic supervision uses the same reviewing model with the task and recent tool evidence; Nexus also provides the structured state summary.

Tokens per task run. Nexus uses slightly more billed tokens than unsupervised execution and slightly fewer than generic supervision. Across the 225 matched cases, unsupervised execution, generic supervision, and Nexus averaged 711k, 768k, and 764k tokens per task run, respectively, counting all actor and supervisor calls. Nexus used 7.4% more than unsupervised execution and 0.5% fewer than generic supervision; the largest increase over unsupervised execution was 14.9% on AppWorld.

Component text overhead. Appendix C measures component text volume per trigger. It reports 6,295 component text tokens per trigger. Averaged over all agent turns, this text volume is 30.7% of the baseline actor’s tokens per turn. WorldMap accounts for 3,865 text tokens per trigger. It makes no model calls; the summary passed to Mentor counts as Mentor input in the per-run totals.

Using the same metric, the component text overhead was 23.4% across AppWorld and WildClawBench and 23.5% when ClawMark was included. Appendix C reports both results alongside the original figure.

Revisions: The component roles and reviewing model are specified in §5 (L409–431, Figure 6) and Appendix C (L1216–1224). The per-run billed tokens and per-turn component text are reported in §5 (L472–491, Figure 7) and Appendix C (L1197–1214, Table 25, and L1230–1235).

We hope these results address your concerns. We would appreciate your reconsideration of the paper’s informativeness, soundness, and overall assessment in light of its empirical contribution: a protocol that distinguishes verbal acknowledgment, behavioral updating, and strict recovery, the findings it reveals, and the supervision method those findings motivate.

Dear Reviewer iZ2s,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Reviewer_L18rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Narrative Lock-in (NLI), a trajectory-level failure in which an early incorrect interpretation guides an agent’s subsequent plans, actions, or artifacts despite counterevidence. The authors introduce a five-dimensional annotation framework and analyze 6,035 trajectories across four benchmarks and five models. Matched task–model comparisons associate NLI with lower task-completion scores. They then compare six correction strategies from the same source states across 728 selected cases, distinguishing immediate responses, behavioral updates, and a strict recovery criterion. Strict recovery rates range from 10.7% to 36.4%, with concrete counterevidence achieving the highest aggregate rate. The paper also introduces Nexus, an external supervision loop that records execution evidence, maintains structured task state, and provides corrections when conflicts arise. On 75 ClawMark task–model cases, Nexus records 44 wins, 15 ties, and 16 losses against the original runs, with an average task-score improvement of approximately 11.1 percentage points. The main contributions are a behavioral operationalization and empirical characterization of NLI, matched correction experiments, and an initial evaluation of evidence-grounded external state supervision.

Critical strengths
  1. The paper presents a systematic account of persistent erroneous interpretation and its association with task outcomes. The five-dimensional rubric characterizes how and where an erroneous interpretation can be incorporated in plans, actions, and artifacts, and how it can persist in the face of counterevidence. The analysis applies to four benchmarks and five models, with outcomes evaluated in isolation from their NLI labels. In same-task, same-model comparisons within this pooled set, NLI runs achieve lower task scores than non-NLI runs in 34 of 42 matching groups. While these matched comparisons are dominated by the contribution of AppWorld, they represent an important step beyond illustrative examples, providing evidence that the patterns identified are associated with worse execution outcomes.
  2. The matched correction experiments characterize immediate responses, behavioral updates, and strict recovery. Within 728 selected cases, six strategies persist from the same source context and world state-enabling comparisons on a shared cohort-and consider subsequent actions and available action outcomes, not just immediate response. Concrete counterevidence leads to strict recovery in 36.4% of cases compared to 10.7% for self-checked examples. By distinguishing outcome improvement and strict recovery, the analysis further shows that improvements need not involve complete resolution of the previously persistent erroneous interpretation pattern and resulting damage. The experiments provide practical guidance for correction strategy design and enable analyses of whether benefits generalize to subsequent execution.
Critical weaknesses
  1. The correction evaluation mixes outcome evidence sources leaving unclear the robustness of the recovery rates and comparisons between strategies. Appendix B.9 describes that official benchmark results are used when an intervention run reaches its scorer; otherwise, the outcome improvement is judged by an LLM from the available continuation and artifact. The paper does not report the coverage of official-scores by condition or continuation lengths or termination status by condition, and does not separately validate the recovery judgments, only the 50-case blinded human check which concerns NLI classification. Consequently, it is not clear if strict-recovery rates and strategy rankings are robust to differences in execution completeness or evaluator judgments. This casts doubt on the strength of the central conclusions about the effectiveness and limitations of correction.
  2. The Nexus evaluation does not sufficiently establish its added value over alternative forms of external supervision. The comparison on original runs on 75 ClawMark cases provides some promising evidence, but lacks fresh-run controls or alternatives evaluated under matched execution budgets, such as generic external correction or task-ledger supervision. The comparison on 29 cases overlapping with the six correction conditions uses different starting points and outcome measures for evaluation, making it impossible to directly assess relative effectiveness. The results therefore only provide evidence for the potential of the integrated system, but not how much benefit comes specifically from the evidence-grounded state maintenance rather than additional supervision or execution resources.
Questions for authors
  1. Robustness of the correction evaluation (clarification and expanded analysis using existing records): per correction condition, what proportion of runs got (or rather, were given) official benchmark scores vs. LLM-judged outcome improvement, and how did the run lengths and termination statuses differ? Where comparable completed runs exist, do the major strategy comparisons remain consistent when looking at official scores to evaluate outcome improvements? Have the strict-recovery judgments been subjected to any sort of human validation beyond the NLI classification check? These considerations would substantially impact my confidence in the reported recovery rates and strategy rankings.
  2. Attribution of Nexus's gains (clarification request; expanded controls if available):Did the original and Nexus runs receive the same task inputs/environment initialization/agent execution budgets, and what determined the 15 ClawMark tasks used? Do existing experiments include fresh baseline reruns or alternative external supervision under comparable resource budgets? If so, how do those results compare with Nexus? I'm aware that the 29-case overlap used different initial conditions and outcome measurements, but I think that understanding what controlled comparisons are available would meaningfully impact my perception of Nexus's added value.
4 comments
AuthorsOpenReview ↗

Response to Reviewer L18r (1/2)

We thank the reviewer for their time and for recognizing the value of the five-dimensional trajectory analysis and the matched correction design, which distinguishes immediate responses, behavioral updates, and strict recovery. We also appreciate the reviewer’s specific suggestions for evaluating the recovery results and Nexus. We added an analysis of continuation lengths and official scoring coverage, an independent human audit of strict recovery, and a new three-arm Nexus comparison.

Q1. Continuation, official scoring, and strict recovery validation

Both strategy rankings held on the common officially scored subset, and we checked strict-recovery judgments in a separate blind audit.

Official scoring coverage. Nearly all runs in each arm received official benchmark scores. Official scores were available for 4,354/4,368 runs (99.7%) across the four benchmarks. For the remaining 14/4,368 runs (0.3%), the LLM judge assessed outcome improvement from the available records and artifacts.

Continuation length and termination. Mean continuation lengths were similar across arms within each benchmark. All six correction arms followed the same protocol from the same 728 starting points. In the three tool-using benchmarks, Stage B required at least one concrete action and continued until task termination or the budget limit. Each of the 297 CL-bench-Life cases continued for one QA turn. The agent gave a revised answer and a justified next step, and the benchmark evaluator scored the revised answer. The first table gives mean continuation lengths by arm and the recorded issues for runs without official scores.

ArmClawMark mean callsWildClawBench mean callsAppWorld mean stepsCL-bench-Life officialTool benchmarks officialAll officialRuns without an official score
SC794536297/297431/431728/7280
MR804735297/297428/431725/7283 (2 no-progress loops, 1 API error with repetition)
L1804637297/297425/431722/7286 (2 stopped after one action, 2 truncated outputs, 1 no-progress loop, 1 API error)
L2785134297/297430/431727/7281 (1 API error)
L3784536297/297429/431726/7282 (1 API error, 1 no-progress loop)
L4794440297/297429/431726/7282 (1 no-progress loop, 1 truncated output)
Total / mean≈79≈46≈361,782/1,7822,572/2,5864,354/4,36814

Revisions: Appendix B.9 clarifies the continuation protocol and reports length, official-score coverage, and recorded issues for unscored runs by arm (L1014–1041, Table 19). It specifies the scoring source for each recovery criterion (L1064–1072).

Six-arm rankings on matched runs with official scores. The six-arm rankings for outcome improvement and strict recovery held when all arms had official scores. Based on the recorded judgments, both rankings on the 718 starting points with official scores in every arm matched those on the full set of 728 starting points.

MetricAll 728 starting points (counts in parentheses)718 starting points with official scores in every arm
Outcome improvementL3 (352) = L4 (352) > MR (297) > L2 (279) > L1 (254) > SC (208)L3 = L4 > MR > L2 > L1 > SC
Strict recoveryL3 (265) > L4 (206) > L2 (183) > MR (152) > L1 (136) > SC (78)L3 > L4 > L2 > MR > L1 > SC

As described in Appendix B.9, we used official benchmark scores to determine whether the outcome improved when scores were available. Otherwise, the LLM judge compared the available continuation and artifacts with the source trajectory. In every arm, the LLM judge assessed whether the original interpretation still guided behavior and whether residual damage remained. The independent human audit below also checked these judgments.

Concrete counterevidence (L3) yielded a strict recovery rate 8.1 percentage points higher than an explicit solution path (L4) in both sets (full set 95% CI [4.4, 12.0]).

Revisions: The six-arm comparison on the common official-score subset is reported in Appendix B.9 (L1078–1091, Table 20).

AuthorsOpenReview ↗

Response to Reviewer L18r (2/2)

Q1 (continued)

Independent human validation of strict recovery. An independent blind audit reproduced the original strict-recovery decisions whenever both annotators agreed on all three criteria with definite labels. The original 50-case blind audit checked N-lock classification. Two annotators independently reviewed 239 runs sampled by correction condition, benchmark, and original recovery label. They saw the trajectory preceding the intervention, the intervention text, and the output that followed. They assessed outcome improvement, whether the original interpretation still guided behavior, and whether residual damage remained. Condition labels, original scores, and original recovery judgments were hidden. Inter-annotator κ was 0.956, 0.935, and 0.891, respectively. On the 169 runs where both annotators gave the same definite answer on all three criteria, the human-derived strict recovery decision matched the original decision on 169/169. In the other 70 runs, uncertainty or disagreement affected at least one criterion. Among those 70 runs, the two annotators agreed on 133 of the 143 criterion-level comparisons (93.0%) for which both gave definite answers.

Revisions: The independent audit of the three strict-recovery criteria is reported in Appendix B.9 (L1094–1101).

Q2. ClawMark task selection and the matched Nexus comparison

The original ClawMark settings were matched, and fresh controls show that Nexus outscored generic supervision under comparable execution budgets.

Original ClawMark settings. The original and Nexus runs used the same task inputs, environment initialization, and agent execution budgets. The submitted comparison used the archived original runs as the baseline.

Task selection. Tasks qualified for sampling if they had a confirmed N-lock trajectory. We expanded the evaluation by retaining the 15 ClawMark tasks sampled by category and adding 15 AppWorld tasks sampled by split and 15 WildClawBench tasks sampled by category, applying the same eligibility rule (at least one confirmed N-lock trajectory) and deduplicating by task ID in all three benchmarks.

Fresh matched controls. Nexus outscored same-model generic supervision on every benchmark at comparable total token use. We crossed each task with five acting models and ran all three conditions afresh: unsupervised execution, generic external supervision, and Nexus. For each task–model pair, the three conditions used the same input, environment initialization, execution budget, and benchmark evaluator. Both supervisors used DeepSeek V4 Pro and received the task and recent tool evidence. Nexus’s Mentor also received structured task state.

BenchmarkMatched casesUnsupervisedGeneric supervisionNexusNexus gain over generic [95% CI]W/T/L
ClawMark7523.625.145.8+20.7 [14.7, 26.7]52/20/3
AppWorld7542.952.175.4+23.3 [14.9, 31.8]33/39/3
WildClawBench7525.626.544.0+17.6 [6.9, 28.8]41/31/3
Overall22530.734.655.0+20.5 [15.5, 25.6]126/90/9

On the 0–100 native score scale, generic supervision improved on unsupervised execution by 3.8 points, calculated from the unrounded means. Nexus added 20.5 points [95% CI 15.5, 25.6] over generic supervision at comparable total token use. Including actor and supervisor calls, the three conditions averaged 711k, 768k, and 764k tokens per run, respectively. This comparison measures what the integrated evidence tracking and state supervision add over generic feedback using the same reviewing model.

Revisions: The archived ClawMark comparison and overlap analysis are documented in §5 (L433–449, Table 3) and Appendix C (L1169–1188, Table 24). The fresh three-arm evaluation is reported in §5 (L450–491, Figure 7 and Table 4), with task selection, matching, and token accounting in Appendix C (L1216–1235). The abstract reports the main result of the expanded comparison (L28–30).

Thank you for identifying these specific checks. The revised manuscript includes the continuation analysis, the strict recovery audit, and the new controls. We hope these results address your concerns and support reconsideration of the paper’s soundness and overall assessment.

Thank you for the detailed responses and additional experiments. The clarifications, independent human audit, and matched Nexus comparison have addressed my main concerns. I have raised my score accordingly.

We sincerely thank you for your constructive feedback, the time and effort you devoted to reviewing our paper, and your positive assessment and support. We will incorporate the valuable suggestions and refinements from our discussion into the final version of the manuscript.

Reviewer_ay8eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents an analysis and proposed (I think!) intervention for counteracting what the authors term "narrative lock-in" in agentic AI systems, when a false supposition early in inference becomes reinforced through subsequent chain of thought and tool calls until even requesting correction from the user or showing contradictory evidence cannot change the course of the agent reasoning. The authors present (I think!) a system for maintaining memory outside of the agent's internal memory system to assist in presenting contradictory evidence during reasoning that supports the true, versus misunderstood, task parameters, and provide analysis showing that this contradictory evidence can be more effective than requests to reconsider or user inputs pointing out incorrect assumptions further along agent reasoning trajectories.

Critical strengths

I think that the analysis presented in this paper is likely interesting to a subset of the community working on LLM agents.

Critical weaknesses

I worry broadly that analyses like these will simply not be relevant to the scientific community within a few months as the benchmarks and agents under investigation age out, and the community moves on to the next hot thing.

Questions for authors

In places, the narrative is hard to follow and starts feeling like LLM-thick sentences (e.g, the opening paragraph of 4.2 has one clearly written sentence surrounded by overly long and obtuse ones, but I like the central one! "The increase through concrete evidence shows that specific contradiction is more useful than a generic request to reconsider.")

In Section 5 the narrative still does not make clear whether the "Nexus" design is a contribution of this paper. The wording now makes me think it probably isn't? But it should be extremely obvious whether it is or not! There should be a citation if it's not, and a statement about it being formed/designed by this work if it's not.

"Narrative Lock-in (NLI)" NLI is a well-established three-letter acronym in NLP already meaning "Natural Language Inference". Suggest the authors to revisit the term to avoid overloading it.

Abstract could identify Nexus as a contribution from this paper, which is still not clear clear yet at the end of the Intro.

In 2.1, n_t as presented as fact. Is it? Or is the task state maintained by agent systems a hypothesis of the authors? If it's always present, why not define it here? Does the paper narrative only apply to agentic systems with an explicit n_t? The level of detail in 2.1 is too high I feel.

Appendix B.8 does not make clear whether human annotators were involved at any point during the described validation of automated annotations. My read suggests "no". How can we move forward with these annotations from LLMs without human validation?

Figure 3 seems like a lot of information about benchmark data points that you /don't/ use, right? If I understood correctly, part of Figure 2 shows where the trajectories come from per benchmark, but Figure 3 backs up to give us an overview of the benchmarks' composition, right? Why does this paper's main content need to tell the reader about the composition of the benchmarks that includes lots of information about tasks/problems that do not appear in the trajectories considered in this paper?

3.1 suggestion: remind us what the "permissive first pass" is before jumping into this stat.

Figure 4: Part of this is just that the paper results are dense (I think in a good way) and there's too many ICLR papers this year sorry. Can you remind us where and how "damage" is measured in 4(c)? Are these numbers native to these benchmarks or annotated via LLM-as-judge on outcomes.

Figure 5 is there room to squeeze the legend description for L into 5(a)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ay8e (1/2)

Thank you for reading the paper so closely and for your constructive feedback. It was encouraging to hear that the trajectory analysis could be useful to researchers working on LLM agents. We especially appreciate your comments on the opening of §4.2, including the sentence you singled out on concrete counterevidence, and your suggestions for presenting Nexus and the figures more clearly.

Relevance as models and benchmarks change

The N-lock criteria and their implications for agent harnesses apply as models and benchmarks change because they concern how agents act on counterevidence during long autonomous runs. Building agents that can operate autonomously and reliably over long periods is a central goal of agent research. N-lock characterizes a behavioral failure mode in such runs. This failure mode warrants evaluation on long-horizon benchmarks and with frontier models. We observed it in all five models we evaluated, including frontier models. Users can also recognize the pattern in their own interactions by looking at what the agent does after seeing counterevidence. In long autonomous runs, the harness should retain tool evidence, track outstanding task requirements, verify progress and completion claims against actions and artifacts, and check whether a correction takes effect in subsequent execution. The dedicated Discussion section in Appendix D develops these implications.

Revisions: Appendix D discusses how the rubric applies to newer models and how harnesses can verify progress and correction in long autonomous runs (L1300–1309).

Clarifications on the study

Nexus contribution and evaluation. We introduce Nexus in this paper. The opening of §5 states this explicitly and explains how the correction and reset findings informed the design.

Revisions: The design motivation and introduction of Nexus are in §5 (L409–431, Figure 6).

The revised abstract, introduction, and contribution list identify Nexus as a method we introduce and evaluate. Section 5 reports the comparison with generic supervision using the same reviewing model across three benchmarks.

Revisions: The abstract (L25–27) and §1 (L70–85) identify Nexus as our method. The abstract reports the main result of the expanded comparison (L28–30), and §5 reports the full three-benchmark comparison (L450–485, Figure 7 and Table 4).

Operative task state. The operative task state ntn_t is the agent’s working interpretation of the task at step tt. The shorter §2.1 explains how we infer it from expressed judgments, tool choices, and actions. The definition applies when the interpretation is expressed through behavior or stored explicitly.

Revisions: Section 2.1 defines the operative task state and explains how it is inferred from the trajectory (L95–111).

AuthorsOpenReview ↗

Response to Reviewer ay8e (2/2)

Was there human validation in Appendix B.8? Yes, we checked the automated labels against human judgments. Appendix B.8 describes the two human validations in separate paragraphs. In the original blind check, a coauthor who had not participated in the original annotation agreed with the automated labels on 49 of 50 trajectories and marked one uncertain. In the new audit, two humans independently labeled all 200 trajectories and agreed on 196/200 (κ = 0.963). The automated labels matched 188/190 (98.9%) of their definite consensus labels.

Revisions: Appendix B.8 presents the original blind check and the new independent human audit in separate paragraphs (L964–975).

First-pass screening. A trajectory proceeds to full evaluation if it scores at least 1 on any rubric dimension. Of the CL-bench-Life trajectories, 98.6% passed this screen and 11.8% met the confirmation criteria.

Revisions: Section 3.1 states the screening rule before the CL-bench-Life screening and confirmation rates (L267–287).

Writing and figure presentation

Opening of §4.2. We shortened the opening and kept the sentence you highlighted: “The increase through concrete evidence shows that specific contradiction is more useful than a generic request to reconsider.” The paragraph gives the recovery range and the key comparison. Table 2 gives the six individual rates.

Revisions: The shorter opening is in §4.2 (L373–377), with all six recovery rates in Table 2 (L328–340).

Terminology. The revised manuscript uses narrative lock-in (N-lock) throughout to avoid ambiguity with the established NLP abbreviation NLI.

Revisions: The revised abstract and introduction use N-lock (L11–26, L54–85), and the manuscript uses it throughout.

Figures 3–5

Figure 3. The figure summarizes the 6,035 trajectories used in the characterization study. Each outer-ring percentage is the share of analyzed trajectories in that category within its benchmark.

Revisions: Section 2.4 and Figure 3 identify the 6,035 analyzed trajectories and the denominator for the outer-ring percentages (L192–199, L231–233).

Figure 4(c). Panel (c) groups LLM annotations into low, medium, and high damage based on how far the agent’s mistaken interpretation affected its actions and artifacts. Panels (a) and (b) use native benchmark scores. The revised caption and §3.3 clarify the two sources. The manuscript also distinguishes the figure’s categories from the 0–3 damage score used in the strict recovery protocol.

Revisions: Figure 4 and §3.3 identify the score and damage sources (L280–285, L310–316). Appendix B.7 defines the damage groups (L864–882, Table 13), and Appendix B.9 describes the strict-recovery damage criterion (L1064–1072).

Figure 5(a). The panel legend now spells out all six conditions, including L1 weak recheck, L2 claim-only correction, L3 concrete counterevidence, and L4 explicit solution path.

Revisions: The six condition names are spelled out in the Figure 5(a) legend (§4.1, L344–366).

Thank you again for showing us where the paper was hardest to follow. We hope the revised manuscript presents the trajectory findings more clearly and makes their connection to the design of Nexus easier to follow.

Dear Reviewer ay8e,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We thank you for your time and constructive feedback on our study of narrative lock-in and Nexus’s design. Your questions guided the human audits, scoring checks, and matched comparisons in our individual responses.

Our rebuttal and revised manuscript report the following.

  • We conducted independent blind human audits and tested robustness using official scores. The audits supported the reliability of N-lock annotations and strict recovery judgments. Strategy rankings held on the common officially scored subset.
  • We verified Nexus’s gains in matched comparisons. With the same reviewing model and execution budget, Nexus outscored generic supervision on all three benchmarks.
  • We documented how Nexus supervises execution and measured its cost. The method section explains how the trajectory findings informed Nexus, how it works, and what resources the gains required.

The table summarizes our responses and corresponding revisions.

ReviewersConcernResponse and manuscript revision
iZ2s, ay8eEmpirical contribution and its relevance across modelsWe clarified what the trajectory analysis reveals about correction. Concrete counterevidence and an explicit solution path improved outcomes equally often, but counterevidence yielded an 8.1-percentage-point higher strict recovery rate. We observed N-lock in all five models tested. Tracking what agents say, what they do next, and what persists in their artifacts distinguishes verbal acknowledgment, behavioral update, and strict recovery. This distinction informed Nexus.<br>Revisions: §§4–5 report the recovery comparison and motivate Nexus (L359–406, L409–431). Appendix D discusses applicability to newer models (L1300–1309).
L18rContinuation and scoring of the correction experimentsWe reported continuation lengths and official-score coverage by arm. Official scores covered 99.7% of runs. On the 718 shared starting points with official scores in all six arms, the full-set rankings for outcome improvement and strict recovery held.<br>Revisions: Appendix B.9 reports continuation and score coverage (L1014–1041, Table 19), outcome scoring (L1064–1072), and common-subset rankings (L1078–1091, Table 20).
L18rIndependent validation of strict recoveryA separate blind audit of 239 runs checked all three strict recovery criteria. In all 169 runs with definite human consensus on all three criteria, the human-derived recovery decision matched the original.<br>Revisions: Appendix B.9 specifies the criteria and reports the independent audit (L1064–1072, L1094–1101).
iZ2s, ay8eN-lock annotation accuracy and applicability to frontier modelsWe added a blind audit of 200 trajectories. Two human annotators independently applied the rubric to observable behavior, including in frontier-model trajectories. Automated labels matched 188 of 190 definite human consensus labels (98.9%).<br>Revisions: Appendix B.8 reports the blind audit and frontier-model comparison (L970–975).
iZ2s, L18rNexus’s added value under matched conditionsWe reran unsupervised execution, generic supervision, and Nexus on 225 matched task–model pairs across three benchmarks. All arms shared inputs, initialization, and execution budgets. Both supervised arms used the same reviewing model. Nexus outscored generic supervision by 20.5 points in mean native score on the 0–100 scale.<br>Revisions: See §5 for reruns (L450–485, Figure 7 and Table 4), Appendix C for task selection and matching (L1216–1228), and the abstract for the expanded evaluation’s main result (L28–30).
iZ2s, L18rNexus implementation and token useWe clarified which component calls the reviewing model and measured total token use. Observer records tool evidence, WorldMap maintains structured task state, and Mentor alone calls DeepSeek V4 Pro. Counting actor and supervisor calls, unsupervised execution, generic supervision, and Nexus averaged 711k, 768k, and 764k tokens per task, respectively.<br>Revisions: §5 (L409–431, Figure 6) and Appendix C (L1221–1224) specify the roles and reviewing model. §5 (L472–491, Figure 7) and Appendix C (L1197–1214 and L1230–1235, Table 25) describe both token metrics.
iZ2s, ay8eTerminology and presentationWe clarified task-state notation, the initial screening rule, and what the figures report. The revision uses N-lock throughout and explicitly introduces Nexus as our contribution.<br>Revisions: The abstract and §1 introduce N-lock and Nexus (L11–30, L54–85). See §§2.1 and 3.1 for task state and screening (L95–111, L267–287), and Figures 3–5 for visual definitions (L231–233, L280–285, L344–366).

Our individual responses detail experimental settings, numerical results, and section and line references for manuscript revisions. We thank you for your time and welcome further discussion.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.