acceptodds
Under review as a conference paper at ICLR 2027

StateSwap: Separating Propagation from Decision Effects in Persistent-State LLM Agents

Abstract

Evaluations of persistent-state agents can conflate three questions: whether an intervention was executed, whether its consequences propagated through retrieval to rendered evidence, and whether the downstream decision responded. StateSwap verifies intervention execution separately and reports, on paired latent worlds, a state-intervention profile : allocation disagreement, rendered-evidence disagreement, observed decision disagreement, and the paired correctness contrast. Its Intervention–Propagation–Decision Separation principle states that rendered-evidence disagreement requires allocation disagreement, whereas complete upstream disagreement does not identify a decision-side effect. We instantiate this protocol with real, externally served readers. Within each pair, the task, plans, repository realisation and paired gold target are fixed; only the stored allocation policy differs, inducing a versus split of a fixed retrieval budget, and the reader is masked from state, arm and gold. In a pre-specified held-out block of 2000 paired synthetic worlds, allocation and rendered evidence changed in every pair while (95% interval -0.003 to +0.023), compatible with a small positive, near-zero or slightly negative effect. An annotation-assisted calibration on 1000 HotpotQA distractor-validation questions under two reader families shows why this distinction matters: two one-paragraph replacements with identical upstream change rates have sharply different decision-side consequences. The mean contrast for replacing an unannotated paragraph relative to baseline met a pre-specified pp equivalence criterion under both readers, whereas replacing an annotated supporting paragraph reduced accuracy by 34.5 pp (DeepSeek) and 32.2 pp (Qwen). A separate fresh-sample validation on 3200 new, R9-disjoint synthetic worlds tested an imposed stored policy. For the pinned Qwen primary, the reliability-dependent decision-side interaction met the prespecified conjunctive sign-reversal criterion; the prespecified DeepSeek secondary did not reach the reversal. The design was fixed after the exploratory R9 outcome was known, so it is not a preregistration or an independent replication, and S35's held-out identity is unchanged. StateSwap therefore treats propagation and decision response as distinct estimands; it neither measures model-internal state nor establishes a general decision benefit from persistent state.

Primary area:
causal reasoning
Keywords:
LLM agents, persistent state, causal intervention, evaluation methodology, retrieval, diagnostic calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_G3r3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StateSwap studies a pipeline from history to stored application state, retrieval allocation, rendered evidence, and an LLM decision. It changes the allocation policy while fixing each paired world's task, repository, and gold answer. Its profile contains allocation disagreement pA, rendered-evidence disagreement pR, observed decision disagreement pD, and the difference in mean correctness tau. It checks intervention execution separately. A deterministic renderer implies pR <= pA. But changes in allocation or rendered evidence need not change answers or average correctness. The HotpotQA calibration replaces either an unannotated or an annotated supporting paragraph. A later synthetic experiment uses 3,200 worlds, changes source reliability, and compares mirror allocation policies.

Critical strengths

The problem definition is interesting. It distinguishes running an intervention from changing allocation, and answer disagreement from improved correctness. Fixing the gold answer prevents the intervention from changing its evaluation target. The profile is not presented as a formal mediation analysis or evidence about model-internal state.

The two kinds of controls are a thoughtful design choice: one holds allocation fixed, and the other holds rendered evidence fixed. The paper reports different outputs even when inputs are byte-identical. The HotpotQA equivalence test uses a prespecified margin and 90% intervals, rather than treating a nonsignificant difference as equivalence.

The authors disclose archive incidents, truncations, parser failures, and a bootstrap seed correction. The two-reader HotpotQA study shows how equal upstream change rates can hide different effects on accuracy.

Critical weaknesses

The paper has not shown what StateSwap adds to a simpler intervention audit. I am not yet convinced that StateSwap adds enough beyond a careful intervention audit. This matters for the claims in Section 1, the profile table in Appendix F.3 and the comparison with related work in Appendix I. Proposition 1 follows from the deterministic pipeline definition. Proposition 2 uses readers that ignore or use changed evidence. Proposition 3 says identical input distributions give equal expected correctness. The authors acknowledge these limits, so the experiments need to show much of the protocol's value.

What I am missing is a case where measuring this profile leads to a better diagnosis or engineering decision. The C1/C2 controls offer a useful starting point: they already provide two known breaks in the pipeline and could support a diagnosis-and-repair test. I would include cases where state does not affect retrieval, changed allocation is lost during rendering, useful evidence is omitted, or the reader ignores evidence. Hide the known cause from the person or method doing the diagnosis, and compare outcome-only evaluation, outcomes plus ordinary traces, and StateSwap at the same call budget.

Questions for authors

Could you point to a diagnosis or repair in the submitted evidence that StateSwap enables beyond paired outcomes with request traces, allocation records, and input hashes? I would like to see precisely what the protocol adds in that case.

Can S35 or the external-policy data distinguish useful but redundant evidence, ignored evidence, and benefits offset by harms across questions? If so, please explain whether that distinction would change the recommended intervention.

I would like to know whether repeatedly serving both inputs on the same worlds would lead to the same interpretation of pD. Please identify the evidence that separates a response to changed evidence from reader or service variability.

Reviewer_9rhhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StateSwap, a paired-intervention evaluation framework for distinguishing the effects of persistent state on retrieval allocation, rendered evidence, and downstream LLM decisions. The central argument is that changes in retrieved evidence do not necessarily translate into changes in model decisions or task correctness. The authors propose a four-dimensional state-intervention profile measuring allocation disagreement, evidence disagreement, decision disagreement, and the paired correctness contrast, supported by formal separation results and controlled interventions. The framework is evaluated using synthetic decision tasks and an external HotpotQA calibration. On 2,000 held-out paired synthetic worlds, changing the stored retrieval-allocation policy alters the retrieved and rendered evidence in every case but yields only a +1.0 percentage-point accuracy difference, with a confidence interval including zero. The HotpotQA experiments show that removing supporting evidence causes substantially larger accuracy degradation than removing non-supporting evidence, despite identical upstream intervention rates. An additional fresh-sample experiment on 3,200 synthetic worlds finds a reliability-dependent decision interaction for Qwen, but not the same sign-reversal result for DeepSeek.

Overall, the paper contributes a controlled measurement protocol for separating evidence propagation from downstream decision effects, rather than a new persistent-memory learning method or evidence that persistent state generally improves agent performance.

Critical strengths
  1. Well-controlled intervention design. The paired-world setup is a strength despite not a novelty. By holding the task, repository, and ground truth fixed while changing only the stored retrieval-allocation policy, the authors can distinguish changes in retrieved evidence from changes in downstream decisions. The explicit logging of intermediate outputs and identical-input controls further strengthen the analysis. This provides a useful experimental protocol for diagnosing where persistent-state interventions actually affect agent behavior.

  2. Careful empirical validation and transparent reporting. The paper includes a pre-specified held-out evaluation on 2,000 paired worlds and an external HotpotQA calibration across two reader families. The latter provides a particularly clear demonstration that equally large changes in retrieved evidence can have very different effects on answer correctness. I also appreciate that the authors report statistically inconclusive findings without overstating them and clearly distinguish confirmatory from exploratory evidence. This strengthens the credibility of the empirical observations, even though the broader methodological novelty remains limited.

Critical weaknesses
  1. Limited methodological novelty and technical contribution. The central observation that changes in retrieved evidence do not necessarily lead to changes in downstream decisions is intuitive and already well understood in causal inference and evaluation research. The formal propositions largely establish straightforward properties of the proposed experimental setup rather than new theoretical insights. While the four-dimensional intervention profile provides a useful way to organize measurements, it is unclear what fundamentally new evaluation capability it offers beyond existing intervention-based approaches. This limits the paper's contribution relative to the novelty expected at ICLR.

  2. Limited validation on realistic persistent-state agents. The main experiments manipulate a fixed retrieval-allocation policy rather than evaluating agents that learn, update, and use persistent memory over time. The primary held-out experiment shows only a 1 percentage-point accuracy difference with a confidence interval including zero. Although this is consistent with the authors' separation argument, it provides limited evidence of the framework's practical value. The HotpotQA calibration further demonstrates an expected result: removing supporting evidence affects accuracy much more than removing non-supporting evidence. Overall, the experiments establish the proposed measurement distinction in controlled settings but do not convincingly demonstrate its utility for evaluating realistic, long-horizon agents.

Questions for authors
  1. Methodological novelty (Clarification): What does StateSwap offer beyond existing paired-intervention evaluations? Can the authors provide a concrete example where the proposed framework reveals insights that standard retrieval and outcome metrics would miss?

  2. Practical relevance (Clarification/Validation): The experiments use an imposed retrieval-allocation policy rather than learned persistent memory. How do the findings generalize to real memory-enabled agents? Evidence of meaningful failure modes in such systems would strengthen the contribution.

Reviewer_pQEFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Presentation is very poor. Concepts are not clearly introduced. The abstract has 4 paragraphs and includes too much detail and not enough explanation. The basic terms are not properly introduced.

Critical strengths

None

Critical weaknesses

Presentation is very poor. Concepts are not clearly introduced. The abstract has 4 paragraphs and includes too much detail and not enough explanation. The basic terms are not properly introduced.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.