- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
StateSwap studies a pipeline from history to stored application state, retrieval allocation, rendered evidence, and an LLM decision. It changes the allocation policy while fixing each paired world's task, repository, and gold answer. Its profile contains allocation disagreement pA, rendered-evidence disagreement pR, observed decision disagreement pD, and the difference in mean correctness tau. It checks intervention execution separately. A deterministic renderer implies pR <= pA. But changes in allocation or rendered evidence need not change answers or average correctness. The HotpotQA calibration replaces either an unannotated or an annotated supporting paragraph. A later synthetic experiment uses 3,200 worlds, changes source reliability, and compares mirror allocation policies.
Critical strengths
The problem definition is interesting. It distinguishes running an intervention from changing allocation, and answer disagreement from improved correctness. Fixing the gold answer prevents the intervention from changing its evaluation target. The profile is not presented as a formal mediation analysis or evidence about model-internal state.
The two kinds of controls are a thoughtful design choice: one holds allocation fixed, and the other holds rendered evidence fixed. The paper reports different outputs even when inputs are byte-identical. The HotpotQA equivalence test uses a prespecified margin and 90% intervals, rather than treating a nonsignificant difference as equivalence.
The authors disclose archive incidents, truncations, parser failures, and a bootstrap seed correction. The two-reader HotpotQA study shows how equal upstream change rates can hide different effects on accuracy.
Critical weaknesses
The paper has not shown what StateSwap adds to a simpler intervention audit. I am not yet convinced that StateSwap adds enough beyond a careful intervention audit. This matters for the claims in Section 1, the profile table in Appendix F.3 and the comparison with related work in Appendix I. Proposition 1 follows from the deterministic pipeline definition. Proposition 2 uses readers that ignore or use changed evidence. Proposition 3 says identical input distributions give equal expected correctness. The authors acknowledge these limits, so the experiments need to show much of the protocol's value.
What I am missing is a case where measuring this profile leads to a better diagnosis or engineering decision. The C1/C2 controls offer a useful starting point: they already provide two known breaks in the pipeline and could support a diagnosis-and-repair test. I would include cases where state does not affect retrieval, changed allocation is lost during rendering, useful evidence is omitted, or the reader ignores evidence. Hide the known cause from the person or method doing the diagnosis, and compare outcome-only evaluation, outcomes plus ordinary traces, and StateSwap at the same call budget.
Questions for authors
Could you point to a diagnosis or repair in the submitted evidence that StateSwap enables beyond paired outcomes with request traces, allocation records, and input hashes? I would like to see precisely what the protocol adds in that case.
Can S35 or the external-policy data distinguish useful but redundant evidence, ignored evidence, and benefits offset by harms across questions? If so, please explain whether that distinction would change the recommended intervention.
I would like to know whether repeatedly serving both inputs on the same worlds would lead to the same interpretation of pD. Please identify the evidence that separates a response to changed evidence from reader or service variability.