- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how language models retain, update, and use conversational state when generating executable database queries directly from dialogue. Analyses across eight models compare state readouts at different positions, intervene on historical sources through blocking and counterfactual substitution, and classify natural omissions using structural signals and conditional value preferences. The resulting State–Action Controller, SAC, edits an existing query using lightweight structural readouts and explicit user-text evidence. In the five-model, single-executable-domain MultiWOZ closed-loop evaluation, mean exact-query accuracy increases from .318 to .621 and task success from .272 to .371. Task success is close to that of T5-DST trained on the same decision examples, while SAC better preserves correct drafts and requires no additional language-model generation calls.
Critical strengths
- The distinction between readable information, causal influence, and behavioral competence is a substantial strength. Larger models can select current values correctly on examples where the binding probe performs poorly. Retaining these results prevents the representation analysis from treating a weak readout as proof that the model lacks the relevant ability.
- The source interventions have informative controls. Matched dialogues preserve source content and position while changing validity, and the blocking and substitution studies include length-matched non-value spans, unrelated content, and wrong-position controls. Fixed-orientation probability contrasts and the natural stale-error analysis jointly support the claim that historical sources affect action and can remain influential after supersession.
- The controller evaluation measures both benefit and harm. Fixed-history ablations preserve the resolver, safeguards, and compiler while changing the selection signal. Closed-loop results then report fidelity, success, preservation, repair, and destructive edits. This provides a more convincing account of selective correction than an aggregate success rate alone.
- The main interactive comparison handles trajectory divergence carefully. Methods are paired by initial goal, while query statistics use the turns actually encountered by each method. Human-history replay and naturalized simulator language add useful coverage, with clear qualifications about single-domain execution and reliance on explicit user evidence.
Critical weaknesses
- The binary correctness contrast in Eq. (1) is given a stronger source-influence interpretation than it supports. The correct candidate reverses between the matched conditions. An identical intervention-induced change in fixed-candidate preference can therefore hurt correctness in one condition and improve it in the other, producing a positive contrast without a difference in source-effect strength. The statistic remains useful for comparing correctness effects. The fixed-orientation graded analysis in Table 13 supports the stronger validity claim, and the paper should distinguish these interpretations.
- The temporal presentation can suggest within-event state decay despite changing eligible cohorts. The number of events falls from 727 to 334 across lags, and later restatements remain available. Figure 2b consequently describes dependence on the original sources in each lag's eligible population; it does not isolate decay in total current-state dependence within a fixed cohort. The appendix already recognizes this limitation, which should also appear in the main text and caption.
- Table 23 resamples omissions even though a goal can contribute several turns and slots. Its intervals may therefore understate uncertainty over goals, particularly for the small binding groups. The point-estimate differences remain useful, but the interval interpretation should match the sampling unit. This concern does not apply in the same way to the goal-level main closed-loop analysis.
- The T5 memory figure needs a clear accounting definition. The reported 60,506,624 float32 parameters require approximately 242 MB for weights alone if resident on the measured GPU, whereas Table 28 lists 40 MB beyond the resident backbone. Whether weights were already in the baseline, excluded, or placed elsewhere materially changes the comparison. This issue does not negate SAC's lack of extra generation calls or its reported low latency.
Questions for authors
- Is the loss in Eq. (1) pre-intervention minus post-intervention correctness? How does the interpretation separate reversal of the correct label from a change in fixed-candidate source-effect strength, and should Table 13 be the primary evidence for the latter?
- What is the implemented resampling unit for Table 23: individual omissions, initial goals, or dialogues? Please clarify the population those intervals are intended to describe.
- What T5 objects are included in the 40 MB entry in Table 28? Please specify model residency before measurement, parameter placement, and the peak-reset baseline so that the figure can be reconciled with the parameter count and precision.
Response to Reviewer RYH3
We thank the reviewer for the review. All four points are addressed below.
W1 and Q1. Yes. As implemented, Lac=Y(H)−Y(Ha), the pre- minus post-intervention value of Y=1[the greedy query emits the currently valid value]. We will replace "favors" with "emits" in §4.2. The reviewer is correct that because the valid value changes between twins, this contrast can be positive without a difference in source-effect strength. Table 13 is the evidence for the stronger claim. Recomputed with the fixed outcome Y′=1[the query emits v1] in both twins, the adjusted effect remains positive and significant on 7 of 8 models (macro +.558 vs +.582). The exception is OLMo-7B-IT, which emits v1 in only 3.7% of current twins. Masking the v1 source removes v1 in .52–.86 of current twins and at most .061 of superseded twins, while the random mask changes Y by at most .005, so the superseded term sits near a floor. We will make the graded Table 13 analysis the primary evidence for validity-dependent source influence in §4.2, interpret the original binary contrast only as a correctness effect and report the fixed-candidate binary analysis separately.
W2. Agreed, and we have run the fixed-cohort analysis. On the 333 events present at every lag, the new-source advantage D declines from lag 1 to lag 4 (Qwen +1.87→−0.37, Llama −0.59→−1.75). We then split the events by whether vnew is restated between the update and the lag. Where it is not restated, D stays positive on Qwen (+5.21→+2.57) and is not distinguishable from zero on Llama (+1.04→+0.04). Where it is restated, D is negative throughout. Restatement status therefore materially affects the original-source contrast, but the contrast declines even without restatements. These analyses measure the relative influence of the original mentions. They do not establish decay in total current-state dependence or a return to the old value, and we will not claim either. We will state in §4.3 and the Figure 2b caption that the curve describes dependence on the original mentions within each lag's eligible population, and add the fixed-cohort and restatement results to the main text.
W3 and Q2. The reviewer is right. The implemented unit in Table 23 was individual omissions, and a goal can contribute several turns and slots, so the intervals understate uncertainty over goals. We recomputed them with resampling by initial goal on all seven checkpoints. The goal-level intervals are about 1.3× wider for Binding and 2.0× wider for Deployment in the pooled rows (per-checkpoint factors range from 1.0× to 2.2×). Binding and Deployment intervals still do not overlap on six of the seven checkpoints or in the pooled row. They overlap only on OLMo-32B-IT, whose Binding cell has 5 omissions from 3 goals. The goal-level interval for the Deployment−Binding difference itself is +.702 [+.623,+.771] pooled, and it excludes zero on six of the seven checkpoints, again with OLMo-32B-IT as the exception. We will report goal-level intervals in the revision, state the resampling unit in the caption, mark Binding cells with fewer than ten omissions as descriptive, and keep the omission-level intervals in the appendix for comparison.
W4 and Q3. The reviewer’s arithmetic is right. T5-small has 60,506,624 float32 parameters, which occupy approximately 242 MB. These weights were resident on the GPU before the peak-reset baseline. The entry reported as “40 MB” in Table 28 corresponds to 40.14 MiB (approximately 42 MB) of peak additional call-time allocations, excluding the resident model weights. We re-measured with the baseline taken before T5 is loaded, under the Table 28 protocol. Peak additional GPU memory is approximately 284 MB on both backbones, comprising 242 MB of resident weights and 42 MB of call-time allocations, compared with 8 MB for the controller. We will replace the Table 28 entry with this accounting and state the residency convention and units in the caption. The latency and extra-call comparisons are unchanged.
We hope these address the concerns and welcome any that remain.