- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how language models resume a task after being interrupted mid-solution by a distractor task. The setup the model sees is:
start to solve task T -> begin distractor task S -> resume solving task T.
A perfect model would manage its context in such a way that the intervening task S would not interfere with T; the resumption not be substantially disturbed by the distractor task.
In practice, over a range of tasks and models, the authors observe that interruption does interfere, as measured both by the model's ability to declaratively recall the interruption T-state, and its ability to carry the task to a successful conclusion. Interestingly, there are sometimes divergences between recall performance and task performance. The authors systemically vary the main and distractor tasks, studying how task structure and content shapes interference.
The tasks include a simple arithemtic task requiring registers, a symbolic state-tracking task, and GSM-8k. The experiments use several Qwen models and Llama-3.1-8B, with most of the main behavioral results coming from models with 3B–14B params.
The authors evaluate several mechanistic hypotheses, for example using activation patching and attention analyses. They cast doubt on several plausible candidates, but don't arrive at a definitive positive explanation.
Critical strengths
- The general question of how models represent and track states across tasks is an interesting one, with potentially important practical implications.
- I appreciated the careful control conditions used for most of the experiments.
- While the mechanistic experiments failed to make a strong positive case for a particular representation scheme, I appreciated the authors' rigor in reporting their negative results.
- I found the main result -- that models exhibit interruption interference even for very simple tasks -- surprising.
Critical weaknesses
In my opinion, the authors haven't yet made a convincing case for the importance of the task as studied, or the results.
I can imagine at least two versions of the paper that I think would have been very strong:
- As noted in the introduction, juggling and interleaving tasks is an important part of modern agentic systems in complex and long and complex trajectories. So, I can imagine a valuable paper focused on behavioral analysis at scale, studying frontier-sized models in complicated, realistic tasks.
- Conversely, I would also love to read a detailed mechanistic study of how models represent symbolic states, and how interference interacts with accessing these states. Even at smaller scales, a satisfying explanation would be compelling and useful.
As it stands, the paper falls between these two poles. The models are too small, and the tasks too toy, to generate lessons directly applicable in practice. And conversely, despite some interesting explorations, the results fall short of a substantially mechanistic account. As it stands, I think this work would fit better at a workshop than the main conference.
Smaller issues:
- The description of the tasks is much too brief and vague; some concrete examples would help a lot.
- Many of the figures, and their text labels, would benefit from being larger.
- Pangram rates the text as 83% AI-generated, though this is consistent with AI use statement.
Questions for authors
No questions
Thank you for such a careful and fair review. We appreciate that you read the control conditions closely, that you valued our negative mechanistic results, and that you described two stronger papers so clearly. You're right that the paper sits between them.
We think both directions are worth doing, and we plan to continue with both as follow-up work: (a) a behavioral study at scale, with frontier-size models and realistic agent trajectories, and (b) a deeper mechanistic study of how models store symbolic state and how interruption interferes with reading it back. We can't do either at full scale in this revision, so for now we made this paper's claims narrower and clearer. We also accept that workshop vs. main conference is a fair question.
We agree, and this was the most important criticism we got. The new wrong-state control gives a positive account at the behavioral level. We keep the interrupted conversation fixed and change only the value we hand back at the resume turn:
So the arithmetic holds up. After the interruption, the model applies the operation to the wrong value. This says where the failure sits, and it could be proven wrong by a different error pattern. It is a behavioral account, not a mechanism inside the model, and the paper now says so.
You're right, and our code-frame transfer is thin. Same-symbol vs. different-symbol across frames gives -7.0 [-12.0, -3.0] pp on Llama-3.1-8B and -5.0 [-9.0, -1.0] pp on Qwen2.5-3B. Five checkpoints sit at ceiling, and one is at the floor because it doesn't keep the output format. We don't claim more than that.
What the controlled tasks do give us is a list of variables to study. The results say interference depends on whether the interruption shares a representational form with the resumed computation, how deep it is, and how far away it is. Real agent runs vary these too. A tool output that overwrites a variable the caller still needs (same register, short distance) is very different from an unrelated file read (different register, long distance). We can't vary these with a known correct answer in a live trace, but we can here. So this work suggests what to measure in agent behavior; it does not measure it yet. This is the first thing we want to do in the larger follow-up.
The last two rows are why we keep two probes and don't report a single "resumption" number.