- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Theseus-AR, which enables autoregressive Transformers to edit their active context while reusing retained KV representations through dynamic RoPE updates. It also introduces Replay Attention for parallel training over editing trajectories. Experiments on ListOps, SAT variable elimination, and typed lambda-calculus reduction show reduced active context and attention-pair counts, with task-dependent improvements in sample efficiency and oracle-anchored single-step accuracy.
Critical strengths
- The method jointly addresses inference-time context editing and parallel training, with Replay Attention supporting changing token visibility and logical positions.
- The paper provides an exact-arithmetic equivalence argument between replay and online execution, supported by numerical checks of attention outputs and gradients.
Critical weaknesses
The Stepwise Transition Evaluation anchors predictions to oracle states, preventing errors from propagating across transitions. While useful diagnostically, it is not supplemented by free-running end-to-end comparisons between Theseus-AR and Transition Trace. This leaves unclear whether the stepwise gains translate into better long-horizon task performance.
Questions for authors
Could you report free-running end-to-end success rates for Theseus-AR and Transition Trace on the three tasks, without resetting intermediate states to the oracle, to assess whether the stepwise gains translate into better complete-task performance?