- Rating
- 2
- Confidence
- 2
- Informativeness
- 2(−1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DOFM, a sparse recurrent observer that tracks feature lifecycles across the depth of a frozen transformer, trained without labels to predict the next normalized residual update from prior states. It unifies three mechanisms: (1) Identity: a single shared decoder D (unit-norm columns) fixes a coordinate's direction at every depth; (2)History: a per-token gated recurrent state, which is unrolled as an attention-discounted sum of writes; (3) Address: cumulative master gates (adapted from ordered neurons) partition the dictionary into slots. DOFM provides clear snapshots alongside the layers across multiple models.
Critical strengths
- The proposed framework genuinely solves a real structural gap, and the exact margin decomposition turns it into a usable attribution tool.
- Experiments across different models demonstrate the effectiveness of the method.
Critical weaknesses
- The scale of backbone models is limited below 2B. I would expect how it scales to larger sizes.
- The training of DOFM seems to be the prediction of residual connection along layers. Therefore, I am confused about whether it really reveals true behaviors that could not be found.
- All experiments are on a single task family (WSD + puns). It remains unclear about the model behavior in downstream scenarios, like QA, summarization, and reasoning.
- DOFM requires a 0.56–0.72B observer per backbone, which can not be ignored compared to the backbone model. It would be expected with a larger reequirement of computation resources on larger backbones.
Questions for authors
- Your central claim is that sense is written early and carried, not recomputed. Can you test this directly against the backbone — e.g., by ablating the backbone at the layer where the observer says the write happened vs. at later "carry" layers, and comparing outcome damage? Would the early-write story survive training the observer with a materially different inductive bias?
Response to Rebuttal
Many thanks to the authors for providing new results and detailed additional explanations.
There are still only comparisons between State Q4, Write Q4 etc... on table 29. Is it possible to include other external baselines?
1280 (Gemma-2) or 1024 (Qwen3-8B) is significantly lower than dmodel=2304 and dmodel=4096, the residual stream dimension that it is trying to describe. Usually you need much larger dimensions for SAEs? Then the baselines are not really fair? You could either retrain the much larger untied crosscoder model, or retrain DOFM with the same smaller width?