- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors claim that in sequential decision-making, models must both maintain a useful internal state and learn to predict behavior from that state. They find that a useful state can become available before behavior extracts its full value. This motivates intervention after the state is formed to improve downstream performance. They validate across a maze task, Craftax, and language.
Critical strengths
- I appreciate multi-setting validation of the experimental findings.
Critical weaknesses
- One weakness is that the scope is perhaps a bit small. The main conclusion is a validation that once state is formed, we should remove pressure for learning the internal state and focus optimization on fitting behavior from that state. This claim is a bit unsatisfying in that it may not provide much new insight to the community, and it’s not clear how this rule can be packaged well into an algorithm, even though the paper claims it is “operationalized”.
- Another weakness is that the experiments aren’t thorough/promising enough. I would appreciate seeing better validation of this insight since it doesn’t seem like the improvements in Craftax are particularly significant.
- Due to these weaknesses, I don’t think this paper is fully complete and for now has a bit of a narrow claim - with more algorithmic work and stronger empiricism, there could be a better submission.
Writing nitpicks:
- What is a “matched belief swap”? It would be good to keep the abstract more high-level. Again, I see “Yet a representation-targeted auxiliary that raises decodability leaves tight-horizon success essentially unchanged” - I can use context clues to assume that training to explicitly decode state doesn’t improve performance? Maybe it’d be good to have examples of these “interventions” if they’re core to the paper’s key insights.
Questions for authors
- Isn’t this addressable by classic loss scheduling? We can just weight down representation supervision when it’s converged and begin weighting up the behavioral loss. When attacking multi-objective systems and weighted losses, many folks can downweight based on the staleness of the objective (i.e. how converged it has become).
- Also, what about cases where it’s unclear what internal state we are forming (i.e. no ground truth)? How do we supervise this or even gauge the convergence?
The revised paper asks how useful historical state develops during learning and what downstream behavior can be recovered from it. The new navigation study connects the measurement to ordinary-start task performance with the encoder fixed.
Fifteen independent replication agents reach 69.60% success within 18 actions from ordinary starts, versus 37.28% for their original policies.
These three co-primary intervals are 99.17% paired-model intervals (df 14), accounting for all six co-primary contrasts across both studies at simultaneous 95% coverage. All three prespecified lower bounds are positive; the joint criterion is met.
The initial five-agent joint test failed and remains separate. This fixed 15-agent replication keeps the method unchanged and accounts for both studies.
The training choice follows a separate development diagnosis. Remembered history changes the fixed controller's action on 188/2,039 validation decisions. We crossed gap/action prediction with uniform/balanced exposure, giving learned/current/random features every choice. Only balanced gap fitting qualified in both development agents. We then fixed it before training or evaluating any confirmation agents. Balanced versus uniform fitting changes final navigation by 4.62 pp [3.44,5.80], descriptive 95% model interval.
Encoder, embeddings and normalization stay frozen, hash-checked. Readouts share 4,096 histories, observed-candidate labels and 500 updates, with no unseen passage labels. Current/random controls share the planner's geometry and computation. The native-policy gain includes added controller resources; matched contrasts estimate learned history. Erasure removes the whole history channel.
The new results and every fresh agent appear in Section 4.2, Table 1; Appendix K. All 242 replication rollout arms were independently replayed on CPU. Earlier unsuccessful map/candidate readouts and the complete development comparison remain in the appendix; they are not pooled with confirmation.
The counter changes only ten native answer rows (970 parameters): five models improve 85.60% to 93.31%; ten separate models improve 84.93% to 92.82% primary and 73.50% to 88.04% strict. Native state sequences, hidden inputs and capacity stay identical. In the ten-model cohort, state-target fitting exceeds supervised/self-distilled heads in every model on both banks. Cohorts and test populations stay separate. The counter controls and label access are in Section 5.2, Table 2. Its direct teacher reaches 98.79%, above the student; symbolic targets work equally well. Prior state supervision and known dynamics remain required, with no overall annotation-efficiency claim.
Craftax's complete five-model continuation gives +.753 full-model/+.571 frozen-body original-reward window return over continued PPO. First-episode returns fall and endings increase, so the result does not identify improved spatial utilization (Section 6, Table 3). The separate full-model retention study fails its joint criterion: the late-strict 97.5% interval includes zero. Ordinary state supervision is comparable; no general scheduling advantage follows.
The manuscript separates decodability, behavioral dependence and recovered performance. Distance thresholds no longer establish state formation; contaminated operations-only results are withdrawn. Probe targets, intervention sites, splits, budgets, all model-level intervals and the independent tests are explicit. We cite concurrent He et al. on readout development. Readout fitting and reweighting are established tools; our contribution is the developmental evidence and the controlled task consequence.