- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether recurrent representations help world-model agents compensate for misleading observations in Craftax-Classic. The authors falsify inventory displays without changing actual resources and compare Δ-IRIS and DreamerV3, followed by within-DreamerV3 recurrent-subspace ablation and donor-state replacement. Removing the selected stone-related component causes greater impairment under a false display, while correct-status donors restore performance more effectively than wrong-status donors. Validation includes simulator-recorded achievements, paired agent-level comparisons, multiplicity correction, and additional probing and repair experiments. The findings support conditional recurrent involvement in behavior, while residual decodability and differences in donor-patch magnitude and context limit resource-specific mechanistic attribution.
Critical strengths
- Informative crossed-intervention design: Combining display falsification with recurrent-state ablation uncovers conditional behavioral effects that isolated interventions miss, showing consistent negative interactions across all ten DreamerV3 agents.
- Direct behavioral validation: Grounding evaluations in simulator-recorded achievements rather than model predictions—paired with matched initial states and multiplicity correction—ensures high empirical credibility.
Critical weaknesses
- Unresolved mechanistic attribution: Correct- and wrong-status donor patches differ in perturbation magnitude, context similarity, and collateral impact on non-target features (e.g., wall coordinates), with norm matching failing predefined criteria. Performance recovery cannot be cleanly attributed to resource content.
- Limited explanatory scope: Strong compensation effects appear only for stone tasks across ten DreamerV3 agents in a single environment. Results for wood, stochastic repair, and policy relearning remain inconclusive, offering only a narrow view of recurrent compensation.
Questions for authors
Q1: Does the performance advantage of the correct donor persist when perturbation magnitude, wall status, and other non-target attributes are jointly matched between patches?
Q2: Does the stone-related interaction hold when stone-tool crafting and stone placement are evaluated separately, especially given the donor patches' collateral impact on wall representations?