- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces TurboSens, a controlled environment for measuring how well world models preserve latent state. The environment is a configurable turbofan simulation, built on the OpenDeckSMR simulator and combines sensor observations with the known engine health state variables. This allows representation quality to be evaluated against the true system state, beyond prediction or reconstruction performance. The environment is designed so that part of the state is only partially observable, e.g., fouling channel, and must be inferred from temporal context, maintenance actions and changing operating conditions. The paper also releases the data produced which contain 500 training episodes.
The authors also evaluate representations from several world model families (e.g., JEPA, LSTM, RSSM model) using frozen encoders and linear inverse probes measuring the ability for latent state recovery, the robustness to distribution shift and the responses to maintenance interventions.
The authors find that self-supervised representations preserve far less of the state than the observations contain. The best self-supervised encoder reaches a mean correlation of 0.73, while a linear probe on the raw observation reaches 0.93.
Critical strengths
-
The pairing of the ground-truth state with an interactive simulator is a great contribution. The environment exposes the full health and fouling state at every step, accepts actions online and replays episodes exactly under counterfactual changes, which enables measurements that existing turbofan benchmarks cannot support.
-
The evaluation protocols are quite detailed. The reference baselines, the episode-level bootstrap, multi-seed reporting and the analysis by time since the last wash are reported in detailed.
Critical weaknesses
-
The environment does not appear to demonstrate the partial observability challenge that motivates the paper. A linear fit on a single raw observation already recovers the wear state at 0.93, an MLP reaches 0.96, and the MLP's recovery drop is very minimal (only 0.027) as fouling builds up. Thus, the claims about action-dependent observability therefore is a very small delta.
-
The conclusion about self-supervised objectives is broader than what is supported by the presented evidence. All JEPA and LSTM models share a single training recipe and the RSSM collapses even though its reconstruction loss converges, which suggests an implementation or normalization problem. The paper also attributes the gap between the self-supervised encoders (0.73) and the raw observation (0.93) to the training objective, but there is a lack of comparison against simple baselines which should isolate the if the objective formulation is the actual cause.
Questions for authors
-
The paper argues that a single observation is consistent with many combinations of wear and fouling, so that only a model integrating the action history can recover the wear state. The reference baselines appear to suggest otherwise. A linear ridge on one raw observation already reaches 0.93, an MLP reaches 0.96, and the MLP's recovery drops by only 0.027 as fouling builds up. Could you explain why TurboSens should still be considered partially observable in a meaningful sense? For example, are there state components, wash strata or episodes in your existing results where single-frame recovery fails and the history is needed? This explanation would address my main concern about the benchmark.
-
Section 5.1 attributes the gap between the best self-supervised encoder and the raw-observation ridge to the self-supervised objective formulation. The matched oracle rules out encoder capacity, but the experiments do not rule out that the training itself changes the encoder very little. Can evidence supporting this be provided? e.g., if the randomly initialized encoder scores far below 0.73 then the training clearly shapes the representation and would isolate the contributions to the objective as the paper argues.
-
The RSSM model only achieves a correlation of 0.045 although its reconstruction loss converges. The paper reads this as evidence that reconstruction quality does not imply state preservation. But, on the other hand, a linear map of the observation already recovers the state at 0.93, so a model that reconstructs the observation well should retain most of this information. This makes the RSSM result unconvincing, particularly since it is based on a single seed. Could you report the RSSM's reconstruction quality on the test split, a linear and an MLP probe on its encoder embedding, and the input normalization used?
We thank the reviewer for the precise review. We added the random initialization control proposed.
W1 / Q1 (partial observability). The small drop you point out (0.027) is due to a few sensors that reveal fouling. Without them, single frame recovery falls to 0.889 or 0.779 and history adds +0.060 to +0.067 R2. The apparent failures in the oldest time since wash strata came from very low label variance and disappear after refitting, so we do not count them.
W2 / Q2 (role of the objective). Untrained encoders with the same architecture match the pretrained ones, while training the same network on the state reaches 0.911. The pretraining objectives are therefore the limiting factor, and the revision states this for the objectives and recipe we tested only.
Q3 (RSSM). You are right that this was an implementation issue. A setting in the KL term let the latent ignore the observations, and both the linear (0.045) and MLP (R2≈0) probes confirm that it holds no state. Inputs were symlog transformed and layer normalised, and we will report reconstruction error with the corrected run.