- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper learns recurrent state representations from observation–reward sequences without action labels. It combines future-occupancy prediction with adaptive recurrent noise and evaluates the learned states on synthetic memory tasks and real laser data.
Critical strengths
-
The distinction between useful predictive information and unnecessary history is well motivated and gives the method a clear focus.
-
The synthetic tasks allow direct checks of state recovery, and the ablations help explain the effects of recurrent noise.
Critical weaknesses
-
The channel-capacity argument requires a bounded recurrent signal, but LayerNorm appears to act only on the readout. The raw noisy hidden state is still fed back, leaving this assumption unjustified.
-
Proposition 1 establishes predictive sufficiency under the collection policy. The stronger claim about preserving information for control needs additional justification, since another policy may require distinctions that the representation removes.
-
The NCLT result labeled as adaptive noise in Table 1 comes from a wider network with fixed noise in Table 14. I suggest correcting the label and comparing the variants at the same network width.
-
The best discrete-control results use true-state information to select the codebook size. I suggest testing label-free selection and applying the same one-hot readout to the no-noise encoder and competing methods.
Questions for authors
-
Which state is normalized and fed back into the GRU, and what guarantee holds for the noise-free encoder?
-
How does adaptive noise compare with fixed noise and no noise at the same network width on NCLT?
5 comments
W2. Predictive sufficiency against sufficiency for control
We agree that this can happen in principle. The causal state merges histories whose future words (including rewards) are the same under the behaviour policy, the policy that collected the data, so a distinction is lost only when it never shows in those future words. In that case, the data without actions carry no evidence that the two histories differ, so the limit comes from learning without actions and not from how ACSI builds the state.
A lost distinction affects control only when it changes the return that some target policy obtains. Where it does not, merging it loses nothing for control and saves states. The cost for control is therefore confined to one case, a distinction that never shows in the recorded data and yet changes the return of some target policy.
The new Corollary 1 (Sec. 2, App. B) states exactly when this case cannot arise. The causal state determines the future words of every policy that acts on it, and with them the values, if and only if every distinction that some action sequence would reveal also shows under the behaviour policy. Under this condition, a state learned without actions is sufficient for control, for any target policy and not only the PPO policy we tested. The new Lemma 3 (App. B.1) gives a simple test. The condition holds when the behaviour policy tries every action, and the next word shows its effect. Remark 1 gives a case in which the condition fails (App. B.1), and the paper records this case as a limitation (App. F).
On the tasks we tested, a target policy acts well on the causal state learned under the behaviour policy. Ten of our eleven tasks satisfy the test of Lemma 3, since the next observation shows the effect of the action through the change of the agent's position. PPO, the target policy here, trained on the code book readout of the frozen ACSI state at its label-free size solves 262 of 264 runs over the eleven RL tasks (Table 14 (a) of the re-uploaded PDF). On task02-keydoor3 and task13-oracle, the two memory tasks we test, the same readout solves 24 of 24 runs on each with the ACSI state, against 2 and 22 for InFOM and 5 and 11 for the DreamerV3 RSSM (Table 14 (c) in our reply to W4).
Our reply to Reviewer uDeq W2 gives the same argument.
W3 and Q2. NCLT row label and variants at the same network width
We agree, and the revised manuscript clarifies this label. The NCLT row marked as adaptive noise came from a wider network with fixed noise. Every learned NCLT row in Table 1 of the re-uploaded PDF uses , the ACSI row is ACSI (adaptive noise) at with its label clarified, and Table 15 compares the three variants at that width.
Table 15 of the re-uploaded PDF, App. E.3.
| NCLT, | aliasing AUC | official AUC | recall@1 |
|---|---|---|---|
| ACSI (no noise) | 0.865 | 0.899 | 0.087 |
| fixed | 0.883 | 0.901 | 0.072 |
| ACSI (adaptive noise), | 0.890 | 0.902 | 0.074 |
At the same width and with the same six seeds, both noise variants raise the aliasing AUC over ACSI (no noise), from 0.865 to 0.883 with fixed noise and to 0.890 with the noise controller, and keep the official AUC level. The noise controller takes one target per data set, and the target 0.40 was chosen by a sweep on the reported NCLT sessions, while at the default 0.15 it stays level with ACSI (no noise), at 0.861 aliasing and 0.895 official AUC (Table 15). At , the width of the submitted row, both noise variants reach 0.892 aliasing AUC against 0.880 for ACSI (no noise) (Table 15).
W1 and Q1. Whether the fed-back state is bounded
We agree that the layer norm acts only on the readout, so it does not bound the fed-back state, and we have clarified this point in the revised manuscript.
The GRU feeds back the raw state ht, and only the readout is normalised. Lemma 2 shows that the GRU itself nevertheless keeps the fed-back state bounded. The deployed state, which carries no noise, stays within 1 in every component for any weights and any sequence length, because each GRU step averages the previous state with a tanh candidate (Lemma 2(a)). During training, the state before the noise of the current step is added has a root-mean-square size per component below a constant B that does not grow with time (Lemma 2(b)). The measurements agree with both bounds. On three Bunniescape tasks and NCLT, the deployed state has a root-mean-square size of 0.46 to 0.82 per component and a largest value over time of 0.52 to 0.87, within the bound of 1, and with the training noise fed back the largest value is 0.53 to 0.97, against B of 1.6 to 4.7 (Table 22).
Table 22 of the re-uploaded PDF, App. E.6, selected columns.
Tables 19 and 20 indicate that the controlled noise does constrain this bounded state. Under the noise controller, the measured contraction ρ^K lies between 0.073 and 0.151 on all eleven tasks (Table 20), against up to 0.319 without noise (Table 19). The compression comes from the controlled noise accumulated over many steps, not from the capacity of a single step, which the revised manuscript states only as a loose bound. For the noise-free encoder you ask about, which is the one we deploy, two guarantees hold. Its state is bounded, and it is sufficient under the behaviour policy, the policy that collected the data, whenever it predicts the occupancy exactly. Its compression is shown by measurement, with the ARI rising from 0.741 to 0.766 and the SPREAD falling from 0.170 to 0.148 at unchanged AUC (Table 1 of the re-uploaded PDF). Within one causal state, the residual of the noise-free recursion, read after 80 replayed words, stays below one on every task, and training with the controlled noise lowers it on eight of nine tasks, from 0.198 to 0.034 on task01 (Table 21 of the re-uploaded PDF).
Normalising inside the loop would bound the state by construction, but it costs accuracy. On NCLT, normalising inside the loop lowers the official AUC of ACSI (adaptive noise) from 0.902 to 0.846 and the aliasing AUC from 0.890 to 0.792 (Table 23 of the re-uploaded PDF), so we keep the layer norm on the readout and obtain a bounded state without the cost in accuracy.
The bounds on the deployed state and the noisy state are stated formally in Lemma 2 (Sec. 2, proof in App. B.5) and measured in Table 22 (App. E.6).