acceptodds
Under review as a conference paper at ICLR 2027

Action-free Causal State Inference for Reinforcement Learning

Abstract

In partially observable environments, the state an agent learns from its history decides how well and how efficiently reinforcement learning succeeds, and causality gives that state its right target, the causal state, which groups histories by the future they lead to and helps the agent tell apart precisely the situations that matter, whereas predictive states may also keep irrelevant history. In this paper, we extend causal state learning to the action-free setting, as humans infer the plot of a film without acting in it, and prove that, under a mild condition on data collection, the causal state learned from observations and rewards alone loses nothing relevant to predicting the future and can be even more compact than its action-based counterpart. We further characterize how fast forgetting removes the rest, so its strength can be set automatically. Building on this, Action-free Causal State Inference (ACSI) injects noise into its recurrent memory, so that prediction preserves what the future requires while the rest decays, with the noise self-calibrated to a target forgetting rate rather than tuned per task. Empirically, ACSI recovers ground-truth causal states 9 AUC points better than action-conditioned, contrastive and bisimulation baselines, and is the only frozen representation to solve both control tasks that provably require memory, using 2.6–5.7X fewer frames than recurrent reinforcement learning. On real robot laser streams, it identifies revisited places 4 AUC points better than all learned baselines, offering a principled route to compact states from action-free data that make reinforcement learning more sample-efficient. The code of our method and environments will be released upon acceptance.

Primary area:
reinforcement learning
Keywords:
Causal States, Action-free Representation Learning, Partially Observable Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EAaiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns recurrent state representations from observation–reward sequences without action labels. It combines future-occupancy prediction with adaptive recurrent noise and evaluates the learned states on synthetic memory tasks and real laser data.

Critical strengths
  1. The distinction between useful predictive information and unnecessary history is well motivated and gives the method a clear focus.

  2. The synthetic tasks allow direct checks of state recovery, and the ablations help explain the effects of recurrent noise.

Critical weaknesses
  1. The channel-capacity argument requires a bounded recurrent signal, but LayerNorm appears to act only on the readout. The raw noisy hidden state is still fed back, leaving this assumption unjustified.

  2. Proposition 1 establishes predictive sufficiency under the collection policy. The stronger claim about preserving information for control needs additional justification, since another policy may require distinctions that the representation removes.

  3. The NCLT result labeled as adaptive noise in Table 1 comes from a wider network with fixed noise in Table 14. I suggest correcting the label and comparing the variants at the same network width.

  4. The best discrete-control results use true-state information to select the codebook size. I suggest testing label-free selection and applying the same one-hot readout to the no-noise encoder and competing methods.

Questions for authors
  1. Which state is normalized and fed back into the GRU, and what guarantee holds for the noise-free encoder?

  2. How does adaptive noise compare with fixed noise and no noise at the same network width on NCLT?

5 comments
AuthorsOpenReview ↗

W1 and Q1. Whether the fed-back state is bounded

We agree that the layer norm acts only on the readout, so it does not bound the fed-back state, and we have clarified this point in the revised manuscript.

The GRU feeds back the raw state hth_t, and only the readout is normalised. Lemma 2 shows that the GRU itself nevertheless keeps the fed-back state bounded. The deployed state, which carries no noise, stays within 1 in every component for any weights and any sequence length, because each GRU step averages the previous state with a tanh candidate (Lemma 2(a)). During training, the state before the noise of the current step is added has a root-mean-square size per component below a constant BB that does not grow with time (Lemma 2(b)). The measurements agree with both bounds. On three Bunniescape tasks and NCLT, the deployed state has a root-mean-square size of 0.46 to 0.82 per component and a largest value over time of 0.52 to 0.87, within the bound of 1, and with the training noise fed back the largest value is 0.53 to 0.97, against BB of 1.6 to 4.7 (Table 22).

Table 22 of the re-uploaded PDF, App. E.6, selected columns.

DataRMSmaxRMS, noisymax, noisyBB
task01-keydoor0.8160.8740.8220.9734.7
task02-keydoor30.4640.5230.4640.5301.6
task12-relay0.5030.6420.5040.6441.9
NCLT0.5130.6790.5140.6912.5

Tables 19 and 20 indicate that the controlled noise does constrain this bounded state. Under the noise controller, the measured contraction ρ^K\hat{\rho}_K lies between 0.073 and 0.151 on all eleven tasks (Table 20), against up to 0.319 without noise (Table 19). The compression comes from the controlled noise accumulated over many steps, not from the capacity of a single step, which the revised manuscript states only as a loose bound. For the noise-free encoder you ask about, which is the one we deploy, two guarantees hold. Its state is bounded, and it is sufficient under the behaviour policy, the policy that collected the data, whenever it predicts the occupancy exactly. Its compression is shown by measurement, with the ARI rising from 0.741 to 0.766 and the SPREAD falling from 0.170 to 0.148 at unchanged AUC (Table 1 of the re-uploaded PDF). Within one causal state, the residual of the noise-free recursion, read after 80 replayed words, stays below one on every task, and training with the controlled noise lowers it on eight of nine tasks, from 0.198 to 0.034 on task01 (Table 21 of the re-uploaded PDF).

Normalising inside the loop would bound the state by construction, but it costs accuracy. On NCLT, normalising inside the loop lowers the official AUC of ACSI (adaptive noise) from 0.902 to 0.846 and the aliasing AUC from 0.890 to 0.792 (Table 23 of the re-uploaded PDF), so we keep the layer norm on the readout and obtain a bounded state without the cost in accuracy.

The bounds on the deployed state and the noisy state are stated formally in Lemma 2 (Sec. 2, proof in App. B.5) and measured in Table 22 (App. E.6).

AuthorsOpenReview ↗

W2. Predictive sufficiency against sufficiency for control

We agree that this can happen in principle. The causal state merges histories whose future words (including rewards) are the same under the behaviour policy, the policy that collected the data, so a distinction is lost only when it never shows in those future words. In that case, the data without actions carry no evidence that the two histories differ, so the limit comes from learning without actions and not from how ACSI builds the state.

A lost distinction affects control only when it changes the return that some target policy obtains. Where it does not, merging it loses nothing for control and saves states. The cost for control is therefore confined to one case, a distinction that never shows in the recorded data and yet changes the return of some target policy.

The new Corollary 1 (Sec. 2, App. B) states exactly when this case cannot arise. The causal state determines the future words of every policy that acts on it, and with them the values, if and only if every distinction that some action sequence would reveal also shows under the behaviour policy. Under this condition, a state learned without actions is sufficient for control, for any target policy and not only the PPO policy we tested. The new Lemma 3 (App. B.1) gives a simple test. The condition holds when the behaviour policy tries every action, and the next word shows its effect. Remark 1 gives a case in which the condition fails (App. B.1), and the paper records this case as a limitation (App. F).

On the tasks we tested, a target policy acts well on the causal state learned under the behaviour policy. Ten of our eleven tasks satisfy the test of Lemma 3, since the next observation shows the effect of the action through the change of the agent's position. PPO, the target policy here, trained on the code book readout of the frozen ACSI state at its label-free size solves 262 of 264 runs over the eleven RL tasks (Table 14 (a) of the re-uploaded PDF). On task02-keydoor3 and task13-oracle, the two memory tasks we test, the same readout solves 24 of 24 runs on each with the ACSI state, against 2 and 22 for InFOM and 5 and 11 for the DreamerV3 RSSM (Table 14 (c) in our reply to W4).

Our reply to Reviewer uDeq W2 gives the same argument.

AuthorsOpenReview ↗

W3 and Q2. NCLT row label and variants at the same network width

We agree, and the revised manuscript clarifies this label. The NCLT row marked as adaptive noise came from a wider network with fixed noise. Every learned NCLT row in Table 1 of the re-uploaded PDF uses d=128d=128, the ACSI row is ACSI (adaptive noise) at ρ⋆=0.40\rho^\star=0.40 with its label clarified, and Table 15 compares the three variants at that width.

Table 15 of the re-uploaded PDF, App. E.3.

NCLT, d=128d=128aliasing AUCofficial AUCrecall@1
ACSI (no noise)0.8650.8990.087
fixed σ=0.03\sigma=0.030.8830.9010.072
ACSI (adaptive noise), ρ⋆=0.40\rho^\star=0.400.8900.9020.074

At the same width and with the same six seeds, both noise variants raise the aliasing AUC over ACSI (no noise), from 0.865 to 0.883 with fixed noise and to 0.890 with the noise controller, and keep the official AUC level. The noise controller takes one target per data set, and the target 0.40 was chosen by a sweep on the reported NCLT sessions, while at the default 0.15 it stays level with ACSI (no noise), at 0.861 aliasing and 0.895 official AUC (Table 15). At d=256d=256, the width of the submitted row, both noise variants reach 0.892 aliasing AUC against 0.880 for ACSI (no noise) (Table 15).

Reviewer_Sb5sOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns history representations for partially observable RL from action-free data: sequences of observation and reward pairs (“words”). The target is the causal state from computational mechanics, which is the equivalence class of histories that induce the same distribution over future words. This is the minimal sufficient statistic of the past for predicting the future. Authors show that if the data policy satisfies a condition on how it depends on history, dropping actions loses nothing and can yield a strictly smaller state. Their method, ACSI has three main stages and experiments use Bunniescape, a new set of tasks with exactly enumerable causal states, PPO on frozen representations, and the NCLT laser dataset. ACSI reaches 0.959 state-recovery AUC against 0.867 for the best baseline. The paper also includes extensive mechanism studies and ablations.

Critical strengths
  1. Principled target. The paper frames representation learning as recovering the causal state and separates sufficiency from minimality. Much prior work blurs the two.
  2. Action-free identification. Props. 1–2 show that dropping actions loses nothing under a stated condition on the data policy, and can even shrink the state.
  3. Exact-ground-truth benchmark. Bunniescape enumerates the true causal states, so recovery and redundancy can be measured directly.
  4. Strong state recovery. The occupancy-prediction encoder alone clearly outperforms a broad set of baselines.
  5. Thorough ablations. The ablations cover noise placement, two-phase training, fixed versus adaptive σ, a VIB rate penalty and different backbones.
Critical weaknesses
  1. Small minimality gains for a lot of added complexity. The noise and controller target minimality. SPREAD falls only from 0.159 to 0.138, the state-count ratio doesn’t improve (1.18 → 1.21), and SPREAD is flat or worse on several tasks. The method adds considerable complexity, and it isn’t compared with simpler regularizers.
  2. The control results rely on ground truth. The 24/24 result uses a codebook sized from the true state count, and baselines get fewer runs.
  3. The NCLT comparison isn’t matched. The reported model uses d = 256 and a fixed σ, while the baselines use d = 128. With the actual method, the lead mostly disappears.
  4. The setting is narrow. The tasks are small and deterministic, the data comes from random policies, and Assumption 1 excludes realistic logs.
Questions for authors
  1. Is the minimality machinery needed? Can you compare full ACSI against simpler ways to compress the state?
  2. Do the control results hold without ground-truth information? What are the results when the codebook is chosen from training or validation data alone?
  3. On NCLT, the reported ACSI model uses d = 256 while all baselines use d = 128. Can you provide a capacity-matched comparison? Moreover, the NCLT row in Table 1 is labelled “adaptive noise,” but Appendix E.3 says it uses a fixed σ = 0.03. Can you report results with the adaptive controller and correct the label? Why does the controller give no gain over the no-noise model on NCLT?
  4. Does the method work when Assumption 1 fails, or in stochastic environments? All the Bunniescape data comes from uniformly random policies in deterministic environments. How does ACSI do with data from expert or ε-greedy policies, and on at least one stochastic task? These are the realistic settings for learning without actions, and the ones where causal states differ most from latent states.
6 comments
AuthorsOpenReview ↗

W1 and Q1 (part 1/2). Minimality gains against simpler regularisers

Your reading of the numbers is the right one, and tracing it task by task shows that the gain differs from task to task. Both ACSI arms are our method, the noise-refining ACSI (no noise), where redundant memory remains at a cost of at most 0.011 AUC on any task. The averages you quote mix four tasks where the noise lowers the SPREAD with five where it does not. On the four tasks where ρ^K\hat{\rho}_K of ACSI (no noise) ends highest, 0.237 to 0.319 against ρ⋆=0.15\rho^\star=0.15, the SPREAD falls by 0.046 at unchanged AUC. On the other five, σ⋆\sigma^\star stays at 0.025 or less, the AUC changes by at most 0.011, and the SPREAD rises by at most 0.031, which is where the flat or worse SPREAD you noticed comes from. All eleven use the conservative default ρ⋆=0.15\rho^\star=0.15, and on task09 the target ρ⋆=0.05\rho^\star=0.05, the best target for task09 in the sweep of App. E.7, lowers the SPREAD from 0.122 to 0.067 while the true causal-state probe rises and the AUC stays within 0.01.

Before the detailed results, we explain the measure in which they are reported. Your note that the state-count ratio moved only from 1.18 to 1.21 led us to a sharper measure, since a count of states does not show whether they merge causal states, as CPC's state does. The re-uploaded PDF therefore reports the adjusted Rand index (ARI) against the true causal states, which penalises both merged and split causal states, and Table 1 recomputes every average over the same eleven tasks.

Only the controlled noise compresses the state at no cost in sufficiency, as the comparison with simpler regularisers that you suggested shows in Table 18 of the re-uploaded PDF. It applies four simple compressions to ACSI (no noise) on the same predictor at the same budget on the same eleven tasks, dropout where the controlled noise enters, decoupled weight decay up to 1.0, an L2 penalty on hh, and a bottleneck GRU at d=32d=32, each at its strength of best mean ARI, chosen first. The bottleneck at d=32d=32 compresses less and has an AUC of 0.859 against 0.954 for ACSI (no noise), the best weight decay reaches an ARI of only 0.749, and dropout and the L2 penalty lose sufficiency as their strength grows.

Table 18 of the re-uploaded PDF, App. E.5.

RegulariserStrengthARI ↑\uparrowSPREAD ↓\downarrowAUC ↑\uparrow
Dropout on hh0.1†0.1^\dagger0.661±0.0090.661 \pm 0.0090.198±0.0160.198 \pm 0.0160.762±0.0110.762 \pm 0.011
0.30.630±0.0140.630 \pm 0.0140.229±0.0100.229 \pm 0.0100.693±0.0090.693 \pm 0.009
0.50.601±0.0070.601 \pm 0.0070.263±0.0320.263 \pm 0.0320.653±0.0050.653 \pm 0.005
Weight decay10−410^{-4}0.740±0.0080.740 \pm 0.0080.168±0.0050.168 \pm 0.0050.953±0.0080.953 \pm 0.008
10−310^{-3}0.739±0.0070.739 \pm 0.0070.168±0.0050.168 \pm 0.0050.954±0.0080.954 \pm 0.008
10−210^{-2}0.739±0.0060.739 \pm 0.0060.168±0.0050.168 \pm 0.0050.953±0.0080.953 \pm 0.008
0.10.745±0.0070.745 \pm 0.0070.174±0.0050.174 \pm 0.0050.955±0.0110.955 \pm 0.011
0.3†0.3^\dagger0.749±0.0100.749 \pm 0.0100.182±0.0040.182 \pm 0.0040.947±0.0100.947 \pm 0.010
1.00.748±0.0190.748 \pm 0.0190.158±0.0300.158 \pm 0.0300.911±0.0230.911 \pm 0.023
L2 penalty on hh10−3†{10^{-3}}^\dagger0.696±0.0090.696 \pm 0.0090.238±0.0100.238 \pm 0.0100.930±0.0090.930 \pm 0.009
10−210^{-2}0.644±0.0220.644 \pm 0.0220.331±0.0410.331 \pm 0.0410.890±0.0270.890 \pm 0.027
10−110^{-1}0.602±0.0180.602 \pm 0.0180.390±0.0270.390 \pm 0.0270.872±0.0210.872 \pm 0.021
Bottleneckd=32†d=32^\dagger0.740±0.0350.740 \pm 0.0350.155±0.0390.155 \pm 0.0390.859±0.0230.859 \pm 0.023
References, printed values of Table 6 of the re-uploaded PDF
ACSI (no noise)0.7410.1700.954
ACSI (adaptive noise)ρ⋆=0.15\rho^\star=0.150.7660.1480.956

The compression also helps downstream RL control when the policy reads the state through a discrete code book and the task requires memory. On task02 (three rooms), PPO on the label-free code book of ACSI solves 16 of 24 runs without the noise and 24 of 24 with it, in 1.05M and 0.60M frames (Table 19 (ii)). With a code book of only ∣S⋆∣|\mathcal{S}^\star| codes the gap widens to 24 against 12. Under the task02 protocol of Table 1, whose encoders have width d=512d=512, weight decay comes closest, solving 19 to 20 of 24 runs, but only the noise controller solves every run. At the narrower width of 128, which the table below does not show, a bottleneck matches the noise controller (App. E.5).

Table 19 (ii) of the re-uploaded PDF, App. E.5.

(ii) ArmMCSSRM_{\mathrm{CSSR}} (label-free)M=∣S⋆∣=280M=\lvert\mathcal{S}^\star\rvert=280M=2∣S⋆∣=560M=2\lvert\mathcal{S}^\star\rvert=560
ACSI (adaptive noise)24/24, 0.60M24/24, 1.10M24/24, 0.60M
ACSI (no noise)16/24, 1.05M12/24, >4.21M17/24, 1.10M
Dropout on hh, 0.13/24, >5.01M1/24, >5.01M4/24, >5.01M
Weight decay, 0.320/24, 0.65M19/24, 1.55M20/24, 0.80M
L2 penalty on hh, 10−310^{-3}2/24, >5.01M0/24, >5.01M6/24, >5.01M
Bottleneck, half width (d=256d=256)4/24, >5.01M4/24, >5.01M5/24, >5.01M
AuthorsOpenReview ↗

W1 and Q1 (part 2/2). Minimality gains against simpler regularisers

This part reports the task-by-task evidence behind part 1 and the scope of the gain. ρ^K\hat{\rho}_K of ACSI (no noise) is measured without labels, before any noise is added. Over the nine tasks where it was logged, its Spearman correlation is 0.88 with the fall in SPREAD and 0.68 with the change in ARI, and on the four tasks where it ends highest, the ARI rises by 0.052, and the SPREAD falls by 0.046 (Table 19 (i), App. E.5). The simple regularisers compared in part 1 treat these four tasks and the other five alike, dropout and the bottleneck compressing both, weight decay neither, and the L2 penalty raising the SPREAD in both (App. E.5).

Table 19 (i) of the re-uploaded PDF, App. E.5.

(i) Taskρ^K\hat{\rho}_K, no noise, end of trainingσ⋆\sigma^\starΔ\DeltaSPREADΔ\DeltaARIMemory needed
task01-keydoor0.3190.087−0.074-0.074+0.032+0.032yes
task02-keydoor30.2370.005−0.039-0.039+0.055+0.055yes
task03-lampnot logged0.108−0.092-0.092+0.070+0.070yes
task04-mirrornot logged0.110−0.053-0.053+0.021+0.021yes
task05-gate0.1750.006+0.006+0.006+0.014+0.014no
task06-swap0.0580.000+0.023+0.023−0.029-0.029no
task07-hazard0.2010.025+0.010+0.010−0.018-0.018no
task09-vault0.2130.011+0.008+0.008+0.028+0.028no
task12-relay0.2530.009−0.020-0.020−0.007-0.007no
task13-oracle0.0710.001+0.031+0.031−0.013-0.013yes
task14-oracle_push0.2440.021−0.050-0.050+0.127+0.127yes

The gain has clear limits. With continuous frozen states fed to PPO, the noise controller changes little except on task02, where the solved runs rise from 6 to 13 of 24 (Table 12 of the re-uploaded PDF). It does not help on tasks that every input already solves, nor on robomimic, where σ\sigma stays at its floor. On NCLT, it raises the aliasing AUC from 0.865 to 0.890 at the target ρ⋆=0.40\rho^\star=0.40, which was chosen by a sweep on the reported NCLT sessions.

The deployed encoder runs without noise at the same cost as ACSI (no noise), and in training the noise controller adds about 3 percent of wall time against a matched two-phase schedule without noise, which itself adds 88 to 206 percent against ACSI (no noise). The noise controller has one setting, the target ρ⋆\rho^\star, and each added part is needed. The two-phase schedule avoids the loss of 0.105 AUC on task02 that noise at σ=0.03\sigma=0.03 from the start of training causes, and a fixed σ=0.1\sigma=0.1, the level that merges task01 and task09, collapses task02 in every run, whereas the noise controller lowers the task02 SPREAD from 0.167 to 0.128 at an AUC at or above that of ACSI (no noise) (App. E.7).

Taken together with part 1, the noise controller compresses most where ρ^K\hat{\rho}_K of ACSI (no noise) ends highest, and there, at the main width of 512, it is the only one we tested that both solves every downstream run and keeps sufficiency and minimality, weight decay coming close in control but not in ARI.

This gain is the second step of ACSI. The first, learning the causal state without actions, does not depend on the noise, since ACSI (no noise) already separates the true causal states of Bunniescape with an average AUC of 0.954, against 0.853 for InFOM, the best baseline, and at most 0.800 for the baselines that read actions (Table 1 of the re-uploaded PDF).

AuthorsOpenReview ↗

W2 and Q2. Control results without ground truth and the run counts

The result does not depend on the number of true causal states. The code book size is set from the training episodes alone, without labels. Codes are fitted by k-means on half of these episodes and merged until a χ2\chi^2 test on their held-out futures distinguishes every remaining pair, which is the state-merging step of CSSR (Shalizi and Klinkner, 2004). Under this rule, the one-hot readout of the ACSI state solves 262 of 264 runs over the eleven RL tasks.

The comparison with the baselines is on equal terms. All experiments, every baseline included, use the same six seeds, 0 to 5, fixed before any run, so that no seed is selected, and on task02 and task13 (oracle task) both ACSI arms, InFOM and the DreamerV3 RSSM are read out through a code book sized by the same rule. These two tasks are where the memory of the state is tested, since a memoryless policy already reaches 0.94 to 1.00 of the optimal value on five of the seven levels we checked, whereas on task13 the best memoryless policy we found falls short and in task02 the carried key is hidden (Table 13 of the re-uploaded PDF). On task02, ACSI solves 16 of 24 runs without and 24 with the noise, against 2 and 5 for InFOM and the DreamerV3 RSSM, while on task13 every sufficient state benefits (Table 14 (c)). The noise refines the state on task02, where control needs a compact readout, and leaves task13 at 24 of 24. In our implementation InFOM predicts the same discounted occupancy under the same TD(λ\lambda) schedule, but its occupancy head is conditioned on one latent intention per trajectory, so that its state alone need not be sufficient (App. D.2 of the re-uploaded PDF).

Table 14 (c) of the re-uploaded PDF, App. E.2.

One-hot readout, label-free sizetask02-keydoor3task13-oracle
ACSI (adaptive noise)24/2424/24
ACSI (no noise)16/2424/24
InFOM2/2422/24
DreamerV3 RSSM5/2411/24

The chosen size is not tuned. It lies between 1.0 and 5.3 times the number of true causal states, and the readout is insensitive to larger sizes, with the ACSI state solving every run from 128 codes onwards.

Reviewer_uDeqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present Action-free Causal State Inference (ACSI) which learns “causal states” (minimal Markov states) to predict future observations and rewards. Overall I found the paper very interesting, but based on my limited understanding, wondered if the "action-free" part is too limiting to be useful in general control; but look forward to discussing with authors

Critical strengths
  • Interesting and sensible implementation of state representation that maximize predictability of future observation–reward occupancy while using adjustable noise in the recurrent state to “forget” information irrelevant to prediction in a measured way.
Critical weaknesses

Large weaknesses:

  1. I found some of the writing style difficult to understand in sections: some parts assume the reader had more context than they do, and some parts require more specificity. I’ve listed some examples in the “small weaknesses” section below. One repeated example is the role of noise to forget irrelevant prediction information prediction, but it could be presented more clearly to the reader in the introduction how.
  2. Lack of motivation for “Action-free identification”. While there is an example of “humans infer the plot of a film without acting in it” on Line 17, the problem was not clearly motivated yet why action-free is important in this domain. Proposition 1 mentions “causal state is therefore never finer than the interventional class, of which it is a coarsening, and it is sufficient for predicting the future under π.” which makes sense, but does this loss of distinction lose anything required for us to decide good actions? A uniform random behavior policy might lead to a collapse of two settings, each with two actions, one setting only “left” action gets the reward, other setting only “right” action gets reward. Here, we’d collapse into a single state, and a future policy could never learn which action to take based on the setting it observes.

Small weaknesses:

  • Line 013: “and causality gives that state its right target, the causal state, which groups histories by the future they lead to and helps the agent tell apart precisely the situations that matter” I didn’t quite understand
  • Line 022: “injects noise into its recurrent memory, so that prediction preserves what the future requires while the rest decays” wasn’t obvious to me why
  • Line 038: “its state should distinguish exactly these situations and nothing more” – unclear what is meant by these situations
  • Line 051: “and removing that timing signal leaves 286 effective states against the 279 that task actually has (Table 5)” the reader has no context yet what these details mean, left with just a reference to the appendix
  • Line 080: “while noise in its memory erases everything prediction does not restore” I was confused by this line
  • Line 081: “so that a policy trained on it spends experience only on the situations that matter.” I was confused what is meant here.
  • Line 082: “This Gaussian noise” this hasn’t been introduced to the reader yet
  • Line 092: “recurrent noise erases the remaining redundancy” unclear what noise is being referred to here
  • Line 096: “9 AUC points above” does this mean additional 9% AUC?
  • Line 106: “so that a statistic sufficient for the future is sufficient for value, and which reduces to the observation alone where no reward is recorded” – wouldn’t the reward
  • Line 116: how does “is a function of h(H)” show sufficiency? I didn’t follow
  • Line 221: “therecurrent” → “the recurrent”, in Figure 2. PS: were all rewards meant to be “r=0”?
Questions for authors
  1. Line 044: “observation and reward pairs, which we call words” why called words ?
  2. Are the claims minimal representation (part of the definition of a causal state on line 42) satisfied? I understand Gaussian noise is added to the hidden state as a mechanism to discard redundant memory, but this can discard useful memory too can it; and overall it can reduce the representation, but it’s not minimal is it? (just wondering since “causal state” seems central to this paper)
6 comments
AuthorsOpenReview ↗

W1 and small weaknesses (part 1/2). Clarity of the introduction

We understand your concern that the introduction relied on context a reader new to the area does not yet have, particularly regarding how the noise makes the state forget, and we took several measures to resolve this and to improve readability.

We revised many expressions and reordered the exposition so that a reader new to the area can follow it. The introduction defines its terms in order, first a word and a history, then the causal state, which it defines through the distribution of future words that a history predicts. It describes the mechanism before its effects, stating that the encoder is trained to predict future words, that Gaussian noise enters its recurrent state during training, set by a noise controller, that prediction keeps what the future needs, and that the noise removes the rest. The names are used consistently. The two variants of the method are called ACSI (adaptive noise) and ACSI (no noise) in the text, tables, and figures, and the mechanism is called the noise controller.

We also added illustrations that guide the reader. Figure 1 of the re-uploaded PDF has two panels, cited at the definition of the causal state, in which two histories with different words but the same distribution of future words merge into one causal state (e), and two histories that end on the same word but have different futures remain distinct (f). The caption of the method figure states that the two example histories share one causal state and are therefore embedded at the same point, and it explains why every word in the example carries the reward r=0r = 0.

Finally, we simplified and tidied the notation so that the theory is easier to read. Each central quantity has its own symbol. The causal state is written as ξO\xi_{\mathrm{O}} and the interventional class as ξI\xi_{\mathrm{I}}, so that σ\sigma refers only to the noise level. Likewise, the italic HH denotes the history and the upright H\mathrm{H} the entropy.

AuthorsOpenReview ↗

W1 and small weaknesses (part 2/2). Line-by-line revisions

Each of the twelve passages you marked is revised, as listed in the table below.

LineChange
013Old "groups histories by the future they lead to and helps the agent tell apart precisely the situations that matter"
066 to 068New "a natural target for that state is the causal state, which treats two histories as equivalent when they give the same distribution of future observations and rewards"
Note The causal state is defined by an explicit equivalence of histories.
022Old "injects noise into its recurrent memory, so that prediction preserves what the future requires while the rest decays"
077 to 079New "adds Gaussian noise to its memory at every step, so that information the prediction keeps using is restored while the rest is gradually attenuated"
Note The noise weakens everything stored in memory at every step, and only information the prediction keeps using is written back, so the rest fades.
038Old "its state should distinguish exactly these situations and nothing more"
102 to 105New "The state the agent needs is the causal state of a history … the class of all histories with the same distribution of future words under the behaviour policy that collected the logs"
Note The vague "situations" are replaced by causal states, defined where they first appear.
051Old "removing that timing signal leaves 286 effective states against the 279 that task actually has (Table 5)"
467 to 468New "What it removes is elapsed time, since the split of one true causal state by elapsed time that ACSI (no noise) retains falls in silhouette from 0.23 to 0.03 (Figure 6)"
Note The state counts and the appendix reference are removed from the introduction, and the timing signal is explained in Section 4, where its context is given, as elapsed time that the noise removes.
080Old "while noise in its memory erases everything prediction does not restore"
149 to 150New "Prediction keeps what the future needs and the noise removes the rest"
Note The confusing sentence is replaced by two plain statements placed right after the noise is introduced, prediction keeps what the future needs and the noise removes the rest.
081Old "so that a policy trained on it spends experience only on the situations that matter"
152New "PPO on its frozen discrete state solves every run on both memory tasks"
Note The vague claim is removed and replaced by the result it stood for, PPO on the frozen ACSI state, read through a label-free code book, solves every run on both memory tasks we test (Tables 13 and 14 of the re-uploaded PDF).
082Old "This Gaussian noise"
148New "ACSI therefore adds Gaussian noise to the recurrent state during training"
Note The noise is introduced, with where and when it enters, before it is referred to.
092Old "recurrent noise erases the remaining redundancy"
159 to 160New "compresses it towards minimality with the controlled noise. We prove that redundancy decays geometrically"
Note The bullet says the controlled noise, which refers back to the Gaussian noise added to the recurrent state and set by the noise controller at lines 148 to 149.
096Old "9 AUC points above the best baseline"
150 to 151New "ACSI separates the true causal states best … (AUC 0.956, against 0.853 for InFOM, the best baseline"
Note Both AUC values appear on their 00 to 11 scale, so the gap is an absolute difference of 0.1030.103 and not a relative percentage.
106Old "so that a statistic sufficient for the future is sufficient for value, and which reduces to the observation alone where no reward is recorded"
170 to 171New "Because every future word carries its reward, a statistic sufficient for the future words is sufficient for the future rewards and hence for the value of π\pi"
Note Each word contains its reward, so a statistic that predicts future words predicts future rewards and hence value, and the case without rewards names where it occurs, the laser logs of Section 4, where a word is only the observation.
116Old "is a function of h(H)h(H)"
186 to 188New "that is, two histories with the same value of hh have the same distribution of futures, so hh keeps everything the history says about the future"
Note The definition spells out why being a function of h(H)h(H) means hh is sufficient.
221Old "therecurrent channel" in Figure 2, with every word labelled "r=0r=0"
333 to 336New "noise in the recurrent channel" … "All six words carry r=0r = 0 because Bunniescape rewards only opening a door and reaching the exit."
Note The typographical error is corrected, and the caption explains why every reward is zero.
AuthorsOpenReview ↗

W2. Why action-free learning matters and what control loses

Action-free learning gains on three counts and has one cost for control, which your example identifies.

The first gain is that data recorded without actions become usable. Much recorded experience has this form, such as sensor logs of robots and vehicles, human demonstrations whose controls were not stored, and videos. In the paper, both real datasets are used this way by encoders that see no actions, NCLT, a laser stream from a robot driven by a person, and robomimic, demonstrations by human operators (Sec. 4). More broadly, a state learned without actions can draw on data collected by other agents and for other purposes, which widens the data available for reinforcement learning well beyond an agent's own interaction.

The second gain is that the state to be learned without actions is never larger than the state to be learned with actions, and it can be smaller. Not every action changes the state of the environment. In some environments the state evolves regardless of what the agent does, and in most environments some actions leave the state unchanged, such as a step into a wall in our grid worlds. In these cases, a state that tracks the actions carries redundancy, since histories that differ only in such actions reach the same state of the environment. A state learned without actions avoids this redundancy in every case, whether or not the actions change the environment. In theory, the causal state is never finer than the interventional class (Proposition 1), and it is strictly coarser when a symmetry between actions is invisible in the words (Proposition 2, Sec. 2). A coarser target has fewer states to tell apart. With the same data, each state then has more samples to learn from, and reinforcement learning on the state needs less experience, since the experience it needs grows with the number of states. On task04 (mirror task), exact enumeration gives 389 interventional classes against 195 causal states, and ACSI (no noise) follows the smaller partition with an AUC against the causal states of 0.991, against 0.989 for an encoder that reads actions (Sec. 4, App. E.6).

The third gain is that nothing is lost for prediction. Under the behaviour policy, the policy that collected the data, the causal state keeps everything that the history of observations and rewards predicts about the future, so it is sufficient for prediction and for the value of that policy (Proposition 1 and Corollary 1, Sec. 2). The effect of a past action largely shows in the observations that follow it, which is why leaving the actions out costs little. On NCLT, giving ACSI the logged actions raises the aliasing AUC by only 0.033 at d=128d = 128 and 0.012 at d=256d = 256 (App. E.3), and ACSI without actions reaches 0.890 at d=128d = 128, against 0.787 and 0.795 for RePo and the DreamerV3 RSSM, which read the actions (Sec. 4).

Turning to control, your example is correct in principle. Histories with the same future words under the behaviour policy share one causal state, so a distinction that never shows in those future words is lost. A lost distinction impacts the control only when it changes the return that some target policy obtains. Where it does not, merging it loses nothing for control and saves states. Among the distinctions that do change the return, those that show in the recorded data are not lost at all, because the reward is part of every word that ACSI predicts. The cost for control is therefore confined to one case, a distinction that never shows in the recorded data and yet changes the return of some target policy. The new Corollary 1 (Sec. 2, App. B) states exactly when this case cannot arise. The causal state determines the future words of every policy that acts on it, and with them the values, if and only if every distinction that some action sequence would reveal also shows under the behaviour policy. The new Lemma 3 (App. B.1) gives a simple test. The condition holds when the behaviour policy tries every action, and the next word shows its effect. Ten of our eleven tasks satisfy it, and PPO trained on the code book readout of the frozen ACSI state solves 262 of 264 runs over the eleven tasks (Table 14 (a) of the re-uploaded PDF). The exception is task04, where a symmetry between two actions hides the distinction, and your example fails the condition in the same way (Remark 1, App. B.1). There the smaller state of the second gain and a guarantee for every policy exclude each other, and App. F records this as a limitation. Only the state is learned without actions. The downstream policy is still trained with actions and rewards.

Taken together, these points say what actions contribute to state learning. Most of the state is already written in the observations and rewards, and actions are irreplaceable only in the one case that Corollary 1 states exactly, which the test of Lemma 3 lets a practitioner check on their own data.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response, to all reviewers and the Area Chair

We thank the three reviewers and the Area Chair, whose comments helped us improve the paper. The revised manuscript has been re-uploaded. The paragraphs below briefly summarise the improvements.

Reviewer uDeq, thank you for the careful reading. Guided by your points W1, W2, Q1 and Q2, we have revised the paper throughout, making its writing clearer and its theory more complete. Your line-by-line notes in W1 showed us where the writing stood in the reader's way. Following them, we clarified each of the twelve passages you marked and the introduction, and gave each concept its own symbol. Your two-setting example in W2 was especially valuable. It led us to add Corollary 1 and Lemma 3, and the paper states clearly that omitting actions loses nothing for prediction under the behaviour policy, the policy that collected the data, and when the causal state suffices for control and when it does not. Your question Q1 showed us that the analogy with language deserved stating, and the definition of a word states it. Your question Q2 made our claim on minimality more precise. We prove a bound on the history that the noisy training state retains and measure the compression of the deployed state, and the forgetting concerns differences between histories of the same causal state.

Reviewer Sb5s, thank you for the thorough and concrete suggestions. Guided by your points W1 to W4 and Q1 to Q4, the revised manuscript adds comparisons and further evidence. Your suggestion in W1 and Q1 to test simpler regularisers gave the paper a comparison it needed. We compared the noise controller with them on the same predictor and traced its gain task by task, which shows that it compresses most where ρ^K\hat{\rho}_K of ACSI (no noise) ends highest, and following your note on the state count, we score compactness against the true causal states. Your comments on the evaluation protocol in W2 and Q2 made it cleaner. We set the code book size without labels from the training episodes alone, run every experiment and baseline on the same seeds fixed in advance, and read both ACSI arms and the two strongest baselines through a code book sized by the same rule. Your comments on NCLT in W3 and Q3 led us to evaluate it at matched width with a clarified label and to split its official AUC into the pairs that history must separate and the pairs that the current scan already separates. Following your request in W4 and Q4, we evaluated on the two settings you asked for, human demonstration logs under occlusion and a task whose environment draws a hidden variable in each episode, and clarified that the theory admits expert behaviour policies and does not assume a deterministic environment. The paper states where the noise controller helps and where it does not, defines a target and criteria that stand apart from the mechanism, so that other methods can be scored on the same benchmark, and reports results on logs collected by human experts, where its state is level with or ahead of the action-free learned baselines.

Reviewer EAai, thank you for the precise technical comments. Guided by your points W1 to W4, Q1 and Q2, the revised manuscript states its assumptions explicitly and compares the methods under matched settings. Your observation in W1 that the layer norm acts only on the readout led us to justify boundedness directly. We proved a bound on the power of the fed-back state, measured that power in the trained encoders, clarified the text that attributed boundedness to the layer norm, and tested a layer norm inside the loop as an ablation. Your question Q1 let us state that the raw state is fed back and its normalisation is read, and that the deployed encoder stays bounded and keeps the sufficiency guarantee. Your distinction in W2 between prediction and control led to a further result. We proved a requirement under which the causal state determines the future words of every policy and therefore suffices for control, showed that this requirement is also necessary for the guarantee on future words, stated the case it excludes as a limitation, and reported that a different policy acts well on the causal state on the tasks we tested. Your comments on NCLT in W3 and Q2 led us to evaluate every learned NCLT encoder at the same width, clarify the label of the ACSI row, and compare the variants of ACSI at that width. Your point on the code book in W4 made the readout free of labels. We set its size from the training episodes alone and apply the same readout to both ACSI arms and the two strongest baselines.

The Summary of Changes page lists where the paper changed per review point, with key words in that reviewer's colour. We would be glad to discuss any point further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.