acceptodds
Under review as a conference paper at ICLR 2027

HETEROGENEOUS RECURRENCE FOR PARTIALLY OBSERVED CONTROL

Abstract

Agents for partially observed control usually rely on a single kind of recurrence. Yet the best memory depends on the hidden part of the state. When the hidden quantity is a derivative of the observation, as a velocity is of observed positions, the masked system is observable. Each new observation then carries a correction signal, and a recurrence can recover the hidden quantity by reading the current observation beside its memory. When the hidden quantity is an integral, no observation corrects its initial error, and a plain gated recurrence does as well. One agent may face both cases. In this paper, we therefore propose DuoCell, a heterogeneous recurrence for partially observed control. It splits one fixed state budget between a structured prior-innovation path for the first case and a GRU for the second. On masked MuJoCo with velocities hidden, DuoCell leads a width-matched GRU by +0.52 normalized IQM with fewer parameters. It also beats eleven other baselines, including an LSTM and a tuned TransformerXL, and its lead over the GRU baselines holds at five times the training budget. Because the regime follows from the sensor layout, it can be read off before training. We fixed this prediction before running eight unseen environments, and it held in both regimes. Where the criterion expected a benefit, the structured recurrence gained +0.33 normalized IQM over the GRU.

Primary area:
reinforcement learning
Keywords:
partial observability, reinforcement learning, recurrent neural networks, memory architectures, observability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vgmQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses that recurrent memory for partially observed control should depend on what the observation mask hides. Hidden velocities define an observable “derivative regime,” where memory combined with the current observation can recover the hidden state. Hidden positions define an unobservable “integral regime,” where a plain GRU suffices. DuoCell splits a fixed state budget between a GRU and a structured prior–innovation recurrence, with a gated reread of the current observation. On POBAX masked MuJoCo, it outperforms twelve baselines in the derivative regime, including a width-matched GRU (+0.52 normalized IQM) and a tuned TransformerXL. A preregistered test on four held-out morphologies confirms the predicted direction of benefit in both regimes. However, a GRU with the same reread shows the same pattern and outperforms DuoCell out of domain.

Critical strengths
  1. Predictive insight beyond the benchmark split. Building on POBAX’s masks, the paper predicts when observation rereading should help and confirms the predicted direction at the regime level on held-out tasks. This offers useful guidance for memory design beyond DuoCell itself.

  2. Strong, carefully controlled empirical results. DuoCell beats twelve baselines in the derivative regime, including a width-matched GRU and a tuned TransformerXL. Capacity controls and factorial ablations strengthen the evidence for its sample-efficiency gains.

Critical weaknesses
  1. The evidence supports the criterion, not DuoCell. On the eight held-out environments, a GRU with the same gated reread beats DuoCell in both the derivative and integral regimes (Table 6), despite trailing it in domain. What generalizes is the reread, not the heterogeneous cell.

  2. The evaluation is too narrow. Training runs for 1M steps, versus 50M in POBAX. At 5M steps an untuned TransformerXL closes most of the gap, and at equal wall-clock the two tie. DuoCell's advantage may not hold at the benchmark's own training scale.

Questions for authors
  1. Mixed-regime evaluation. DuoCell is motivated by platforms that mix both regimes, but every experiment hides either all velocities or all positions. Can the authors test a mixed mask?

  2. Iterated correction. The structured path corrects once per step. Would iterating the correction–reread loop within a step help, as in an iterated Kalman update? Could a single correction step explain why zeroing the innovation has no detected effect?

Reviewer_DaAwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the best recurrent memory for partially observed control depends on what the observation hides. Using a discretized double integrator, the authors show that hiding velocities makes the system observable and new observations carry information about the hidden state. Based on this, the authors build DuoCell, which splits a fixed state budget between a structured path and a GRU. The authors compare the performance of DuoCell against baselines on masked-MuJoCo tasks of POBAX.

Critical strengths
  1. The experiments and evaluation are in general rigorous and clean. The authors use paired seeds, IQM with bootstrap intervals, and frozen protocols. Robustness to the normalizer, raw returns, and deterministic final-policy evaluation are checked.
Critical weaknesses
  1. The theory the paper proposes does not 100% imply the architecture’s advantage. Proposition 1 describes what any causal estimator can recover. But it does not say one architecture should beat another.

  2. DuoCell is phrased as a safe choice when the regime is unknown. However, in the integral regime DuoCell and the GRU give quite close results. The authors note this and run pre-registered replication twice with only the second passed. This weakens what the replication can confirm.

  3. Some statements in the paper are not very clear and contain mistakes, see Questions For Authors for details.

Questions for authors
  1. In the paragraph of Connection to observers of Section 3.2, it is claimed that eq. 2-4 reduce to a Luenberger observer. Does it ignore the LayerNorm applied to the carried state in eq. 4. LayerNorm is nonlinear and fixes the mean and norm of the state. A linear observer’s state cannot be held at a fixed norm, so the structured path does not exactly contain the steady-state filter.

  2. In the paragraph of Connection to observers of Section 3.2, it is also claimed that a GRU has no such template. This statement is false at the level of function class. A GRU with constant gates and tanh in its linear range is a linear recurrence driven by (et,at−1)(e_t, a_{t-1}). It can implement the observer s^t=(I−KC)(As^t−1+Bat−1)+Kyt\hat{s}_t = (I - KC)(A \hat{s}_{t-1} + B a_{t-1}) + K y_t. Could you please check and explain this?

  3. To my understanding, in the integral regime, the current observed velocity is exactly the quantity that dead reckoning integrates. The theory does not explain why rereading it is harmful. Could you please explain this?

  4. Could you please explain how is "GRU with the same gated reread" defined? Eq. 5 takes LN(bt)LN(b_t), LN(Wpet)LN(W_p e_t) and ϵt\epsilon_t as inputs. A GRU has no innovation ϵt\epsilon_t. What replaces it here?

Reviewer_C9ywOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Main contribution of this paper is new RNN cell called DuoCell which is according to the authors a hetero- geneous recurrence for partially observed control.

My understanding is that this cell is a cell well-suited for problems where the input are state variables related to ODE, and where some of those variables are derivatives or integral from others.

They can with these new cells obtain better results on some benchmarks.

Critical strengths

I think that the new cell the authors propose can have some potential but for now the paper lacks scientific maturity. They need to come with a much better problem statement, identify the class of problems where their approach could make a difference.

My advice is for them to start working on a very small benchmark they could almost study analytically.

Critical weaknesses

Much too immature scientifically.

Questions for authors

Why haven't you chosen to better characterize the problems on which your approach could have some interest?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.