acceptodds
Under review as a conference paper at ICLR 2027

Functional Agreement of Computationally Equivalent States

Abstract

Learned stateful sequential models can implement the same semantic computation through different histories but arrive at different internal states. The behavior of these states may diverge during subsequent model rollouts. We introduce Functional Agreement of Computationally Equivalent States (FACES), an auxiliary loss that compares their future behavior under the same continuation. This objective encourages future behavioral agreement without forcing the internal states themselves to be equal. We introduce several FACES variants that target agreement at different future depths. Geometrically, we show that the objective induces a local quadratic form that weights state-perturbation directions according to their effects on future behavioral readouts. We first perform a mechanistic study on simple affine programs and show that learned actions can support much more accurate long-horizon execution under exact affine dynamics. We also show that route-induced state differences can have strongly direction-dependent effects on future behavior. We then show that FACES reduces long-horizon prediction error relative to ordinary supervised training and immediate agreement on controlled visual transformation sequences. Using real-world data, we show that multiscale AFACES achieves the lowest mean test error among the evaluated methods across temporal factorizations of the EuRoC IMU trajectories. We also show that continuation agreement improves over immediate agreement on a discrete, noninvertible problem. These results show that continuation-based behavioral regularization can improve both the interchangeability of states representing the same semantic computation and task performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sequential modeling, consistency regularization, compositional generalization, functional equivalence, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pE17OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FACES, an auxiliary training objective that encourages recurrent models to produce consistent future behavior from internal states reached through semantically equivalent computation histories. Unlike immediate agreement, FACES compares task relevant outputs after a shared continuation, exposing state differences over longer rollouts. The paper develops exact-depth, cumulative, anchored, and multiscale variants and derives a local quadratic characterization of their sensitivity to state perturbations. Experiments on affine programs, visual SE(2) transformations, EuRoC inertial trajectories, and a discrete memory-writing task show improvements in long-horizon accuracy and route consistency over several baselines.

Critical strengths
  1. A well motivated objective with a useful theoretical interpretation.
  2. Consistent experimental comparisons with meaningful controls.
Critical weaknesses
  1. Limited evidence of general applicability. Most experiments use synthetic or highly controlled settings. Although EuRoC provides a real world evaluation, it remains unclear whether FACES benefits more general recurrent architectures, larger scale sequential models, or tasks where equivalent computation histories are harder to specify.
  2. Novelty relative to existing consistency objectives needs sharper validation. The paper distinguishes FACES from path consistency, semigroup consistency, and predictive state approaches, but the empirical comparisons do not fully establish its advantage over a broader range of closely related methods.
Questions for authors
  1. How sensitive are the gains to the choice and sampling distribution of shared continuations? Can the method fail when sampled futures do not expose the relevant state discrepancies?
  2. Can the authors demonstrate FACES on a different sequential modeling task or architecture, with comparisons against a closely related consistency objective?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pE17 (1/5)

Dear Reviewer pE17,

Thank you for carefully reading and for your constructive review of our work.

Because there are multiple questions and OpenReview comments have a length limit, we split our response into five comments.

We provide our answers to your questions below:

1. How sensitive are the gains to the choice and sampling distribution of shared continuations? Can the method fail when sampled futures do not expose the relevant state discrepancies?

FACES is indeed sensitive to the choice of shared continuations. Yes, it can fail when sampled futures do not expose relevant route defects. Our current experiments already show both sensitivity to continuation choice and failure cases with unfavorable continuations. In fact, the local expected quadratic discrepancy developed in our theory can characterize when training continuations control discrepancies under another future distribution.

A. Sensitivity to continuation choice

In EuRoC, AFACES-≤K\le K uses a cumulative set of horizons from 00 to KK. Recall the paper's local expected quadratic discrepancy:

Jjh(δ)=δ⊤Γj(h)δ+O(∥δ∥3).\mathcal J _j^h(\delta) = \delta^\top\Gamma _j(h)\delta + O(\|\delta\|^3).

More generally, if the cumulative objective weights the included horizons by positive weights αj\alpha_j,

J≤Kh(δ)=∑j=0KαjJjh(δ)=δ⊤Γ≤K(h)δ+O(∥δ∥3),\mathcal J _{\le K}^h(\delta) = \sum _{j=0}^{K}\alpha _j\mathcal J _j^h(\delta) = \delta^\top\Gamma _{\le K}(h)\delta + O(\|\delta\|^3),

where

Γ≤K(h)=∑j=0KαjΓj(h).\Gamma _{\le K}(h) = \sum _{j=0}^{K}\alpha _j\Gamma _j(h).

For positive weights,

ker⁡Γ≤K=⋂j:αj>0ker⁡Γj.\ker\Gamma _{\le K} = \bigcap _{j:\alpha _j>0}\ker\Gamma _j.

Thus cumulative FACES can expose any route-defect direction exposed by at least one included horizon, while changing KK changes the set of directions covered and their relative weighting.

Empirically, this dependence is substantial. On EuRoC validation, the best cumulative horizon changes from K=1K=1 at c=0,.25c=0,.25 to K=4K=4 at c=.50,.75,1c=.50,.75,1. For example, AFACES-≤1\le1 outperforms AFACES-≤4\le4 at c=0c=0 (0.08309 vs. 0.11500), while AFACES-≤4\le4 performs better over the coarse regimes c=.75,1c=.75,1 (0.05352 vs. 0.07044 averaged over the two). This sensitivity motivated MS-AFACES, which combines multiple continuation scales.

Our task-qualification studies (details in Appendix G) also contain direct failure cases. In RNAR Heapsort and 3D-R2N2, more than 90% of the initial route discrepancy is removed by the common future. These futures are poor probes because ordinary future computation itself erases most route dependence, leaving little discrepancy for FACES to correct.

B. Failure when sampled futures do not expose the discrepancy

Suppose training and evaluation future laws induce Γtr(h)\Gamma_{\rm tr}(h) and Γev(h)\Gamma_{\rm ev}(h), possibly with different horizons jtrj_{\rm tr} and jevj_{\rm ev}. Let VV be an orthonormal basis of the attainable route-defect subspace R\mathcal R, and define

Gtr=V⊤Γtr(h)V,Gev=V⊤Γev(h)V.G _{\rm tr} = V^\top\Gamma _{\rm tr}(h)V, \qquad G _{\rm ev} = V^\top\Gamma _{\rm ev}(h)V.

A finite constant κ\kappa satisfying

Gev⪯κGtrG _{\rm ev} \preceq \kappa G _{\rm tr}

exists exactly when

ker⁡Gtr⊆ker⁡Gev.\ker G _{\rm tr} \subseteq \ker G _{\rm ev}.

The smallest such constant is

κ⋆=λmax⁡(Gtr†/2GevGtr†/2),\kappa _\star = \lambda _{\max} \left( G _{\rm tr}^{\dagger/2} G _{\rm ev} G _{\rm tr}^{\dagger/2} \right),

where Gtr†/2G_{\rm tr}^{\dagger/2} is the positive square root of the Moore-Penrose pseudoinverse and λmax⁡\lambda_{\max} denotes the largest eigenvalue. This gives, for sufficiently small route defects,

Jjevh(δ)≤κ⋆Jjtrh(δ)+O(∥δ∥3).\mathcal J _{j _{\rm ev}}^h(\delta) \le \kappa _\star \mathcal J _{j _{\rm tr}}^h(\delta) + O(\|\delta\|^3).

We provide the full proof in a separate comment.

The kernel condition characterizes whether coverage exists at all: every attainable route-defect direction that matters under evaluation futures must also be exposed by training futures. However, finite coverage does not imply strong control. κ⋆\kappa_\star depends on the relative geometry of GtrG_{\rm tr} and GevG_{\rm ev}; if evaluation strongly amplifies a direction that training exposes only weakly, κ⋆\kappa_\star can be large and the bound correspondingly weak. Thus continuation choice affects both which relevant directions are exposed and how strongly they are constrained.

Importantly, this result is not restricted to different horizons. Even when jtr=jevj_{\rm tr}=j_{\rm ev}, changing the sampling distribution over continuations changes GtrG_{\rm tr} and GevG_{\rm ev}, and the same coverage condition applies.

Therefore, a shorter jtrj_{\rm tr} can control a longer jevj_{\rm ev} if the shorter futures already expose every evaluation-relevant direction, which may help explain the extreme extrapolation in our new StateBench-inspired experiment below. Conversely, if a direction is invisible during training but becomes visible under evaluation futures, the kernel inclusion fails and no finite κ⋆\kappa_\star exists. In that case, the training FACES discrepancy cannot control the evaluation discrepancy.

(1/5)

AuthorsOpenReview ↗

Response to Reviewer pE17 (2/5)

Full Proof: Upperbounding Evaluation Route Discrepancy Using Training Route Discrepancy

This continues from the notation in the first rebuttal comment. Let VV be an orthonormal basis of the attainable route-defect subspace R\mathcal R, write δ=Vz\delta=Vz, and define

Gtr=V⊤Γtr(h)V,Gev=V⊤Γev(h)V.G _{\rm tr} = V^\top\Gamma _{\rm tr}(h)V, \qquad G _{\rm ev} = V^\top\Gamma _{\rm ev}(h)V.

Because Γtr\Gamma_{\rm tr} and Γev\Gamma_{\rm ev} are expectations of matrices of the form Ju⊤JuJ_u^\top J_u, both GtrG_{\rm tr} and GevG_{\rm ev} are positive semidefinite.

We first ask whether the training discrepancy can control the evaluation discrepancy. This requires the existence of a finite constant κ\kappa such that

Gev⪯κGtr,G _{\rm ev} \preceq \kappa G _{\rm tr},

or equivalently,

z⊤Gevz≤κz⊤Gtrzz^\top G _{\rm ev}z \le \kappa z^\top G _{\rm tr}z

for every possible route-defect direction zz. Here, ⪯\preceq denotes the positive-semidefinite matrix order.

Proposition: A finite such κ\kappa exists exactly when

ker⁡Gtr⊆ker⁡Gev.\ker G _{\rm tr} \subseteq \ker G _{\rm ev}.

Proof of necessity: Suppose there exists a finite κ\kappa satisfying

z⊤Gevz≤κz⊤Gtrz.z^\top G _{\rm ev}z \le \kappa z^\top G _{\rm tr}z.

Take any v∈ker⁡Gtrv\in\ker G_{\rm tr}. Since both matrices are positive semidefinite,

0≤v⊤Gevv≤κv⊤Gtrv=0.0 \le v^\top G _{\rm ev}v \le \kappa v^\top G _{\rm tr}v = 0.

Therefore,

v⊤Gevv=0.v^\top G _{\rm ev}v = 0.

For a positive-semidefinite matrix, this implies Gevv=0G_{\rm ev}v=0. Hence,

v∈ker⁡Gev,v\in\ker G _{\rm ev},

and therefore

ker⁡Gtr⊆ker⁡Gev.\ker G _{\rm tr} \subseteq \ker G _{\rm ev}.

Proof of sufficiency and the smallest admissible constant: Now suppose

ker⁡Gtr⊆ker⁡Gev.\ker G _{\rm tr} \subseteq \ker G _{\rm ev}.

Let

S=range⁡(Gtr)=(ker⁡Gtr)⊥.\mathcal S = \operatorname{range}(G _{\rm tr}) = (\ker G _{\rm tr})^\perp.

Here, (⋅)⊥(\cdot)^\perp denotes the orthogonal complement. On S\mathcal S, GtrG_{\rm tr} is positive definite. Any component in ker⁡Gtr\ker G_{\rm tr} also belongs to ker⁡Gev\ker G_{\rm ev}, so it contributes to neither quadratic form. Therefore, it is sufficient to consider nonzero z∈Sz\in\mathcal S.

The smallest admissible constant is then

κ⋆=sup⁡z∈S,  z≠0z⊤Gevzz⊤Gtrz.\kappa _\star = \sup _{z\in\mathcal S,\;z\ne0} \frac{ z^\top G _{\rm ev}z }{ z^\top G _{\rm tr}z }.

Let

w=Gtr1/2z.w = G _{\rm tr}^{1/2}z.

On the supported range,

z=Gtr†/2w.z = G _{\rm tr}^{\dagger/2}w.

Here, Gtr†/2G_{\rm tr}^{\dagger/2} denotes the positive square root of the Moore-Penrose pseudoinverse.

Substituting gives

κ⋆=sup⁡w≠0w⊤Gtr†/2GevGtr†/2ww⊤w.\kappa _\star = \sup _{w\ne0} \frac{ w^\top G _{\rm tr}^{\dagger/2} G _{\rm ev} G _{\rm tr}^{\dagger/2} w }{ w^\top w }.

This is the Rayleigh quotient of

Gtr†/2GevGtr†/2.G _{\rm tr}^{\dagger/2} G _{\rm ev} G _{\rm tr}^{\dagger/2}.

Therefore,

κ⋆=λmax⁡(Gtr†/2GevGtr†/2).\kappa _\star = \lambda _{\max} \left( G _{\rm tr}^{\dagger/2} G _{\rm ev} G _{\rm tr}^{\dagger/2} \right).

Here, λmax⁡\lambda_{\max} denotes the largest eigenvalue.

We now return to the expected quadratic discrepancies. From the paper,

Jjtrh(δ)=δ⊤Γtr(h)δ+O(∥δ∥3),\mathcal J _{j _{\rm tr}}^h(\delta) = \delta^\top\Gamma _{\rm tr}(h)\delta + O(\|\delta\|^3),

and

Jjevh(δ)=δ⊤Γev(h)δ+O(∥δ∥3).\mathcal J _{j _{\rm ev}}^h(\delta) = \delta^\top\Gamma _{\rm ev}(h)\delta + O(\|\delta\|^3).

Since δ=Vz\delta=Vz,

δ⊤Γev(h)δ=z⊤Gevz≤κ⋆z⊤Gtrz=κ⋆δ⊤Γtr(h)δ.\delta^\top\Gamma _{\rm ev}(h)\delta = z^\top G _{\rm ev}z \le \kappa _\star z^\top G _{\rm tr}z = \kappa _\star \delta^\top\Gamma _{\rm tr}(h)\delta.

Substituting the two local expansions gives

Jjevh(δ)≤κ⋆Jjtrh(δ)+O(∥δ∥3).\mathcal J _{j _{\rm ev}}^h(\delta) \le \kappa _\star \mathcal J _{j _{\rm tr}}^h(\delta) + O(\|\delta\|^3).

Thus, training-future discrepancy locally controls evaluation-future discrepancy exactly when every route-defect direction that matters under the evaluation future law is also exposed by the training future law. For affine behavioral maps, the quadratic expansions are exact, so the remainder term disappears.

(2/5)

AuthorsOpenReview ↗

Response to Reviewer pE17 (3/5) - Edited to clarify additional training details

2. Can the authors demonstrate FACES on a different sequential modeling task or architecture, with comparisons against a closely related consistency objective?

To address this concern, we performed a StateBench-inspired (reference [1], anonymous) permutation state-tracking experiment. This is not the exact StateBench benchmark. However, we took the idea of permutation/state tracking under extremely long extrapolation horizons and constructed a sequential computation task around it. This experiment uses a different recurrent architecture from the GRU-based experiments in our paper. We also added related consistency objectives, namely Semigroup consistency and Latent Path Consistency, as controls.

We split this response into two comments. The experimental protocol and results are now included in Appendix E of the revised manuscript.

A. StateBench-inspired permutation state-tracking task

The task consists of 12 registers containing distinct values from 0 to 11. The initial state is a random permutation. At each step, two registers are swapped by a transposition τab\tau_{ab}. After a sequence of swaps, the model must output the current value of every register. All 12 registers must be correct for the prediction to count as accurate. Training uses relatively short swap sequences, while evaluation extends the same recurrent computation to extremely long horizons.

The recurrent model maps each register value into an 8-dimensional latent vector. When registers aa and bb are swapped, their latent states are updated using a learned symmetric transition:

za′=tanh⁡(Aza+Bzb+β),zb′=tanh⁡(Azb+Bza+β).z'_a = \tanh(Az_a+Bz_b+\beta), \qquad z'_b = \tanh(Az_b+Bz_a+\beta).

Here, AA and BB are learned 8×88\times8 matrices and β∈R8\beta\in\mathbb{R}^8 is a learned bias vector. A shared linear decoder predicts the value in each register. This architecture provides a structural bias while being genuinely different from a GRU. Small transition errors can compound over long recurrent horizons, making the task suitable for studying whether FACES improves long-horizon performance.

To construct exact equivalences, we use the Coxeter braid relation. For three distinct registers a,b,ca,b,c,

τabτbcτab=τbcτabτbc.\tau_{ab}\tau_{bc}\tau_{ab} = \tau_{bc}\tau_{ab}\tau_{bc}.

Each paired route executes 6 shared history swaps, then branches through the two equivalent three-swap braid sequences, after which a shared continuation is applied. This braid is the shortest nontrivial exact equivalence involving interacting swaps: the routes differ and share registers, but implement exactly the same permutation. Although interchangeable under any shared future, a learned recurrent model can map them to different hidden states.

B. FACES and consistency baselines

We compare Ordinary, FACES-0, FACES-Future, and related consistency controls. We also tested a dense four-probe FACES objective under a shorter training schedule.

All methods use final-step cross-entropy supervision; Ordinary uses only this objective. FACES-0 encourages functional agreement immediately after the two equivalent braid routes join.

We also tested four dense FACES probes at the join and the next three continuation depths, 0,1,2,30,1,2,3, with a four-swap training continuation. With 6 history, 3 braid, and 4 continuation swaps, the training rollout is 13 steps. Under this short-horizon protocol, all methods failed at long horizons: at H=4096H=4096, Ordinary, Semigroup, both Latent-PC variants, FACES-0, and the dense four-probe FACES variant all obtained about 39-40% exact accuracy. Thus, this 13-step, four-probe rollout is insufficient for extreme extrapolation.

We therefore extended the shared training continuation to 12 swaps. To keep four probes while covering the longer continuation, FACES-Future (4 probes) evaluates functional agreement at depths 0,4,8,120,4,8,12. Thus, the probes span the full continuation rather than concentrating near the route join. The training rollout is then 21 steps, with cross-entropy supervision at the final step.

We also include a Semigroup (involution) baseline enforcing

τab2=id⁡\tau_{ab}^2=\operatorname{id}

by encouraging the latent state to return to its original representation after applying the same swap twice. This local equivalence differs from the braid equivalence used by the other methods.

Finally, Latent-PC-0 encourages the latent states produced by the two braid routes to agree at the join, while Latent-PC-Future (4 probes) enforces latent agreement at the same depths 0,4,8,120,4,8,12 used by FACES-Future. Thus, Latent-PC-Future and FACES-Future use the same equivalent routes and probe locations; they differ only in whether agreement is enforced in hidden-state or functional prediction space.

References:

[1] Anonymous. A fine-grained hierarchy and benchmark for state tracking in language models. In Submitted to The Fifteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=1oon6lKQdR. under review.

(3/5)

Reviewer_Wc9qOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Suppose we have two state space models that are trained to predict an output sequence given an input sequence. This paper introduces an auxiliary loss called FACES. FACES compares two predicted outputs given externally-defined "similar" states propagated through time with the same input sequence. Since the initial states are similar in some sense, the loss encourages the similar states to evolve to states with similar outputs under identical inputs. Several FACES variants are introduced that weight horizons differently or treat the two states asymmetrically. The proposed loss is tested on both synthetic and aerial vehicle trajectory data.

Critical strengths

An auxiliary loss that targets forecasts from similar states is a reasonable idea to better learn the multistep and longer-horizon statistics of input/output sequences.

Critical weaknesses

There are major clarity issues that prevent me from providing more helpful feedback. I am unsure why many choices are done because the text decides the choices, but not their motivations. Additionally, I am unsure what the proposed auxiliary loss is supposed to do conceptually, beyond its definition. See question 1. The experiments are similarly opaque to me, even after a careful reading. Literal results of insufficiently described experiments are presented without interpretations or their larger significance. The introduction section is similar: none of the notation in equation 1 is defined in this section; the prior work paragraph reads like a list of unconnected ideas; the summary of FACES and MS-AFACES results is unintelligible to someone who hasn't yet read the paper.

Questions for authors
  1. Is there a motivating practical example or perhaps a theoretical reason why FACES is proposed? A single good example could provide the necessary motivation and conceptual anchor for the reader.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Wc9q (1/3)

Dear Reviewer Wc9q,

We answer your question first, then clarify several material mischaracterizations in the review.

Reviewer's Question: Is there a motivating practical example or perhaps a theoretical reason why FACES is proposed? A single good example could provide the necessary motivation and conceptual anchor for the reader.

The paper motivates FACES in the abstract and opening of the Introduction, and provides both a practical and a theoretical example. The first paragraph of the Introduction states:

“A stateful sequential model can execute two semantically equivalent histories correctly and still fail to treat their resulting states as interchangeable. The difference may be invisible in the current output, then emerge after computation continues. This exposes a gap in ordinary task supervision: it trains predictions along observed histories, but does not require histories that represent the same computation to support the same later behavior.”

The same paragraph gives concrete examples: transformations can be decomposed or composed, and the same dynamical evolution can be factorized at different temporal scales.

Immediately after Eq. 1, the paper states the conceptual purpose of FACES:

“Rather than forcing internal states to be equal, we ask whether one can replace the other without changing future task behavior. The continuation acts as a probe to expose route-dependent differences that are invisible at the immediate task readout but consequential after later transitions.”

The abstract gives the same motivation:

“Learned stateful sequential models can implement the same semantic computation through different histories but arrive at different internal states. The behavior of these states may diverge during subsequent model rollouts. We introduce Functional Agreement of Computationally Equivalent States (FACES), an auxiliary loss that compares their future behavior under the same continuation. This objective encourages future behavioral agreement without forcing the internal states themselves to be equal.”

A concrete practical example is our EuRoC experiment, described at the beginning of Section 4.3 on page 7. Many physical motions can be processed through different temporal factorizations. In the EuRoC case, adjacent IMU increments can be processed separately or replaced by their exact composition. Such refactorizations arise naturally in robotics when sampling rates, integration windows, batching, or keyframe selection change the number and scale of recurrent updates. These histories represent the same nominal physical evolution, yet a learned recurrent estimator need not produce states that remain behaviorally interchangeable under subsequent motion. FACES targets this failure.

There is also a direct theoretical example near the beginning of Section 3 on page 4 illustrating why immediate agreement can miss future differences. Consider

Q(x,z)=x,F(x,z)=(z,x).Q(x,z)=x,\qquad F(x,z)=(z,x).

The states (0,0)(0,0) and (0,ϵ)(0,\epsilon) have identical immediate readouts, but after one shared transition their readouts differ by ϵ\epsilon. Conversely, a state difference can be visible immediately and disappear after a future transition. Thus, readout agreement at one computational depth does not, in general, imply readout agreement at another.

We then test whether correcting this specific failure mode improves task performance, rather than treating consistency as an end in itself. Across the paper, we examine whether continuation-based agreement improves long-horizon execution and whether it improves performance relative to immediate behavioral agreement, additional task supervision, and alternative structural regularization.

(1/3)

AuthorsOpenReview ↗

Response to Reviewer Wc9q (2/3)

Second, we would like to clarify several material mischaracterizations of our paper. These concern the basic setup and conceptual purpose of the method, and therefore affect the subsequent assessment.

A. The Paper Summary section:

  • “Suppose we have two state space models that are trained to predict an output sequence given an input sequence.”

The paper uses a single stateful model: Section 2 defines one learned transition FθF_\theta and one task readout QωQ_\omega. What we compare are two semantically equivalent computational histories executed by that same model, not two models. The opening sentence of the Introduction states:

“A stateful sequential model can execute two semantically equivalent histories correctly and still fail to treat their resulting states as interchangeable.”

The setup is one model and two semantically equivalent histories.

  • “FACES compares two predicted outputs given externally-defined ‘similar’ states propagated through time with the same input sequence.”

We do not define externally specified “similar” states. The externally specified object is the equivalence between computation histories, p≃qp\simeq q, not a similarity relation between hidden states. Section 2 states:

“More generally, p≃qp\simeq q denotes an externally specified equivalence between histories implementing the same semantic computation.”

The states reached through those histories are explicitly allowed to differ:

“The states Hθ(p;h)H_\theta(p;h) and Hθ(q;h)H_\theta(q;h) may retain route-dependent information not reflected in the current task readout. We require functional interchangeability under shared futures, allowing their hidden states to differ.”

Describing the construction as merely propagating two states “with the same input sequence” also omits the role of the shared continuation. It is used as a behavioral probe to test whether states reached by semantically equivalent histories remain functionally interchangeable under future computation. This is stated immediately after Eq. 1:

“The continuation acts as a probe to expose route-dependent differences that are invisible at the immediate task readout but consequential after later transitions.”

  • “Since the initial states are similar in some sense, the loss encourages the similar states to evolve to states with similar outputs under identical inputs.”

The compared states are not initial states and are not assumed to be similar. The two histories begin from the same initial state hh. The states compared by FACES are the states reached after executing the equivalent histories, Hθ(p;h)H_\theta(p;h) and Hθ(q;h)H_\theta(q;h).

FACES also does not require these hidden states, or their subsequent hidden-state trajectories, to become similar. It constrains their task-relevant behavior under the same future continuation.

The construction is: one model executes two semantically equivalent histories from the same starting state, may arrive at different internal states, and FACES trains those states to remain functionally interchangeable under future computation.

  • “Several FACES variants are introduced that weight horizons differently or treat the two states asymmetrically.”

The first part is correct. Several FACES variants differ in how agreement is weighted across continuation depths. In AFACES, the asymmetry is specifically between a reference route and a student route: stop-gradient is applied to the reference route output, while the auxiliary gradient is propagated through the student route.

B. The Critical Strength section:

  • “An auxiliary loss that targets forecasts from similar states is a reasonable idea to better learn the multistep and longer-horizon statistics of input/output sequences.”

This is not the conceptual objective of FACES. FACES does not start from “similar states” and its purpose is not generic learning of multistep or longer-horizon input/output statistics. It uses known semantic equivalences between computation histories and penalizes route-dependent differences when those differences alter task-relevant behavior under a shared future.

FACES targets future functional interchangeability of states reached by equivalent computations, not hidden-state similarity or generic forecast consistency.

(2/3)

AuthorsOpenReview ↗

Response to Reviewer Wc9q (3/3)

C. The Critical Weaknesses section

The Critical Weaknesses section raises several clarity concerns. We address each point below.

  • “I am unsure why many choices are done because the text decides the choices, but not their motivations.”

The main methodological and experimental choices are motivated where they are introduced. Section 4 explains why all methods continue from a competent common checkpoint: in an untrained model, trivial agreement may occur without solving the task, so the common checkpoint lets us test whether FACES improves an already competent model while controlling the starting point and additional task training. Section 4.2 introduces visual SE(2) specifically to increase task complexity while retaining exact equivalent routes and ground-truth future targets, allowing controlled comparisons against immediate agreement and additional task supervision. Section 4.3 moves to EuRoC to test the method on real-world inertial data and explains why temporal refactorization is practically relevant in robotics. The observed dependence of useful continuation depth on temporal factorization then motivates MS-AFACES. Section 4.4 explains why SparseWriter is included and reports the result against FACES-0, while the Introduction and Conclusion state what this shows about discrete, noninvertible computation.

  • “I am unsure what the proposed auxiliary loss is supposed to do conceptually, beyond its definition.”

This is addressed in our response to the reviewer’s question in (1/3). The conceptual purpose is also stated immediately after Eq. 1: the continuation is a behavioral probe for route-dependent differences that may be invisible at the current readout but consequential after later computation.

  • “Literal results of insufficiently described experiments are presented without interpretations or their larger significance.”

The experimental sections include both motivation and interpretation. In Section 4.1 (Affine), the paper first diagnoses the source of length-96 error, then studies the behavioral sensitivity of route perturbations, and then tests whether reducing their future effect improves long-horizon execution. Section 4.2 (Visual SE(2)) contains separate paragraphs titled “Continuation benefit”, “Controls for supervision depth”, and “Control for action composition”, each interpreting what the corresponding comparison establishes. Section 4.3 (EuRoC) explains the practical significance of temporal refactorization, shows that the preferred continuation depth changes with factorization, and states: “This tradeoff motivates MS-AFACES.” Section 4.4 explains why SparseWriter is included and reports the result against FACES-0, while the Introduction and Conclusion state the broader takeaway for discrete, noninvertible computation. Each main experiment in Sections 4.1–4.3 includes the task description, construction of the equivalent routes used by FACES, the model used, and the reported evaluation metrics. Each subsection in Section 4 also points to its corresponding appendix. The Reproducibility Statement specifies that Appendices B–E provide the task constructions, model architectures, training procedures, hyperparameters, data processing, and evaluation protocols needed to reproduce the reported results.

  • “none of the notation in equation 1 is defined in this section”

Equation 1 is the high-level signature of the method in the Introduction. The sentence immediately preceding it introduces the equivalent histories p≃qp\simeq q, common starting state hh, and shared continuation uu, and Eq. 1 defines BuB_u. The formal definitions of HθH_\theta and QωQ_\omega then appear at the beginning of Section 2 on the following page.

  • “the prior work paragraph reads like a list of unconnected ideas”

The paragraph follows a specific progression: path sensitivity in learned trajectories, prediction invariance across valid computation paths, consistency between direct and composed transitions, and then predictive-state views based on future observable behavior. It concludes by stating the connection to our method:

“FACES builds on this view by turning known equivalences between histories into an objective for future behavioral interchangeability.”

  • “the summary of FACES and MS-AFACES results is unintelligible to someone who hasn't yet read the paper.”

The Introduction gives the experimental progression directly in the paragraph immediately preceding the contribution list: affine programs are used to study the mechanism; visual SE(2) tests whether continuation agreement improves long-horizon execution beyond immediate agreement and additional task supervision; and EuRoC shows that the useful continuation horizon changes across temporal factorizations. The same passage then states that this observation “motivates MS-AFACES, which improves mean performance across these factorizations.”

The same experimental progression is also summarized in the latter half of the abstract.

(3/3)

AuthorsOpenReview ↗

Additional note following rebuttal revisions

We have uploaded a revised version of the manuscript with additional theoretical and experimental results.

We have intentionally kept our responses above unchanged, as they address the review of the original submission. The quotations, section references, and appendix references therefore refer to the original manuscript, preserving the context in which the review and our responses were written.

Reviewer_JrVTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper investigates whether training transitions along different but equivalent paths to produce the same decoded outputs improves prediction over longer sequences. FACES-0 encourages agreement immediately after the equivalent paths, while FACES-k encourages agreement after k additional shared steps. The synthetic experiments include affine transformations, 2D geometric transformations, and SparseWriter, where parts of a bit sequence are overwritten at each step. The real-data experiments use increments computed from Inertial Measurement Unit (IMU) over short time intervals to estimate position, velocity, and orientation.

Critical strengths
  • The proposed method itself is sound.
  • The paper evaluates the method on various datasets and tasks, including real data for drone state estimation, and demonstrates its effectiveness.
Critical weaknesses
  • I do not see any particularly critical weaknesses. Please refer to the minor questions below.
Questions for authors
  • In Table 1, why do the single/dual-route exact-target controls perform worse than matching the predictions of the two routes? Intuitively, aligning the routes using ground-truth targets seems preferable. Could this be because it causes overfitting to the particular sequences used during training?
  • How does the symmetric version, which allows gradients to flow through both routes, perform on the EuRoC data (without anchoring)? Conversely, how does the anchored version perform on the synthetic data?
  • How would a baseline trained with standard ground-truth supervision on sequences matching the total length used to compute the proposed loss, including the additional future steps, compare with the proposed method?
5 comments
AuthorsOpenReview ↗

Response to Reviewer JrVT (1/3)

Dear Reviewer JrVT,

Thank you for carefully reading and for your constructive review of our work.

We did some more experiments to answer your questions. Details for each are below. The response is split into three comments, one comment for each question, because of the OpenReview limit:

1. In Table 1, why do the single/dual-route exact-target controls perform worse than matching the predictions of the two routes? Intuitively, aligning the routes using ground-truth targets seems preferable. Could this be because it causes overfitting to the particular sequences used during training?

We directly tested the overfitting hypothesis by evaluating the models in Table 1 (Visual SE(2) experiment) on regenerated sequences that were encountered during training and on 4096 fresh sequences per seed independently sampled from the same distribution. The fresh vs. replay error gaps are extremely small and broadly comparable across all methods, averaged over five seeds:

MethodFresh - replay RMSE
Ordinary continued training1.11×10−41.11\times10^{-4}
FACES-07.69×10−57.69\times10^{-5}
FACES-45.85×10−55.85\times10^{-5}
Single-route exact target9.43×10−59.43\times10^{-5}
Dual-route exact target9.61×10−59.61\times10^{-5}
Additive-action control6.70×10−56.70\times10^{-5}

Additionally, for fresh sequences of length 6, FACES-4 has terminal MSE 6.31×10−56.31\times10^{-5}, while the single- and dual-route exact-target controls have lower errors of 5.78×10−55.78\times10^{-5} and 5.39×10−55.39\times10^{-5}. This ordering also holds throughout lengths 2-6. Thus, the exact-target controls actually achieve slightly lower error than FACES-4 on fresh unseen sequences at these lengths. Overfitting to the sampled training sequences therefore does not explain their weaker Table 1 performance.

At length 24, reported in the paper, this ordering reverses strongly: FACES-4 has substantially lower error than either exact-target control.

There is also a simple objective-level explanation for this difference. Let y^P\hat y_P and y^Q\hat y_Q be the predictions from the two equivalent routes, and let y⋆y^\star be their common exact target. Their prediction errors are

eP=y^P−y⋆,eQ=y^Q−y⋆.e_P=\hat y_P-y^\star,\qquad e_Q=\hat y_Q-y^\star.

We can decompose these errors as

eP=m+d,eQ=m−d,e_P=m+d,\qquad e_Q=m-d,

where

m=eP+eQ2=y^P+y^Q2−y⋆m=\frac{e_P+e_Q}{2}=\frac{\hat y_P+\hat y_Q}{2}-y^\star

is the prediction error shared by the two routes, while

d=eP−eQ2=y^P−y^Q2d=\frac{e_P-e_Q}{2}=\frac{\hat y_P-\hat y_Q}{2}

is the route-dependent disagreement.

The mean squared dual-route exact-target loss then decomposes as

12(∣∣eP∣∣2+∣∣eQ∣∣2)=∣∣m∣∣2+∣∣d∣∣2.\frac{1}{2}\left(||e_P||^2+||e_Q||^2\right)=||m||^2+||d||^2.

In contrast, directly matching the two route predictions penalizes

∣∣y^P−y^Q∣∣2=∣∣eP−eQ∣∣2=4∣∣d∣∣2.||\hat y_P-\hat y_Q||^2=||e_P-e_Q||^2=4||d||^2.

The important comparison is that both losses are used as auxiliaries on top of the ordinary task objective. Ordinary supervision already penalizes task prediction error. Adding the dual-route exact-target loss therefore allocates its additional supervision to both the shared task-error component mm and the route-dependent component dd. In contrast, FACES allocates its auxiliary supervision entirely to route-dependent disagreement.

Thus, relative to ordinary task training, FACES places proportionally greater additional pressure on the component associated with taking different but semantically equivalent computational routes. The dual-route exact-target objective instead spends part of its auxiliary supervision on prediction error already targeted by the task objective. The single-route exact-target control is even less targeted toward route disagreement because it directly supervises only one route.

Consistent with this explanation, we also diagnosed the median auxiliary gradient-energy fraction assigned to each model component across multiple randomly sampled batches:

ObjectiveControllerReceiver encoderRecurrent transitionReadout
FACES-446.2%13.8%27.1%11.5%
Single-route exact target13.2%11.5%20.3%53.5%
Dual-route exact target12.6%11.4%20.3%54.3%

The two exact-target controls allocate most of their auxiliary gradient energy toward the readout. FACES-4 has the opposite behavior. It allocates much more toward the controller and recurrent transition, while the readout receives the least.

This supports the objective-level explanation above. The exact-target controls emphasize directly improving task predictions. FACES places more pressure on the learned computation controlling how route-dependent errors propagate under future transitions.

We therefore believe FACES improves the learned computation in a way that matters under longer continuation. Its advantage is not explained by greater overfitting of the exact-target controls to the sampled training sequences.

(1/3)

AuthorsOpenReview ↗

Response to Reviewer JrVT (2/3)

2. How does the symmetric version, which allows gradients to flow through both routes, perform on the EuRoC data (without anchoring)? Conversely, how does the anchored version perform on the synthetic data?

We did perform an initial experiment using symmetric FACES on EuRoC with seed 42. Symmetric FACES-4 without anchoring decreased disagreement between the two routes: position disagreement decreased from 0.0054 to 0.0039 and velocity disagreement decreased from 0.0135 to 0.01045. However, task error increased: position RMSE increased from 0.01523 to 0.01802 and velocity RMSE increased from 0.0716 to 0.08166.

This suggests that allowing gradients to flow through both routes can reduce their disagreement by also pulling the native finer route, which is the factorization directly trained by the ordinary supervised loss, toward the coarser route. We therefore detach the native route from the FACES loss and use it as the reference, preventing gradients from flowing through the native branch under the agreement objective. This was our motivation for formulating AFACES.

Following your comment, we additionally trained and evaluated anchored variants for the synthetic experiments. There are two anchored variants: “Direct-route anchored FACES,” where the route with the composite primitive is detached, and “Sequential-route anchored FACES,” where the route with two consecutive primitives is detached. We lifted the existing results from the paper and added the two new variants at the bottom of the tables, averaged over five seeds:

Affine, L96RMSE
Ordinary0.10626
FACES-00.07724
Symmetric continuation FACES0.05794
Direct-route anchored FACES0.05841
Sequential-route anchored FACES0.82215
Visual SE(2), L24RMSE
Ordinary continued training0.1424
FACES-00.0702
Symmetric FACES-40.0340
Single-route exact target0.0516
Dual-route exact target0.0506
Direct-route anchored FACES0.0296
Sequential-route anchored FACES0.4265

In both synthetic settings, the Sequential-route anchored variant performed much worse than the symmetric variant, while the Direct-route anchored variant performed similarly to symmetric FACES on Affine and slightly better on Visual SE(2). The symmetric variant therefore has the advantage that it does not require us to know beforehand which route should be used as the reference, while still performing well.

This result was somewhat unexpected because ordinary task supervision trains the model using the sequential native route. Intuitively, the sequential route should be the better reference. The synthetic results show that this is not necessarily the case. This made us revisit our EuRoC MS-AFACES experiment and train an additional MS-AFACES variant where the coarser direct route is detached instead of the finer native sequential route. The table reports the EuRoC error EE averaged across the five temporal factorization levels across five seeds.

SplitSequential anchorDirect anchorDirect − SequentialDirect wins
Dev0.07006 ± 0.020690.13292 ± 0.02119+0.06286 ± 0.010680/5
Test0.05330 ± 0.017390.14004 ± 0.01935+0.08674 ± 0.026410/5

In the EuRoC case, the result is the opposite: our current formulation of MS-AFACES in the paper, where the finer native sequential route is detached, performs substantially better. The direct-anchor variant loses on all five seeds on both development and test sets.

Taken together, these results suggest that which direction to anchor is task-dependent. On the two synthetic tasks, using the direct route as the reference works much better than using the sequential route, while symmetric FACES performs well without requiring us to choose a reference direction. On EuRoC, however, the finer native sequential route is empirically the better reference, supporting the AFACES formulation used in the paper.

(2/3)

AuthorsOpenReview ↗

Response to Reviewer JrVT (3/3)

3. How would a baseline trained with standard ground-truth supervision on sequences matching the total length used to compute the proposed loss, including the additional future steps, compare with the proposed method?

A baseline trained with standard ground-truth supervision on sequences matching the total depth used to compute the FACES-4 loss corresponds to ordinary training on sequences extended by four additional steps. We performed this requested control on the Visual SE(2) task over five seeds. The Ordinary and FACES-4 values are the corresponding results reported in the paper.

MethodL24 RMSETransition-only L24 RMSE
Ordinary continued training0.1424 ± 0.01810.2064 ± 0.0293
Ordinary, N+4 supervision0.0585 ± 0.00730.1541 ± 0.0199
FACES-40.0340 ± 0.00210.1095 ± 0.0132

Ordinary N+4 supervision substantially improves performance relative to ordinary continued training. However, it still underperforms FACES-4 at the L=24 horizon, despite receiving the stronger privileged signal of exact ground-truth supervision for four additional timesteps.

Comparing Ordinary N+4 supervision and FACES-4 across evaluation horizons gives:

LengthOrdinary N+4FACES-4
60.007560.00771
80.008530.00970
120.011680.01472
180.026700.02340
240.058550.03396

Ordinary N+4 performs slightly better than FACES-4 at the shorter horizons, L=6-12, but FACES-4 becomes better at L=18 and substantially better at L=24. Thus, additional ground-truth supervision over the longer training sequences is clearly helpful, but it does not account for the longer-horizon benefit of continuation functional agreement.

(3/3)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all reviewers for their feedback. We have uploaded a revised manuscript addressing the questions and concerns raised during the review process. The substantive additions are highlighted in blue.

The main changes are:

  1. New theoretical result (Proposition 3, Section 3; proof in Appendix A.6). We derive a sharp training-to-evaluation discrepancy bound, establishing a necessary and sufficient condition under which functional disagreement measured using training continuations locally bounds disagreement under evaluation continuations. We also characterize the optimal bounding constant, clarifying how the choice and sampling distribution of shared futures affect the discrepancies that FACES can constrain.

  2. New permutation state-tracking experiment (Section 4.4; Appendix E). We introduce a StateBench-inspired [1] experiment using a different recurrent architecture and exact equivalences based on the Coxeter braid relation. We compare FACES against Ordinary supervision, Semigroup consistency, and latent path consistency. Under extreme length extrapolation to 4,096 future swaps, FACES-Future achieves 99.51% full-state exact accuracy, compared with 87.77% for Ordinary, 89.84% for Semigroup, 88.30% for matched future latent consistency, and 97.75% for FACES-0.

  3. Additional experimental controls and diagnostics (Sections 4.1–4.3; Appendices B–D). We compare symmetric FACES against both anchoring directions on Affine and Visual SE(2), and evaluate both reference-route choices for MS-AFACES on EuRoC. For Visual SE(2), we additionally introduce Ordinary N+4, which provides four extra steps of exact task supervision, and report fresh-versus-replay evaluations and auxiliary gradient-allocation diagnostics to investigate the performance differences between FACES-4 and the exact-target controls.

We have also updated the abstract, Introduction, and experimental appendices to incorporate these findings and clarify the corresponding protocols.

All other points are addressed in our individual responses to the reviewers. We welcome further discussion.

References: [1] Anonymous. A fine-grained hierarchy and benchmark for state tracking in language models. In Submitted to The Fifteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=1oon6lKQdR. under review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.