- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces FACES, an auxiliary training objective that encourages recurrent models to produce consistent future behavior from internal states reached through semantically equivalent computation histories. Unlike immediate agreement, FACES compares task relevant outputs after a shared continuation, exposing state differences over longer rollouts. The paper develops exact-depth, cumulative, anchored, and multiscale variants and derives a local quadratic characterization of their sensitivity to state perturbations. Experiments on affine programs, visual SE(2) transformations, EuRoC inertial trajectories, and a discrete memory-writing task show improvements in long-horizon accuracy and route consistency over several baselines.
Critical strengths
- A well motivated objective with a useful theoretical interpretation.
- Consistent experimental comparisons with meaningful controls.
Critical weaknesses
- Limited evidence of general applicability. Most experiments use synthetic or highly controlled settings. Although EuRoC provides a real world evaluation, it remains unclear whether FACES benefits more general recurrent architectures, larger scale sequential models, or tasks where equivalent computation histories are harder to specify.
- Novelty relative to existing consistency objectives needs sharper validation. The paper distinguishes FACES from path consistency, semigroup consistency, and predictive state approaches, but the empirical comparisons do not fully establish its advantage over a broader range of closely related methods.
Questions for authors
- How sensitive are the gains to the choice and sampling distribution of shared continuations? Can the method fail when sampled futures do not expose the relevant state discrepancies?
- Can the authors demonstrate FACES on a different sequential modeling task or architecture, with comparisons against a closely related consistency objective?
5 comments
Response to Reviewer pE17 (2/5)
Full Proof: Upperbounding Evaluation Route Discrepancy Using Training Route Discrepancy
This continues from the notation in the first rebuttal comment. Let be an orthonormal basis of the attainable route-defect subspace , write , and define
Because and are expectations of matrices of the form , both and are positive semidefinite.
We first ask whether the training discrepancy can control the evaluation discrepancy. This requires the existence of a finite constant such that
or equivalently,
for every possible route-defect direction . Here, denotes the positive-semidefinite matrix order.
Proposition: A finite such exists exactly when
Proof of necessity: Suppose there exists a finite satisfying
Take any . Since both matrices are positive semidefinite,
Therefore,
For a positive-semidefinite matrix, this implies . Hence,
and therefore
Proof of sufficiency and the smallest admissible constant: Now suppose
Let
Here, denotes the orthogonal complement. On , is positive definite. Any component in also belongs to , so it contributes to neither quadratic form. Therefore, it is sufficient to consider nonzero .
The smallest admissible constant is then
Let
On the supported range,
Here, denotes the positive square root of the Moore-Penrose pseudoinverse.
Substituting gives
This is the Rayleigh quotient of
Therefore,
Here, denotes the largest eigenvalue.
We now return to the expected quadratic discrepancies. From the paper,
and
Since ,
Substituting the two local expansions gives
Thus, training-future discrepancy locally controls evaluation-future discrepancy exactly when every route-defect direction that matters under the evaluation future law is also exposed by the training future law. For affine behavioral maps, the quadratic expansions are exact, so the remainder term disappears.
(2/5)
Response to Reviewer pE17 (3/5) - Edited to clarify additional training details
2. Can the authors demonstrate FACES on a different sequential modeling task or architecture, with comparisons against a closely related consistency objective?
To address this concern, we performed a StateBench-inspired (reference [1], anonymous) permutation state-tracking experiment. This is not the exact StateBench benchmark. However, we took the idea of permutation/state tracking under extremely long extrapolation horizons and constructed a sequential computation task around it. This experiment uses a different recurrent architecture from the GRU-based experiments in our paper. We also added related consistency objectives, namely Semigroup consistency and Latent Path Consistency, as controls.
We split this response into two comments. The experimental protocol and results are now included in Appendix E of the revised manuscript.
A. StateBench-inspired permutation state-tracking task
The task consists of 12 registers containing distinct values from 0 to 11. The initial state is a random permutation. At each step, two registers are swapped by a transposition . After a sequence of swaps, the model must output the current value of every register. All 12 registers must be correct for the prediction to count as accurate. Training uses relatively short swap sequences, while evaluation extends the same recurrent computation to extremely long horizons.
The recurrent model maps each register value into an 8-dimensional latent vector. When registers and are swapped, their latent states are updated using a learned symmetric transition:
Here, and are learned matrices and is a learned bias vector. A shared linear decoder predicts the value in each register. This architecture provides a structural bias while being genuinely different from a GRU. Small transition errors can compound over long recurrent horizons, making the task suitable for studying whether FACES improves long-horizon performance.
To construct exact equivalences, we use the Coxeter braid relation. For three distinct registers ,
Each paired route executes 6 shared history swaps, then branches through the two equivalent three-swap braid sequences, after which a shared continuation is applied. This braid is the shortest nontrivial exact equivalence involving interacting swaps: the routes differ and share registers, but implement exactly the same permutation. Although interchangeable under any shared future, a learned recurrent model can map them to different hidden states.
B. FACES and consistency baselines
We compare Ordinary, FACES-0, FACES-Future, and related consistency controls. We also tested a dense four-probe FACES objective under a shorter training schedule.
All methods use final-step cross-entropy supervision; Ordinary uses only this objective. FACES-0 encourages functional agreement immediately after the two equivalent braid routes join.
We also tested four dense FACES probes at the join and the next three continuation depths, , with a four-swap training continuation. With 6 history, 3 braid, and 4 continuation swaps, the training rollout is 13 steps. Under this short-horizon protocol, all methods failed at long horizons: at , Ordinary, Semigroup, both Latent-PC variants, FACES-0, and the dense four-probe FACES variant all obtained about 39-40% exact accuracy. Thus, this 13-step, four-probe rollout is insufficient for extreme extrapolation.
We therefore extended the shared training continuation to 12 swaps. To keep four probes while covering the longer continuation, FACES-Future (4 probes) evaluates functional agreement at depths . Thus, the probes span the full continuation rather than concentrating near the route join. The training rollout is then 21 steps, with cross-entropy supervision at the final step.
We also include a Semigroup (involution) baseline enforcing
by encouraging the latent state to return to its original representation after applying the same swap twice. This local equivalence differs from the braid equivalence used by the other methods.
Finally, Latent-PC-0 encourages the latent states produced by the two braid routes to agree at the join, while Latent-PC-Future (4 probes) enforces latent agreement at the same depths used by FACES-Future. Thus, Latent-PC-Future and FACES-Future use the same equivalent routes and probe locations; they differ only in whether agreement is enforced in hidden-state or functional prediction space.
References:
[1] Anonymous. A fine-grained hierarchy and benchmark for state tracking in language models. In Submitted to The Fifteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=1oon6lKQdR. under review.
(3/5)
Response to Reviewer pE17 (1/5)
Dear Reviewer pE17,
Thank you for carefully reading and for your constructive review of our work.
Because there are multiple questions and OpenReview comments have a length limit, we split our response into five comments.
We provide our answers to your questions below:
1. How sensitive are the gains to the choice and sampling distribution of shared continuations? Can the method fail when sampled futures do not expose the relevant state discrepancies?
FACES is indeed sensitive to the choice of shared continuations. Yes, it can fail when sampled futures do not expose relevant route defects. Our current experiments already show both sensitivity to continuation choice and failure cases with unfavorable continuations. In fact, the local expected quadratic discrepancy developed in our theory can characterize when training continuations control discrepancies under another future distribution.
A. Sensitivity to continuation choice
In EuRoC, AFACES-≤K uses a cumulative set of horizons from 0 to K. Recall the paper's local expected quadratic discrepancy:
Jjh(δ)=δ⊤Γj(h)δ+O(∥δ∥3).
More generally, if the cumulative objective weights the included horizons by positive weights αj,
J≤Kh(δ)=∑j=0KαjJjh(δ)=δ⊤Γ≤K(h)δ+O(∥δ∥3),
where
Γ≤K(h)=∑j=0KαjΓj(h).
For positive weights,
kerΓ≤K=⋂j:αj>0kerΓj.
Thus cumulative FACES can expose any route-defect direction exposed by at least one included horizon, while changing K changes the set of directions covered and their relative weighting.
Empirically, this dependence is substantial. On EuRoC validation, the best cumulative horizon changes from K=1 at c=0,.25 to K=4 at c=.50,.75,1. For example, AFACES-≤1 outperforms AFACES-≤4 at c=0 (0.08309 vs. 0.11500), while AFACES-≤4 performs better over the coarse regimes c=.75,1 (0.05352 vs. 0.07044 averaged over the two). This sensitivity motivated MS-AFACES, which combines multiple continuation scales.
Our task-qualification studies (details in Appendix G) also contain direct failure cases. In RNAR Heapsort and 3D-R2N2, more than 90% of the initial route discrepancy is removed by the common future. These futures are poor probes because ordinary future computation itself erases most route dependence, leaving little discrepancy for FACES to correct.
B. Failure when sampled futures do not expose the discrepancy
Suppose training and evaluation future laws induce Γtr(h) and Γev(h), possibly with different horizons jtr and jev. Let V be an orthonormal basis of the attainable route-defect subspace R, and define
Gtr=V⊤Γtr(h)V,Gev=V⊤Γev(h)V.
A finite constant κ satisfying
Gev⪯κGtr
exists exactly when
kerGtr⊆kerGev.
The smallest such constant is
κ⋆=λmax(Gtr†/2GevGtr†/2),
where Gtr†/2 is the positive square root of the Moore-Penrose pseudoinverse and λmax denotes the largest eigenvalue. This gives, for sufficiently small route defects,
Jjevh(δ)≤κ⋆Jjtrh(δ)+O(∥δ∥3).
We provide the full proof in a separate comment.
The kernel condition characterizes whether coverage exists at all: every attainable route-defect direction that matters under evaluation futures must also be exposed by training futures. However, finite coverage does not imply strong control. κ⋆ depends on the relative geometry of Gtr and Gev; if evaluation strongly amplifies a direction that training exposes only weakly, κ⋆ can be large and the bound correspondingly weak. Thus continuation choice affects both which relevant directions are exposed and how strongly they are constrained.
Importantly, this result is not restricted to different horizons. Even when jtr=jev, changing the sampling distribution over continuations changes Gtr and Gev, and the same coverage condition applies.
Therefore, a shorter jtr can control a longer jev if the shorter futures already expose every evaluation-relevant direction, which may help explain the extreme extrapolation in our new StateBench-inspired experiment below. Conversely, if a direction is invisible during training but becomes visible under evaluation futures, the kernel inclusion fails and no finite κ⋆ exists. In that case, the training FACES discrepancy cannot control the evaluation discrepancy.
(1/5)