acceptodds
Under review as a conference paper at ICLR 2027

Complex Reasoning Tasks have Directional Structure

Abstract

While many datasets are believed to lie near low-dimensional manifolds, many different geometric structures can explain the same observations. What differentiates one manifold from another is its geometry: which inputs are close together and which are far apart. In language, this locality is typically defined by word co-occurrences. As linguist J. R. Firth put it, "You shall know a word by the company it keeps." In vision, locality is often defined via data augmentations: images that differ by a random crop or a small jitter should be similar. However, how to define this locality for downstream decision-making tasks is unclear. Decision-making, is directional: one state can be close to another in the direction of task progression, but not vice versa. Thus, we argue that to represent sequential tasks, a manifold requires a notion of direction in addition to static representations. In this paper, we show how to equip contrastive reinforcement learning with bits about time, regularising the learned representations. We show that this regularisation improves performance on complex reasoning tasks, reducing the number of steps required to solve the Rubik's Cube by 26% and improving learning efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive Reinforcement Learning, Combinatorial Puzzles, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x9w4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Predictive Temporal Contrastive Representations (PTCR), introducing time-dependent rotations into a shared representation space to model temporal directionality. Maze and tree experiments examine the limitations of symmetric critics, followed by evaluation on five combinatorial planning tasks. The strongest gains appear on Rubik’s Cube and Sokoban.

Critical strengths

S1. The problem is clearly motivated through controlled experiments.

The paper distinguishes dataset directionality from critic symmetry. The maze experiment varies trajectory directionality, while the tree experiment examines increasing depth. These studies help explain when directional representations are useful.

S2. The multi-task evaluation provides positive empirical evidence.

PTCR is competitive across five tasks and improves over CRTR particularly on Rubik’s Cube and Sokoban. Figure 6 reports cumulative solution-length distributions, the main experiments use five seeds, and Appendix G examines different solution budgets.

S3. The method is compact.

PTCR retains a shared encoder and introduces time-dependent asymmetry through structured rotations. This design integrates directly into the existing contrastive learning framework.

Critical weaknesses

W1. The theoretical basis of the latent-prediction interpretation needs clarification.

Appendix C combines strictly bounded representations in Assumption 5 with a nondegenerate Gaussian marginal in Assumption 6. Assumption 7 also applies the transformation to the future state, unlike the objective in Section 5.2. These inconsistencies leave the connection between the proposed objective and the latent-prediction result unclear.

Please state the objective consistently and derive Theorem 8 and Corollary 9 under compatible assumptions, explaining how they apply to the proposed parameterization.

W2. A key comparison is missing to establish the contribution of conditioning on elapsed time.

The experiments compare symmetric critics, separate encoders, and full PTCR. They do not determine whether the gains require time-dependent transformations or can be obtained through a shared encoder with a single asymmetric transformation, as discussed in Section 5.2.

A matched comparison between training with a one-step transformation and training with time-dependent transformations is needed to establish the value of the central design.

W3. The reported 26% reduction in solution length requires a matched comparison.

The Rubik’s Cube means of 460 and 340 steps are computed over each method’s successfully solved instances. Even if CRTR’s solved set is contained in PTCR’s, the two means can reflect different instance difficulty.

Please report paired solution lengths on commonly solved instances, alongside overall success rates and uncertainty estimates for the length difference.

Questions for authors

Q1. Under which compatible assumptions do Theorem 8 and Corollary 9 hold, and how do these results apply to the proposed objective and parameterization?

Q2. How does training with a one-step transformation throughout compare with conditioning the transformation on elapsed time under otherwise identical settings?

Q3. What is the reduction in Rubik’s Cube solution length on instances solved by both methods, and what is the uncertainty of this paired difference?

2 comments

Dear Reviewer,

Thank you for your detailed feedback. We have already improved the paper thanks to it.

W1/Q1. Under which compatible assumptions do Theorem 8 and Corollary 9 hold, and how do these results apply to the proposed objective and parameterization?

Thank you for pointing out the errors in the Appendix C assumptions! We have corrected Assumptions 5 and 7.

Assumption 5: The bound should be imposed in expectation rather than pointwise, making it compatible with the nondegenerate Gaussian marginal in Assumption 6.

Assumption 7: The temporal transformation should be applied to the current-state representation, consistently with our proposed objective.

We agree that the theoretical assumptions are strong and do not claim that they hold exactly in the environments considered. Theorem 8 and Corollary 9 provide an idealized interpretation of the temporal transformation as latent prediction.

We additionally provide empirical support in Appendix D. In Colorful Maze, visualizing trajectory representations using PCA shows that applying the transformation to the current state representation moves it approximately in the direction of the next state.

Q2/W2. How does training with a one-step transformation throughout compare with conditioning the transformation on elapsed time under otherwise identical settings?

We conducted a preliminary controlled experiment comparing the proposed AΔA^\Delta transformation with applying AA independently of the temporal gap, keeping all other hyperparameters fixed. The results are:

EnvironmentSingle ACRTRPTCR
Sokoban (late)0.080.090.24
Digit Jumper (early)0.450.820.97
Digit Jumper (late)0.910.910.98

In Sokoban, the Single AA variant performs similarly to CRTR (0.08 vs. 0.09), while PTCR achieves substantially higher performance (0.24). This suggests that introducing asymmetry alone is insufficient to explain PTCR's improvement.

In Digit Jumper, the Single AA variant eventually reaches CRTR's performance (0.91), but learns considerably more slowly, achieving 0.45 at the earlier checkpoint compared with 0.97 for PTCR. PTCR also retains an advantage at the later checkpoint (0.98 vs. 0.91).

Overall, the preliminary results suggest that conditioning the transformation on elapsed time might be providing benefits beyond introducing asymmetry alone, both in final performance and learning speed. These experiments are currently limited to two environments and one seed. We plan to extend the comparison to additional environments and seeds in the revised paper.

Q3/W3. What is the reduction in Rubik’s Cube solution length on instances solved by both methods, and what is the uncertainty of this paired difference?

While the lengths were actually measured on paired sets, we agree that the description we provided in the experimental section did not explicitly say that this was the case.

To be precise, if we increase the Rubik’s Cube budget to 10,000 from 1000, both CRTR and PTCR are able to solve all the problem instances. That is the setup, where we measured the average solution lengths. We updated the paper to be precise about this.

With that budget, the average solution length across 5 seeds of PTCR is 340 with standard error of around 16 and for CRTR is 460 with standard error of around 22.

The relative increase is 26.1% and its standard error is 5.2%.

We would be grateful for a confirmation of whether we have addressed the reviewer’s concerns.

Thank you for clarifying the Rubik’s Cube results. With the 10,000-step budget, both methods solve every instance, so the reduction from 460 to 340 steps is measured on the same set of problems. Together with the standard errors reported across five seeds, this resolves W3 for me.

I also appreciate the new comparison for W2. Applying the same transformation once regardless of the time gap is a useful control, and the results on Sokoban and Digit Jumper support the authors’ interpretation that conditioning on elapsed time contributes to PTCR’s gains.

My remaining concern is W1. The revision clarifies the norm assumption and applies the transformation to the current state, as in the training objective. However, Assumption 7 still uses a positive distance in the exponent, while the objective uses a negative squared distance. As written, I still do not see how Theorem 8 yields the proposed latent-prediction interpretation for the objective used to train PTCR. This is the main reason I am keeping my original rating of weak rejection.

Reviewer_Cz9MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Predictive Temporal Contrastive Representations (PTCR), which introduces temporal direction into contrastive representations through a shared encoder and a time-dependent latent rotation. Specifically, it defines dataset directionality, studies symmetric critics in controlled environments, and evaluates the method on five puzzle-solving tasks. The results demonstrate nontrivial improvements over the baseline on several tasks.

Critical strengths
  • The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology.
  • The proposed parameterization is simple and structured, using a shared encoder with a latent rotation to introduce asymmetry.
  • The paper conducts comprehensive experiments across five environments, demonstrating improved performance on several tasks and better learning efficiency.
Critical weaknesses
  • Theorem 2 establishes asymmetry in transition probabilities, but does not show that asymmetric representations are necessary for effective goal-directed control. As Section 6.4 acknowledges, symmetric Euclidean representations can still encode distance to the goal.

  • The paper lacks a controlled ablation comparing AΔA^\Delta with AA, leaving it unclear whether modeling elapsed time provides benefits beyond asymmetry alone.

  • The model accounts for different time gaps during training but always uses the one-step transformation at evaluation. This may work in practice, but the paper does not explain why this is the right choice. An ablation comparing different time gaps would help justify the evaluation strategy.

  • Figure 5 demonstrates faster learning on a fixed dataset, but does not test whether PTCR needs fewer training examples. The claim of improved efficiency on all tasks also conflicts with the reported lack of improvement on 15-Puzzle.

  • The reported 26% reduction in solution length compares different sets of solved problems. Comparing both methods on the same solved instances would clarify how much PTCR actually shortens solutions.

Questions for authors
  • How sensitive is planning performance to the temporal horizon used at inference?
  • How do the directionality estimates change when measured toward terminal goals rather than nearby sampled goals?
1 comment

Dear Reviewer,

Thank you for the time spent reviewing our paper!

Questions

How sensitive is planning performance to the temporal horizon used at inference?

For each of the 5 environments, we tried 3 additional inference horizons (2, 4, and 8).

EnvN=1N=2N=4N=8
Sokoban0.264±0.0100.260±0.0090.256±0.0130.260±0.014
Rubik0.976±0.0090.966±0.0070.966±0.0070.966±0.017
N-Puzzle0.972±0.0040.980±0.0040.976±0.0020.974±0.004
Lights Out0.940±0.0270.998±0.00200
Digit Jumper0.937±0.0030.900±0.0060.897±0.0230.863±0.019

For all environments except Lights Out and 15-Puzzle, N=1N=1 performs best. For these two, N=2N=2 is better. Performance generally does not collapse rapidly, except for Lights Out at N≥4N\geq 4. Digit Jumper also declines as N grows. We conclude that N=1N=1 is a good default, though increasing it might sometimes yield better performance. See Appendix H for full plots.

How do the directionality estimates change when measured toward terminal goals rather than nearby sampled goals?

For Rubik’s Cube, calculating directionality toward the final goal is intractable. We added estimates for the other environments (Appendix K.2):

DatasetToward terminal goalsToward sampled goals
Colorful Maze ε=0.01.001.00
Colorful Maze ε=0.20.940.91
Colorful Maze ε=0.40.870.84
Colorful Maze ε=0.60.800.75
Colorful Maze ε=0.80.710.74
Lights Out0.720.93
Digit Jumper0.650.82
15-Puzzle0.500.64
Rubik's Cuben/a0.96
Sokoban0.960.97

The results change significantly, particularly for Lights Out, Digit Jumper, and 15-Puzzle. However, the relative ordering of environments remains unchanged, and the monotonic decrease in directionality as ϵ\epsilon increases in Colorful Maze is preserved.

Weaknesses

Theorem 2 establishes asymmetry in transition probabilities, but does not show that asymmetric representations are necessary for effective goal-directed control.

We agree that Theorem 2 does not establish that asymmetric representations are necessary for successful goal-directed control in all environments. Our claim concerns the expressivity required to recover the optimal contrastive critic.

Under the standard assumptions for backward InfoNCE, the optimal critic satisfies

d∗(s,g)=−log⁡pt+(g∣s)p(g)+c(g).d^*(s,g)=-\log\frac{p_{t+}(g\mid s)}{p(g)}+c(g).

For a fixed goal, the critic orders states by their future-goal probabilities, enabling greedy planning when these probabilities increase toward the goal.

However, when directional datasets induce asymmetries in the optimal density-ratio critic, a symmetric parameterization cannot, in general, represent it. A symmetric critic may still learn a different useful distance (Section 6.4), but asymmetry removes one obstacle to converging to the theoretical optimum.

The paper lacks a controlled ablation comparing AΔA^\Delta with AA.

We conducted a preliminary experiment comparing AΔA^\Delta with applying AA once, keeping all other hyperparameters fixed:

EnvironmentSingle ACRTRPTCR
Sokoban (late)0.080.090.24
Digit Jumper (early)0.450.820.97
Digit Jumper (late)0.910.910.98

In Sokoban, the single-AA variant performs similarly to CRTR (0.08 vs. 0.09), while PTCR achieves 0.24. In Digit Jumper, single-AA eventually reaches CRTR's performance but learns considerably more slowly (0.45 vs. 0.97 for PTCR early). PTCR also retains an advantage later (0.98 vs. 0.91).

These preliminary results suggest that PTCR is something more than asymmetry alone. The experiments are currently limited to two environments and one seed. We plan to extend them to additional environments and seeds.

Figure 5 does not directly test data efficiency, and the claim of improved efficiency on all tasks conflicts with 15-Puzzle results.

We agree that the wording was inaccurate and removed the claim of improvement on all tasks. In all environments except Sokoban, each training example is seen once, so fewer training steps implies fewer distinct examples (for the same batch size). However, Figure 5 does not directly evaluate performance as a function of total dataset size. We replaced "data efficiency" with "faster learning" or "training efficiency" throughout the paper.

The reported 26% reduction in solution length compares different sets of solved problems.

The lengths were actually measured on paired sets, but the manuscript did not explicitly state this. At a Rubik’s Cube budget of 10,000 instead of 1,000, both CRTR and PTCR solve all instances. We measured average solution lengths at this budget. We clarified this in Section 6.2.

We hope these additional experiments and clarifications address your main concerns. We are happy to discuss any remaining questions.

Reviewer_kjEtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PTCR, a time-dependent asymmetric contrastive critic for puzzle solving. It reports improved learning efficiency over CRTR and shorter Rubik’s Cube solutions.

Critical strengths
  1. Clear motivation and simple design. The method incorporates temporal ordering while retaining a shared representation space.

  2. Useful controlled experiments. The maze and tree experiments provide evidence supporting the directional representation hypothesis.

  3. Meaningful empirical gains. Evaluation covers five environments and five seeds, with substantial improvements over CRTR on Rubik’s Cube and Sokoban.

Critical weaknesses
  1. Incomplete comparison with structured asymmetric representations. The paper is missing out some important and relevant baselines include Learning Temporal Distances (ICML 2024) and Offline Goal-conditioned RL with Quasimetric Representations (NeurIPS 2025). These comparisons would clarify the advantage of PTCR’s particular parameterization. The separate-encoder ablation is informative but uses shorter training runs and does not replace comparison with stronger structured alternatives.

  2. Limited validation of important algorithmic choices. Training uses a transformation raised to the sampled temporal gap, whereas inference always uses a one-step transformation. The authors acknowledge this heuristic, but I think it will still be valuable to do comparisons across inference horizons. Likewise, sensitivity to the rotation angles and evidence supporting fixed versus learned angles would help assess applicability to new environments.

  3. Some empirical claims need more precise support. The reported 26% solution-length reduction compares averages over different sets of solved instances. It will be more clear if the paper shows paired results on the common solved subset alongside the useful cumulative success curves already provided.

  4. There are some errors with the mathematical presentation. Section 3.2 defines the critic as squared distance but uses a positive-distance exponential in the contrastive objective, reversing the intended preference for nearby positive pairs. The stated optimal critic also omits the logarithmic relationship to the density ratio.

Questions for authors

see weaknesses

1 comment

Dear Reviewer,

We thank you for all the suggestions!

Incomplete comparison with structured asymmetric representations.

We thank the reviewer for suggesting these additional baselines. We agree that CMD (Learning Temporal Distances, ICML 2024) and TMD (Offline Goal-conditioned RL with Quasimetric Representations, NeurIPS 2025) are relevant comparisons. We ran preliminary experiments on Colorful Maze, adapting CMD and TMD to our experimental setting. Both baselines achieve lower final performance than CRTR and PTCR. The results are summarized below:

MethodPeakFinal @ 270k
CRTR0.527 @ 256k0.410
PTCR0.911 @ 180k0.852
CMD0.371 @ 230k0.262
TMD0.527 @ 100k0.351

TMD initially performs well, reaching the same peak performance as CRTR, but its performance deteriorates later in training. We experimented with different auxiliary-loss coefficients but were unable to prevent this deterioration. The training curves are provided in Section 6.5, with a more detailed description of the experimental setup and adaptations in Appendix N. We plan to extend these comparisons to the remaining environments and will share additional results during the discussion period if they become available.

Limited validation of important algorithmic choices.

We agree that including a grid over different horizons is a valuable signal about the stability of our method. For each of the 5 environments (Rubik’s Cube, Sokoban, Digit Jumper, Lights Out and 15-Puzzle) we try 3 additional inference horizons (2, 4, and 8).

EnvN=1N=2N=4N=8
Sokoban0.264±0.0100.260±0.0090.256±0.0130.260±0.014
Rubik0.976±0.0090.966±0.0070.966±0.0070.966±0.017
N-Puzzle0.972±0.0040.980±0.0040.976±0.0020.974±0.004
Lights Out0.940±0.0270.998±0.00200
Digit Jumper0.937±0.0030.900±0.0060.897±0.0230.863±0.019

For all the environments, with the exception of Lights Out and 15-puzzle, the value of one is either the best or tied best when compared to higher values. For Lights Out and 15-puzzle, the value of one still performs well, but the value of two is slightly better. Moreover there usually isn’t really a rapid collapse. For Lights Out, we observe that N=2N=2 is significantly better, but higher values essentially have a performance of 0. For Digit Jumper the performance also seems to decline as NN grows. We conclude that 1 is a good default value, though increasing it might sometime yield a better performance. See Appendix H for the full plots.

While we agree that a careful ablation of the angles would be a meaningful addition to the paper, we were unable to get the full training results for that. If we are able to get them before the end of discussion period, we will post an update, otherwise we will add them to our manuscript.

Some empirical claims need more precise support.

While the lengths were actually measured on paired sets, we agree that the description we provided in the experimental section did not explicitly say that this was the case.

To be precise, if we increase the Rubik’s Cube budget from 1000 to 10000, both CRTR and PTCR are able to solve all the problem instances. That is the setup, where we measured the average solution lengths. We updated the manuscript to be precise about this (see Section 6.2).

Issues with the mathematical formalism.

Thanks for pointing this out! We fixed the mathematical error in the paper.

Thanks again for the review! We are be happy to respond to any possible follow up questions, that might appear based on our rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.