acceptodds
Withdrawn from ICLR 2027

Off-Diagonal Atlas Transport

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Next-token prediction determines what a language model must predict, but does not uniquely determine how task-relevant information is organized across token positions. Temporal representation objectives can shape this organization. However, matching the outputs of a learned projection across positions admits an easy shortcut: a persistent output direction can satisfy the objective without tracking hidden-state changes. We introduce Off-Diagonal Atlas Transport (OAT), a training-time regularizer that maps each hidden state through multiple learned projections and rewards temporal compatibility only across different projection indices. Because off-diagonal matching can itself become trivial when projections of the same state become nearly identical, OAT also suppresses positive overlap among them. Across three instruction-tuned language-model families and three low-data fine-tuning tasks, OAT improves mean accuracy over next-token prediction in all nine backbone–task combinations, with its clearest gains on mathematical reasoning. Ablations show that allowing same-index temporal matches or removing cross-projection redundancy suppression makes the auxiliary objective easy to satisfy. Yet the downstream benefit largely disappears under these changes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language models, representation learning, temporal representation learning, auxiliary objectives, data-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c3RLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes and tests a training-time auxiliary loss called OFF-DIAGONAL ATLAS TRANSPORT (OAT), designed to align projections of sampled pairs of past and future tokens, in addition to regular next-token prediction. OAT is compared to vanilla next-token prediction and a recent method Semantic Tube Prediction, and the work found improvements over NTP across 3 instruction-tuned model families in low-data fine-tuning settings, with advantages over STP on GSM datasets.

Critical strengths
  • This writing is very clear and generally easy to follow.
  • The experiments are comprehensive, covering multiple model families and datasets, ablations, cost tradeoff, and exploration of training-from-scratch beyond finetuning.
  • This work opens some interesting questions, e.g. what was learned in the atlas.
Critical weaknesses
  • While the methodology and quantitative results are described thoroughly, I find the paper a bit thin on the speculation and discussion on the behavioral intuition of why this method worked. That could be useful for a broad audience. E.g., if some behavioral analysis could offer specific insights on what types of temporal relationship/alignment are better preserved under OAT, or why OAT supported math reasoning better, that would make the work much more interesting.
Questions for authors
  • Have you considered interpreting what was learned in the atlas/frames?
  • How might OAT scale with larger models or longer sequences? Curious to hear your intuition on if the cost efficiency holds when the model scales up e.g. when more token pairs are needed to realize the benefit.
Reviewer_kfwiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Off-Diagonal Atlas Transport (OAT), a training-time auxiliary loss that regularizes temporal relationships among hidden states during language-model fine-tuning. Each hidden state is mapped through F learned projections ("frames"); each view at position t is encouraged to match its most similar view of a different index at a future position u (diagonal exclusion), while positive overlap among views of the same state is suppressed (redundancy suppression). Together these constraints aim to prevent trivial solutions such as constant projection outputs. The projections are discarded at inference, so inference cost is unchanged. On three small instruction-tuned models (Llama-3.2-1B, OLMo-2-1B, Qwen2.5-0.5B) and three low-data tasks (NL-RX SYNTH/TURK, GSM8K), OAT improves mean accuracy over NTP in all nine combinations, although STP is stronger on the regular-expression tasks, and gains largely vanish with full data and at 3B scale. Ablations show that removing either constraint drives the auxiliary loss to nearly zero while eliminating the downstream benefit.

Critical strengths

A careful and systematic analysis of shortcut solutions. The structural ablations (Tables 4–5) show that allowing same-index matches or removing redundancy suppression drives the auxiliary loss to nearly zero while eliminating the downstream benefit, and the proposed diagnostics (same-index similarity, within-state redundancy, target occupancy) clearly distinguish the two failure modes. The paper further tests whether matching reduces to a position-conditioned routing rule (Appendix B) and identifies a permutation-like shortcut that arises when training from scratch (Appendix C). The method is thus built up by identifying and closing successive shortcuts. The lesson that an easily optimized auxiliary objective need not be useful is valuable for designing temporal and JEPA-style representation objectives, independent of OAT itself.

Critical weaknesses
  1. The writing explains how the method works but not why it should help. The paper carefully describes the mechanics of OAT, but it is hard to see what problem it is trying to solve. What the paper directly addresses is a design problem: constructing a flexible temporal auxiliary objective that cannot be satisfied by trivial solutions. It does not answer the question that should precede this, namely what internal organization of representations is desirable and why it should improve downstream performance. The authors themselves acknowledge that they cannot determine whether the gains come from future-conditioned supervision or from generic regularization (l.418–424). Figure 1 likewise illustrates only the computation and does not show the central contrast of the paper: how the naive objective collapses and how OAT prevents it. As a result, the method is built up as a chain of patches against successively discovered shortcuts (constant outputs → redundant views → positional routing → permutation-like switching). Without a defined notion of the desired representation, it is unclear how many shortcuts must be closed before the objective is sufficient.

  2. The auxiliary coefficients are selected on test accuracy. According to Appendix A, the coefficients for both OAT and STP are chosen as those with the highest five-seed mean test accuracy. The Llama SYNTH and GSM8K entries in Table 1 are the very runs used for this selection, and the OLMo and Qwen results on the same tasks use coefficients chosen on the same test sets. Only TURK and GSM1K can be considered independently evaluated. Moreover, the effect is sensitive to the coefficient: on Llama/GSM8K, λ=0.01 underperforms NTP (30.46 vs. 30.67) while λ=0.04 yields +1.76 pts (Table 8). Most improvements are small (often 0.3–1 pt), and no significance tests or confidence intervals are reported. A rough estimate from the five-seed means and standard deviations suggests that several cells, such as OLMo on SYNTH and TURK (t≈1.6), are not clearly significant. It is therefore impossible to judge whether the reported gains exceed the bias introduced by test-set selection, which bears directly on the validity of the main results.

Questions for authors

Clarifications (could change my assessment)

  1. Validation-based selection. Do the gains in Table 1 hold when the auxiliary coefficients are selected on a held-out validation split rather than the test set? Since seeds are matched, please also report paired tests or confidence intervals.

  2. Temporal structure or regularization? Across Tables 5, 6 and 10, downstream gains appear only when the auxiliary loss remains far from zero, and accuracy increases with F together with the residual loss. Does this suggest that the benefit comes from the optimization pressure of a hard-to-satisfy objective (i.e., regularization) rather than from the temporal structure it is meant to induce? Controls that could distinguish these explanations include:

    • matching to past, randomly chosen, or other-example positions instead of future positions;
    • comparing with a generic regularizer (e.g., dropout or stronger weight decay) in the fine-tuning setting.
  3. When does OAT help? Can the authors explain the pattern of positive and negative results in terms of when OAT should be expected to help?

    • STP outperforms OAT on all six regular-expression cells.
    • Gains vanish with full data, where OAT trails NTP on SYNTH.
    • Gains are negligible at 3B.
    • Trends across data fractions are non-monotonic.

    In particular, why is NTP on GSM8K lower at 1/4 data (27.98) than at 1/16 data (30.67) (Table 2)? Could this be an artifact of the fixed-presentation-budget protocol?

Suggestions

  1. Replacing the geometric metaphor (atlas, frame, transport) with a direct description of the actual structure, and adding to Figure 1 a contrast between the collapsing naive objective and OAT, would make the motivation much clearer.
Reviewer_r6dpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Off-Diagonal Atlas Transport (OAT), a training-time regularizer that maps hidden states through multiple learned projections and encourages temporal compatibility between different projection indices. The paper evaluates OAT on low-data fine-tuning for three tasks, with additional ablations, data-fraction studies, a small 3B scale check, and controlled scratch-training experiments.

Critical strengths
  1. The off-diagonal matching and redundancy suppression design is interesting and well motivated.
  2. The experimental framework is well controlled. The main comparison covers three model families, three task families, five matched seeds, data-fraction sweeps, a 3B scale check, ablations, sensitivity, and a scratch-training study.
Critical weaknesses
  1. The scope and practical impact seem limited. The main results cover three relatively small models, mostly in 0.5B-1B. The single 3B check shows only a very small benefit on GSM8K and no benefit on GSM1K. In addition, the improvements are modest and diminish in some full-data and larger-model settings, making the broader practical impact unclear.
  2. The comparison with STP suggests task-dependent benefits, where OAT is stronger on mathematical reasoning, and STP is stronger on regular-expression generation. The paper could more explicitly characterize what task properties determine this difference.
  3. The paper does not clearly identify what information is transported across OAT frames or why this helps mathematical reasoning.
  4. The method introduces several design choices and hyperparameters, while robustness to layer choice, temporal gap, projection configuration, and sampling choices is only partially established.
Questions for authors

See weaknesses.

Comments on the paper

1 comment

We have decided to withdraw our submission after further consideration. We sincerely thank the reviewers and Area Chair for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.