acceptodds
Under review as a conference paper at ICLR 2027

Local Temporal Contrasts on Human Motion Data Induce a Latent Geometry

Abstract

Time series such as recordings of human motion data often encode structure across multiple temporal scales, and when a latent state determining this sequence evolves, this structure needs to survive in the representation space for downstream use. Yet the currently prevalent attention-based encoders paired with sequence-level pooling have made single-vector, pointwise comparison the default mode of inference, discarding the trajectory a sequence traces through the representation space. We propose recovering this geometric structure directly, without relying on forecasting-based objectives, by training an encoder model (SemTAC) with a temporal-neighborhood contrastive loss: representations that satisfy local consistency everywhere must give rise to a coherent global structure, letting inference happen at temporal scales beyond those seen during training. We evaluate SemTAC against four baselines on a five-step motion sequence discrimination task: the same architecture trained with a classification or a standard contrastive objective, and two established self-supervised time-series methods, TS-TCC and TS2Vec. At the full sequence length all five are broadly competitive, but on incomplete, truncated sequences SemTAC holds a large advantage: a median F1 of 0.77 one step before the sequence completes, against 0.27–0.70 for the four baselines. A hyperparameter-matched ablation attributes this effect specifically to the temporalcontrastive objective. SemTAC's representation trajectories are also the only ones among these five to exhibit interpretable geometric structures, exceeding a no-structure chance baseline on a displacement-smoothness metric which we introduce. This is evidence that its objective, not classification accuracy alone, produces genuinely temporally-ordered trajectories.

Primary area:
learning on time series and dynamical systems
Keywords:
Self-supervised Representation Learning, Contrastive Learning, Time Series Representation Learning, Temporal Neighborhood Coding, Latent Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tySwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how to preserve the temporal structure by propose SemTAC, which is built upon a Semantic Encoder and is trained using an InfoNCE objective. To validate, the authors use MDM to synthesize a human motion dataset and construct 1,420 sequences. For evaluation, the authors introduce metrics including smoothness, order recoverability, silhouette score, and kNN consistency, aiming to measure temporal organization and branching properties in the latent space. The experimental results show that contrastive learning based on local temporal adjacency can induce a latent geometry that reflects global temporal structure.

Critical strengths
  1. The idea of inducing global geometric structure from local temporal relationships is interesting and potentially insightful.
  2. The authors introduce metrics such as smoothness and order recoverability to evaluate the representation space. These analyses provide additional interpretability to the experiments.
Critical weaknesses
  1. The paper is not well written, and the overall presentation lacks clarity, particularly in the method section. The paper frequently uses terms such as latent geometry, global structure and trajectory structure, but these concepts are not clearly defined. More importantly, the paper lacks a complete method overview figure, making the proposed approach difficult to understand.
  2. The proposed method is relatively incremental, with limited technical novelty. The author acknowledges that the method is closely related to Temporal Neighborhood Coding.
  3. The final dataset contains only 1,420 sequences, which is limited in scale and does not provide sufficiently strong evidence.
  4. The experiments are conducted on a dataset synthesized by MDM, the generalization ability to real-world data has not been validated.
Questions for authors

please see weakness.

2 comments

Weakness 1: clarity and method overview

We thank Reviewer tySw for the review. We agree that the method section and the terminology need to be clearer. Figures 1 and 2 convey the concept and the data, but there is no pipeline figure. In the revision we will review and define those terms on first use, and add a method overview figure showing pair sampling, the encoder and InfoNCE objective, prototype construction, and DTW/Fréchet inference.

Weakness 2: novelty

We thank the reviewer for recognizing the central idea as interesting and potentially insightful, and we agree that its positioning can be made clearer in the introduction, which we will revise. As acknowledged in the paper, the closest prior work is TNC; the difference is that we define positive pairs across stages (different motions), and our contribution is the finding, the ablation analysis and the evaluation metrics rather than a new objective (see also our reply to Reviewer UXNo).

Weakness 3: dataset size

The dataset is designed as a controlled probe of whether temporal structure is recovered, not as a benchmark for absolute accuracy. For that purpose, we consider its scale adequate: the test split alone contains about 280 sequences (20% of 1,420), results are aggregated over multiple runs per model (10 seeds for the baselines, about 20 grid-search configurations for SemTAC), and the effects are large (rank-biserial correlations up to 1.0, with Holm-Bonferroni correction). The number of sequences is bounded by the quality filtering step. We agree that sensitivity to dataset size is a legitimate question and we would welcome the reviewer’s view on what scale would be convincing.

Weakness 4: real-world data

We agree that generalization to real-world data has not been validated; Sec. 5 identifies this as the next step. Evaluating on synthetic data was a deliberate design choice: measuring how much of a temporal structure is recovered requires ground truth about that structure, and we are not aware of public real-world datasets annotated in a way that allows this. Generating the motions with MDM, followed by quality filtering, let us control the structure (shared steps, gradual lateralization, label noise). We will make this scope explicit in the abstract and conclusion.

Thank you to the author for the response. After carefully reviewing the additional information and experiment results, I find that my concerns have been partially addressed. However, I still have reservations about the paper’s novelty and writing quality. I have therefore decided to raise my score from 1 to 2.

Reviewer_UXNoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The author artificially designed a five-stage synthetic human action sequence, and then used adjacent action comparative learning to form a similar trajectory structure in the feature space. Finally, it was demonstrated that the model can recognize this artificially designed structure.

Critical strengths

The research perspective on potential trajectories does have some value.

Critical weaknesses
  1. The most serious problem is that all experiments are based on human body movements generated by MDM, and the author artificially constructed a five-stage, two-branch sequence. The intermediate stage introduces category bias through preset ratios of 60/40 and 80/20, and the final stage directly uses different actions to distinguish categories. This makes the evaluation task highly consistent with the temporal adjacency used by the method. The model's better performance on this task does not mean that it can learn the more complex and unknown temporal structure of real human motion. It is in the truncated sequence that the left and right waving in stage 4 itself has 80% category indicativeness. The paper lacks a simple baseline that only uses actions at this stage for judgment, so it cannot clearly prove that the model's advantage comes from global trajectory geometry rather than local action semantics.
  2. SemTAC basically follows the existing Transformer, InfoNCE and Temporal Neighborhood Coding ideas. The main modification is to treat adjacent action stages as positive samples. I think there is little methodological innovation.
Questions for authors

Why do I need SemTAC's underlying spatial geometry trajectory if I already know whether the stage 4 motion is a left-hand swing or a right-hand swing? I need the authors to prove with a baseline that this global trajectory structure indeed provides information that cannot be obtained by local action recognition.

1 comment

Weakness 1: evaluation design

We thank Reviewer UXNo for the careful review. We agree that the dataset is synthetic and that the evaluation structure is deliberately aligned with the temporal adjacency used by the method: having ground truth about the structure is what allows us to measure recovery. The results therefore show that such a structure can be recovered, not that the more complex, unknown temporal structure of real human motion can be, and we will state this limitation more explicitly. The label noise (60/40, 80/20) was introduced precisely so that the sequence type is not trivially identifiable from the first lateralized step.

Weakness 2: methodological novelty

We agree that SemTAC combines existing components (a Transformer encoder, InfoNCE, and a TNC-style temporal contrast), and the paper says so (Sec. 3.3 and Sec. 5). The difference to TNC is that positive pairs are also defined across stages (different motions), which TNC would not consider (Sec. 2). Our contribution is not a new architecture or objective, but (i) the finding that adjacency-based contrasts give rise to representation spaces organized like the underlying temporal structure, (ii) ablations that isolate the adjacency transformation as the responsible component, and (iii) the metrics and geometry-level inference used to measure and exploit this. Most label-free baselines and the Self-similarity ablation show displacement smoothness close to or below the −0.5 chance level (−0.46 to −0.58, including three additional methods we have since evaluated, see our reply to Reviewer EFQE), with SimMTM at the sequence level (−0.38) the one clear exception, whereas SemTAC reaches −0.26 on average (with high variance). Strong smoothness therefore does not appear to be a generic by-product of contrastive time-series learning, although the margin to SimMTM remains to be tested. We also note that order recoverability is less discriminative: the Classification ablation (0.73) and, when applied per segment, SimMTM (0.82) reach values comparable to or above SemTAC (0.79).

Question: value of the global structure over local action recognition

This is a fair and important point. As noted in Sec. 4, the label noise at step 4 caps the F1 at truncated length at about 0.8, a level that a classifier using only the step-4 motion, with access to motion labels, could also reach. We therefore do not claim that SemTAC extracts information that such a classifier cannot, nor that it is the only method that can solve the truncated task: under DTW several label-free methods also approach this ceiling (TS2Vec 0.70 in the paper; SoftCLT 0.70, or 0.74 when applied per segment, in the additional experiments), while under Fréchet inference SemTAC has the higher mean (0.68 vs 0.40–0.56). Our claim is narrower: SemTAC reaches this level (median F1 0.77/0.78, mean 0.68/0.72) using only adjacency between unlabeled segments, without motion-type labels and without knowing which motions indicate which sequence type, and it additionally shows the most temporally organized geometry (highest mean displacement smoothness, branching metrics; Tables 3–4). We will also state clearly that this benchmark is a controlled probe of structure recovery, not evidence of benefit on real data of unknown structure.

Reviewer_EFQEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a model called SemTAC, which contrasts temporally proximal states against distant ones. The authors claim that this approach could enable comparisons of entire representation geometries rather than individual points. However, I find the overall quality of the paper poor and significantly below the standard expected for ICLR.

Critical strengths

(1) Novel Perspective on Latent Geometry: The concept of geometry-level inference via sequence similarity introduces an interesting perspective to the field. To the best of my knowledge, this specific angle has not been explicitly explored in prior literature. However, its concrete design and underlying motivation remain somewhat unclear.

(2) Structural Evaluation: The authors successfully evaluate and report the structural properties of the learned representation spaces. The experimental results indicate that the proposed contrastive approach, rooted in local temporal interactions, outperforms some structured space baselines. Table 3 provides supporting evidence that SemTAC induces a different temporal organization from the comparison methods, while Tables 4 and 5 reveal trade-offs with branch separation and local motion classification. Nevertheless, its empirical advantages are inconsistent, as it underperforms existing baselines on several metrics.

Critical weaknesses

(1) The motivation and benefits of this work for the domain are not convincing at all. In both the abstract and the contribution list, it is very difficult to understand which research question the authors are trying to address and how the work could contribute to the field.

(2) The writing and presentation are very poor. The framing and explanations are difficult to follow, and the sentences are hard to understand. I think the manuscript needs substantial polishing.

(3) The experimental evaluation is not solid at all. Only two models are compared, and they are very basic methods such as TS-TCC and TS2Vec, which are not state of the art at all. Tables 4 and 5 indicate that SemTAC introduces severe trade-offs rather than outperforming the existing models. For example, SemTAC does not achieve the best F1-score (mean ± standard deviation) for single-motion classification. Therefore, I find it very confusing how these results support the main claim. Only synthetic datasets are used for evaluation.

(4) The technical contributions are very limited. DTW and Fréchet distances are used for evaluation. Only two baseline models, TS-TCC and TS2Vec, are used for comparison, but they are not the best trending model at all.

Questions for authors

1.Could the authors provide further justification for why this research question is important and how addressing it could benefit the field?

2.Could the authors further explain SemTAC’s performance drop in Tables 4 and 5?

3.Could the evaluation be extended beyond the existing synthetic dataset to include more real-world datasets?

3 comments

Weaknesses (3)–(4): baselines and technical contribution

We thank the reviewer for pushing on the comparison set. Our aim is to characterize how different training signals shape the geometry of local embeddings, so the comparisons are meant to show what each signal adds, not to rank methods. Besides TS-TCC and TS2Vec, the paper includes two ablations with SemTAC’s architecture that differ only in the objective (self-similarity only; supervised classification). Following the reviewer’s comment, we also evaluated SoftCLT, Temporal Cycle-Consistency (TCC) and SimMTM with the same protocol as TS2Vec and TS-TCC (same splits, whole sequences encoded jointly, 10 seeds, hyperparameters from the original papers); results are in the table below.

The methods show different geometric profiles. Adjacency contrasts give the highest means for smoothness, order recoverability, silhouette and kNN consistency, and the highest truncated-sequence F1 under Fréchet (0.68 vs 0.41–0.55). The whole-sequence encoders SimMTM and TCC also lift smoothness above the −0.5 chance level (−0.38 and −0.46); applied per segment, SimMTM falls to chance (−0.50), so cross-motion context is a plausible explanation we have not tested. Under DTW, truncated F1 is similar for SoftCLT, TS2Vec and SemTAC (0.70, 0.70, 0.72), and per-segment SimMTM reaches a higher order recoverability (0.82). Single-motion classification is where adjacency contrasts cost something (Table 5). We read these results as a map of what each objective preserves in the geometry, not as a ranking, and we will present them that way, with statistical testing, and revise claims such as “consistently outperforming” and “the only model above chance-level smoothness”.

ModelF1 full (Fréchet / DTW)F1 steps 1–4 (Fréchet / DTW)SmoothnessOrder rec.Silhouette (Fréchet / DTW)kNN cons. (Fréchet / DTW)
SemTAC (paper)0.87±0.12 / 0.86±0.060.68±0.21 / 0.72±0.19−0.26±0.300.79±0.100.13±0.13 / 0.20±0.110.95±0.07 / 0.94±0.06
TS2Vec (paper)0.81±0.04 / 0.91±0.020.40±0.19 / 0.70±0.05−0.57±0.020.67±0.040.05±0.02 / 0.10±0.010.87±0.02 / 0.90±0.02
TCC (sequence)0.683±0.055 / 0.764±0.0500.552±0.159 / 0.564±0.179−0.459±0.0340.472±0.0530.053±0.028 / 0.056±0.0250.689±0.055 / 0.713±0.076
SoftCLT (sequence)0.805±0.041 / 0.911±0.0160.409±0.180 / 0.698±0.050−0.568±0.0220.668±0.0370.045±0.017 / 0.097±0.0090.868±0.021 / 0.901±0.015
SimMTM (sequence)0.529±0.025 / 0.822±0.0350.492±0.009 / 0.545±0.043−0.377±0.0100.608±0.0460.010±0.004 / 0.089±0.0070.837±0.024 / 0.879±0.014
SoftCLT (per-segment)0.785±0.043 / 0.836±0.0170.420±0.264 / 0.738±0.028−0.533±0.0090.731±0.0490.033±0.012 / 0.107±0.0060.879±0.024 / 0.901±0.024
SimMTM (per-segment)0.638±0.024 / 0.882±0.0160.564±0.017 / 0.675±0.025−0.502±0.0060.824±0.0110.006±0.003 / 0.036±0.0020.678±0.031 / 0.806±0.029

Note: Means (± std where available). SemTAC and TS2Vec values are from Tables 1–4 of the paper (TS2Vec: 10 seeds; SemTAC: about 20 configurations from the hyperparameter grid search, so its std also reflects hyperparameter sensitivity). “Sequence” = encoded jointly with cross-motion context, the protocol used for TS2Vec and TS-TCC; “per-segment” = each motion encoded independently. All new methods: 10 seeds. All baselines use the hyperparameters recommended in their original papers.

Continued reply to Reviewer EFQE.

Weakness (2): writing and presentation

We thank Reviewer EFQE for the detailed review. We agree that the presentation needs improvement. In a revision we will (i) state the research question explicitly in the abstract and the contribution list, (ii) review and define introduced terms, (iii) add a method overview figure covering pair sampling, the encoder and InfoNCE objective, prototype construction, and DTW/Fréchet inference, and (iv) edit the text throughout for clarity.

Question 1: motivation and benefit

The research question is: can contrasting only temporally adjacent segments, without class labels, induce a latent geometry that mirrors the global temporal structure of a dataset, and can that geometry be used for inference on sequences that have not been completed? The potential benefit is representations that encode where a state lies within a process and what tends to precede or follow it, rather than only which cluster it belongs to, which is relevant for early prediction and monitoring of evolving states (e.g., disease progression). The method is self-supervised from temporal adjacency and uses no motion-type labels, so the structure it learns is not given to it directly. In our experiments this structure supports sequence-type inference from truncated sequences (Table 2: significantly better than all compared models except the Classification ablation with DTW; additional baselines are discussed in our separate comment on baselines) and organizes the space along time more than the baselines (Table 3; the difference in order recoverability to the Classification ablation is not significant). We view recovering a known structure as necessary to quantify the observed effect that local temporal contrasts induce a structure in latent space that corresponds to the data’s temporal structure. Applying it to datasets of unknown structure is the motivation and the future work, not something this paper claims to demonstrate. We will state the research question explicitly in the abstract and contributions.

Question 2: Tables 4 and 5

Table 4. Table 4 measures how well the embeddings separate the two sequence types. SemTAC is second best on every entry, behind only the Classification ablation, which is trained to isolate each motion type into its own cluster; since the final steps of the two sequence types are entirely different motions, this alone separates the sequence types without having to encode the global temporal structure. The gap is small for kNN consistency (0.95/0.94 vs 1.00) and larger for silhouette (0.13/0.20 vs 0.29/0.35). The point of Table 4 is that the temporal organization of SemTAC’s representations (Table 3) does not come at the cost of separating the two sequence types.

Table 5. Table 5 measures retention of single-motion information. All models are far above the chance level of 0.125, but SemTAC (0.85) is below TS2Vec (0.98), Classification (0.92) and TS-TCC (0.86), and the Self-similarity ablation is lower still (0.80). Because that ablation has no adjacency contrast, we read the cost as tied to the self-similarity transformation rather than to the branching structure (Sec. 5). This is a suggestion, not a tested conclusion. We acknowledge it as a genuine trade-off, will make it more prominent, and leave controlling it to future work.

Question 3: real-world data

We agree that evaluation on real-world data is a limitation, and we state it as the next step in Sec. 5. Our evaluation requires ground-truth knowledge of the temporal structure of the data in order to measure how much of it is recovered, and we are not aware of public real-world datasets annotated in a way that allows this. We therefore used synthetic data as a deliberate design choice. We will make this scope explicit in the abstract and conclusion.

Reviewer_EFQEOpenReview ↗

Response to the rebuttal

Thank you for the clarification and the additional experiments. The rebuttal addresses my concern about the breadth of the baselines and clarifies that the paper is a study of what different training objectives preserve, rather than a claim of consistent superiority. I also agree that synthetic data are appropriate for studying recovery of known temporal structure.

My main concern remains that the current experiments do not clearly separate recovery of global temporal geometry from improved use of branch-specific cues in this particular five-step construction. An order-insensitive control using the same embeddings, or a second controlled construction that varies sequence order while preserving local motion content, would help distinguish these explanations. The new comparisons are informative, but SemTAC's variation across hyperparameter configurations is not directly comparable to the baselines' variation across random seeds. I therefore appreciate the substantive response but retain my original recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.